From 4452cfd57beec98d5af26f2d63d1a181e81b57ba Mon Sep 17 00:00:00 2001 From: Jack Date: Fri, 16 Jan 2026 14:34:23 -0800 Subject: [PATCH 1/2] Align logo better --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index adeca591..8ec217f4 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ ![hero](https://github.com/user-attachments/assets/7ed92cea-4c81-4ccd-9f7f-738763bfdee4) -# FutureSearch everyrow SDK +# FutureSearch everyrow SDK The everyrow python SDK is an interface to [everyrow.io](https://everyrow.io) for intelligent data processing utilities powered by AI agents. Rank, Dedupe, Merge, and Screen your dataframes using natural language instructions, or apply web agents to research every row. From 421b0b6e9568c2b705497f773634cc3e568b8111 Mon Sep 17 00:00:00 2001 From: Jack Date: Fri, 16 Jan 2026 15:40:08 -0800 Subject: [PATCH 2/2] Expand the README --- README.md | 229 ++++++++++---------- docs/DEDUPE.md | 125 ++++++----- docs/MERGE.md | 87 ++++++++ docs/RANK.md | 86 ++++++++ docs/SCREEN.md | 104 +++++++++ thematic_screen_results_20260114_172950.csv | 10 - 6 files changed, 463 insertions(+), 178 deletions(-) create mode 100644 docs/MERGE.md create mode 100644 docs/RANK.md create mode 100644 docs/SCREEN.md delete mode 100644 thematic_screen_results_20260114_172950.csv diff --git a/README.md b/README.md index 8ec217f4..3956edae 100644 --- a/README.md +++ b/README.md @@ -2,34 +2,28 @@ # FutureSearch everyrow SDK -The everyrow python SDK is an interface to [everyrow.io](https://everyrow.io) for intelligent data processing utilities powered by AI agents. Rank, Dedupe, Merge, and Screen your dataframes using natural language instructions, or apply web agents to research every row. +Python SDK for [everyrow.io](https://everyrow.io). Rank, dedupe, merge, and screen your dataframes using natural language—or run web agents to research every row. ## Table of Contents -- [Getting Started](#getting-started) -- [Installation](#installation) -- [Claude Code Plugin](#claude-code-plugin) -- [Usage](#usage) - - [Rank](#rank) - - [Dedupe](#dedupe) - - [Merge](#merge) - - [Screen](#screen) - - [Agent Tasks](#agent-tasks) - - [Async Operations](#async-operations) -- [Case Studies](#case-studies) -- [Development](#development) -- [License](#license) +New to everyrow? Head to [Getting Started](#getting-started) + +Looking to use our agent-backed utilities? Check out: +- [Rank](#rank) +- [Dedupe](#dedupe) +- [Merge](#merge) +- [Screen](#screen) +- [Agent Tasks](#agent-tasks) ## Getting Started Get an API key at [everyrow.io](https://everyrow.io). ```bash -# Set in your environment or .env file export EVERYROW_API_KEY=your_api_key_here ``` -## Installation +### Installation ```bash pip install everyrow @@ -42,11 +36,11 @@ uv pip install -e . uv sync ``` -**Requirements:** Python >= 3.12 +Requires Python >= 3.12 -## Claude Code Plugin +### Claude Code Plugin -This repository includes a plugin for [Claude Code](https://code.claude.com/) that teaches Claude how to write code using the everyrow SDK. +There's a plugin for [Claude Code](https://code.claude.com/) that teaches Claude how to use the SDK: ```sh # from Claude Code @@ -58,109 +52,95 @@ claude plugin marketplace add futuresearch/everyrow-sdk claude plugin install everyrow@futuresearch ``` -## Usage +## Rank -The SDK provides flexible session management. For quick one-off operations, sessions are created automatically: +Score rows based on criteria you can't put in a database field. The AI researches each row and assigns scores based on qualitative factors. ```python -from everyrow.ops import single_agent +from everyrow.ops import rank -# Simplest usage - session created automatically -result = await single_agent( - task="What is the capital of France?", - input={"country": "France"}, +result = await rank( + task="Score by likelihood to need data integration solutions", + input=leads_dataframe, + field_name="integration_need_score", ) ``` -For multiple operations, use an explicit session to group them together: - -```python -from everyrow import create_session - -async with create_session(name="My Session") as session: - print(f"View session at: {session.get_url()}") - # All operations in this block share the same session -``` - -For maximum control, create the client explicitly: - -```python -from everyrow import create_client, create_session - -async with create_client() as client: - async with create_session(client=client, name="My Session") as session: - # ... use session for operations -``` +Say you want to rank leads by "likelihood to need data integration tools"—Ultramain Systems (sells software to airlines) looks similar to Ukraine International Airlines (is an airline) by industry code, but their actual needs are completely different. Traditional scoring can't tell them apart. -Sessions are logical groupings of tasks visible on the [everyrow.io](https://everyrow.io) interface. +**Case studies:** [Lead Scoring with Data Fragmentation](https://futuresearch.ai/lead-scoring-data-fragmentation/) (1,000 leads, 7 min, $13) · [Lead Scoring Without CRM](https://futuresearch.ai/lead-scoring-without-crm/) ($28 vs $145 with Clay) -### Rank - -Score and rank rows based on complex criteria. - -```python -from everyrow.ops import rank - -result = await rank( - task="Score by contribution to AI research", - input=dataframe, - field_name="contribution_score", -) -print(result.data) -``` +[Full documentation →](docs/RANK.md) ### Dedupe -Intelligently deduplicate data using AI-powered equivalence detection. +Deduplicate when fuzzy matching falls short. The AI understands that "AbbVie Inc", "Abbvie", and "AbbVie Pharmaceutical" are the same company, or that "Big Blue" means IBM. ```python from everyrow.ops import dedupe result = await dedupe( - input=dataframe, - equivalence_relation="Two entries are duplicates if they represent the same research work", + input=crm_data, + equivalence_relation="Two entries are duplicates if they represent the same legal entity", ) -print(result.data) ``` +The `equivalence_relation` tells the AI what counts as a duplicate—natural language, not regex. Results include `equivalence_class_id` (groups duplicates), `equivalence_class_name` (human-readable cluster name), and `selected` (the canonical record in each cluster). + +**Case studies:** [CRM Deduplication](https://futuresearch.ai/crm-deduplication/) (500→124 rows, 2 min, $1.67) · [Researcher Deduplication](https://futuresearch.ai/researcher-dedupe-case-study/) (98% accuracy with career changes) + +[Full documentation →](docs/DEDUPE.md) + ### Merge -Match and merge two tables using AI. +Join two tables when the keys don't match exactly—or at all. The AI knows "Photoshop" belongs to "Adobe" and "Genentech" is a Roche subsidiary, even with zero string similarity. ```python from everyrow.ops import merge result = await merge( - task="Match trial sponsors with parent companies", - left_table=trial_data, - right_table=pharma_companies, - merge_on_left="sponsor", - merge_on_right="company", + task="Match each software product to its parent company", + left_table=software_products, + right_table=approved_suppliers, + merge_on_left="software_name", + merge_on_right="company_name", ) -print(result.data) ``` +Handles subsidiaries, abbreviations (MSD → Merck), regional names, typos, and pseudonyms. Fuzzy matching thresholds always fail somewhere—0.9 misses "Colfi" ↔ "Dr. Ioana Colfescu", 0.7 false-positives on "John Smith" ↔ "Jane Smith". + +**Case studies:** [Software Supplier Matching](https://futuresearch.ai/software-supplier-matching/) (2,000 products, 91% accuracy, $9) · [HubSpot Contact Merge](https://futuresearch.ai/merge-hubspot-contacts/) (99.9% recall) · [CRM Merge Workflow](https://futuresearch.ai/crm-merge-workflow/) + +[Full documentation →](docs/MERGE.md) + ### Screen -Evaluate and filter rows based on criteria that require research. +Filter rows based on criteria that require research—things you can't express in SQL. The AI actually researches each row (10-Ks, earnings reports, news) before deciding pass/fail. ```python from everyrow.ops import screen -from pydantic import BaseModel +from pydantic import BaseModel, Field -class VendorAssessment(BaseModel): - approved: bool - risk_level: str - recommendation: str +class ScreenResult(BaseModel): + passes: bool = Field(description="True if company meets the criteria") result = await screen( - task="Evaluate vendor security and financial stability", - input=vendors, - response_model=VendorAssessment, + task=""" + Find companies with >75% recurring revenue that would benefit from + Taiwan tensions - CHIPS Act beneficiaries, defense contractors, + cybersecurity firms. Exclude companies dependent on Taiwan manufacturing. + """, + input=sp500_companies, + response_model=ScreenResult, ) -print(result.data) ``` +Works for investment theses, geopolitical exposure, vendor risk assessment, job posting filtering, lead qualification—anything requiring judgment. Screening 500 S&P 500 companies takes ~12 min and $3 with >90% precision. Regex gets 68%. + +**Case studies:** [Thematic Stock Screen](https://futuresearch.ai/thematic-stock-screening/) (63/502 passed, $3.29) · [Job Posting Screen](https://futuresearch.ai/job-posting-screening/) (>90% vs 68% regex) · [Lead Screening Workflow](https://futuresearch.ai/screening-workflow/) + +[Full documentation →](docs/SCREEN.md) + ### Agent Tasks For single-input tasks, use `single_agent`. For batch processing, use `agent_map`. @@ -182,9 +162,48 @@ result = await agent_map( ) ``` +Our agents are tuned on [Deep Research Bench](https://arxiv.org/abs/2506.06287), a benchmark we built for evaluating web research on questions that require extensive searching and cross-referencing. + +## Advanced + +### Sessions + +For quick one-off operations, sessions are created automatically: + +```python +from everyrow.ops import single_agent + +result = await single_agent( + task="What is the capital of France?", + input={"country": "France"}, +) +``` + +For multiple operations, use an explicit session: + +```python +from everyrow import create_session + +async with create_session(name="My Session") as session: + print(f"View session at: {session.get_url()}") + # All operations here share the same session +``` + +If you want more explicit control over the client (for example, to reuse it across sessions or configure custom settings), you can create it directly: + +```python +from everyrow import create_client, create_session + +async with create_client() as client: + async with create_session(client=client, name="My Session") as session: + # ... +``` + +Sessions are visible on the [everyrow.io](https://everyrow.io) dashboard. + ### Async Operations -All utilities have async variants for background processing. These require an explicit session since the task needs to persist beyond the function call. +All utilities have async variants for background processing. These need an explicit session since the task persists beyond the function call: ```python from everyrow import create_session @@ -204,7 +223,19 @@ async with create_session(name="Async Ranking") as session: ## Case Studies -The `case_studies/` directory contains example workflows. To run them: +More at [futuresearch.ai/solutions](https://futuresearch.ai/solutions/). + +**Notebooks:** +- [CRM Deduplication](case_studies/dedupe/case_01_crm_data.ipynb) +- [Thematic Stock Screen](case_studies/screen/thematic_stock_screen.ipynb) +- [Oil Price Margin Screen](case_studies/screen/oil_price_margin_screen.ipynb) + +**On futuresearch.ai:** +- [Lead Scoring with Data Fragmentation](https://futuresearch.ai/lead-scoring-data-fragmentation/) +- [Software Supplier Matching](https://futuresearch.ai/software-supplier-matching/) +- [Researcher Deduplication](https://futuresearch.ai/researcher-dedupe-case-study/) + +To run notebooks: ```bash uv sync --group case-studies @@ -212,40 +243,20 @@ uv sync --group case-studies ## Development -### Setup - ```bash uv sync lefthook install ``` -### Running Tests - -```bash -uv run pytest -``` - -### Linting & Formatting - -```bash -uv run ruff check . -uv run ruff check --fix . -uv run ruff format . -``` - -### Type Checking - -```bash -uv run basedpyright -``` - -### Generating OpenAPI Client - ```bash -./generate_openapi.sh +uv run pytest # tests +uv run ruff check . # lint +uv run ruff format . # format +uv run basedpyright # type check +./generate_openapi.sh # regenerate client ``` -Note: The `everyrow/generated/` directory is excluded from linting as it contains auto-generated code. +The `everyrow/generated/` directory is excluded from linting (auto-generated code). ## License diff --git a/docs/DEDUPE.md b/docs/DEDUPE.md index 19108101..3be1fb42 100644 --- a/docs/DEDUPE.md +++ b/docs/DEDUPE.md @@ -1,83 +1,90 @@ -# everyrow_sdk.dedupe Documentation +# Dedupe -AI-powered deduplication for messy datasets. +Deduplicate when fuzzy matching can't cut it. -## Relevant case-studies -### ... -### ... +## The problem -## How It Works +You've got a messy CRM export. "AbbVie Inc", "Abbvie", and "AbbVie Pharmaceutical" are obviously the same company. So are "Big Blue" and "IBM Corporation". A person who was at "BAIR Lab" last year and "Google DeepMind" this year is still the same person. -The `dedupe` operation deduplicates data through a five-stage pipeline: +Fuzzy matching forces you to pick a threshold. Set it high (0.9) and you miss "Big Blue" ↔ "IBM" (completely different strings). Set it low (0.7) and you false-positive on "John Smith" ↔ "Jane Smith". There's no threshold that works. -1. **Semantic Item Comparison**: Each row is compared against others using an LLM that understands context—recognizing that "A. Butoi" and "Alexandra Butoi" are likely the same person, or that "BAIR Lab (Former)" indicates a career transition rather than a different organization. +## How it works -2. **Association Matrix Construction**: Pairwise comparison results are assembled into a matrix of match/no-match decisions. To scale efficiently, items are first clustered by embedding similarity, so only semantically similar items are compared. +You describe what "duplicate" means for your data. The system figures out which rows match. -3. **Equivalence Class Creation**: Connected components in the association graph form equivalence classes. If A matches B and B matches C, then A, B, and C form a single cluster representing one entity. +```python +from everyrow.ops import dedupe + +result = await dedupe( + input=crm_data, + equivalence_relation="Two entries are duplicates if they represent the same legal entity", +) +``` + +The `equivalence_relation` is natural language. Be as specific as you need: -4. **Validation**: Each multi-member cluster is re-evaluated to catch false positives—cases where the initial comparison was too aggressive. +```python +result = await dedupe( + input=researchers, + equivalence_relation=""" + Two rows are duplicates if they're the same person, even if: + - They changed jobs (different org/email) + - Name is abbreviated (A. Smith vs Alex Smith) + - There are typos (Naomi vs Namoi) + - They use a nickname (Bob vs Robert) + """, +) +``` + +## What you get back -5. **Candidate Selection**: For each equivalence class, the most complete/canonical record is selected as the representative (e.g., preferring "Alexandra Butoi" over "A. Butoi"). +Three columns added to your data: -## Sample Usage +- `equivalence_class_id` — rows with the same ID are duplicates of each other +- `equivalence_class_name` — human-readable label for the cluster ("Alexandra Butoi", "Naomi Saphra", etc.) +- `selected` — True for the canonical record in each cluster (usually the most complete one) + +To get just the deduplicated rows: -### Code ```python -from everyrow import create_client, create_session -from everyrow.ops import dedupe -import pandas as pd - -input_df = pd.read_csv("researchers.csv") - -async with create_client() as client: - async with create_session(client, name="Researcher Dedupe") as session: - result = await dedupe( - session=session, - input=input_df, - equivalence_relation=( - "Two rows are duplicates if they represent the same person " - "despite different email/organization (career changes). " - "Consider name variations like typos, nicknames (Robert/Bob), " - "and format differences (John Smith/J. Smith)." - ), - ) - result.data.to_csv("deduplicated.csv", index=False) +deduped = result.data[result.data["selected"] == True] ``` -The `equivalence_relation` parameter tells the AI what counts as a duplicate. Unlike regex or fuzzy matching, this is natural language that captures the semantic intent. +## Example -### Example Input +Input: -| row_id | name | organization | email | github | -|--------|------|--------------|-------|--------| -| 2 | A. Butoi | Rycolab | alexandra.butoi@personal.edu | butoialexandra | -| 8 | Alexandra Butoi | Ryoclab | — | butoialexandra | -| 43 | Namoi Saphra | — | nsaphra@alumni | nsaphra | -| 47 | Naomi Saphra | Harvard University | nsaphra@fas.harvard.edu | nsaphra | -| 18 | T. Gupta | AUTON Lab (Former) | — | tejus-gupta | -| 26 | Tejus Gupta | AUTON Lab | tejusg@cs.cmu.edu | tejus-gupta | +| name | org | email | +|------|-----|-------| +| A. Butoi | Rycolab | a.butoi@edu | +| Alexandra Butoi | Ryoclab | — | +| Namoi Saphra | — | nsaphra@alumni | +| Naomi Saphra | Harvard | nsaphra@harvard.edu | -*Rows 2+8, 43+47, and 18+26 are duplicates (same person, different records).* +Output (selected rows only): -### Example Output +| name | org | email | +|------|-----|-------| +| Alexandra Butoi | Rycolab | a.butoi@edu | +| Naomi Saphra | Harvard | nsaphra@harvard.edu | -After running `dedupe`, duplicate rows are merged into canonical representatives: +## Parameters -| row_id | name | organization | email | github | -|--------|------|--------------|-------|--------| -| 2 | Alexandra Butoi | Rycolab | alexandra.butoi@personal.edu | butoialexandra | -| 43 | Naomi Saphra | Harvard University | nsaphra@fas.harvard.edu | nsaphra | -| 18 | Tejus Gupta | AUTON Lab | tejusg@cs.cmu.edu | tejus-gupta | +| Name | Type | Description | +|------|------|-------------| +| `input` | DataFrame | Data with potential duplicates | +| `equivalence_relation` | str | What makes two rows duplicates | +| `session` | Session | Optional, auto-created if omitted | -*The most complete/canonical record is selected as the representative for each equivalence class.* +## Performance -## API Reference +| Rows | Time | Cost | +|------|------|------| +| 200 | ~90 sec | ~$0.40 | +| 500 | ~2 min | ~$1.67 | +| 2,000 | ~8 min | ~$7 | -### `dedupe(session, input, equivalence_relation)` +## Case studies -| Parameter | Type | Description | -|-----------|------|-------------| -| `session` | `Session` | Session created via `create_session()` | -| `input` | `pd.DataFrame` | Input dataframe with potential duplicates | -| `equivalence_relation` | `str` | Natural language description of what makes two rows duplicates | +- [CRM Deduplication](https://futuresearch.ai/crm-deduplication/) — 500 rows down to 124 (75% were duplicates) +- [Researcher Deduplication](https://futuresearch.ai/researcher-dedupe-case-study/) — 98% accuracy handling career changes and typos diff --git a/docs/MERGE.md b/docs/MERGE.md new file mode 100644 index 00000000..a17231c3 --- /dev/null +++ b/docs/MERGE.md @@ -0,0 +1,87 @@ +# Merge + +Join two tables when the keys don't match exactly—or at all. + +## The problem + +You've got a list of software products and a list of approved vendors. You need to match them up. But: + +- "Photoshop" → "Adobe" (zero string similarity) +- "Genentech" → "Roche" (subsidiary) +- "MSD" → "Merck" (regional name) +- "VS Code" → "Microsoft" (product vs company) + +Fuzzy matching won't help—these aren't typos, they're semantic relationships. And the relationships aren't in any lookup table you can buy. + +## How it works + +You describe how the tables should match. Agents figure out the mapping. + +```python +from everyrow.ops import merge + +result = await merge( + task="Match each software product to its parent company", + left_table=software_products, + right_table=approved_vendors, + merge_on_left="product_name", + merge_on_right="company_name", +) +``` + +For ambiguous cases, add context: + +```python +result = await merge( + task=""" + Match clinical trial sponsors to parent pharma companies. + + Watch for: + - Subsidiaries (Genentech → Roche, Janssen → J&J) + - Regional names (MSD is Merck outside the US) + - Abbreviations (BMS → Bristol-Myers Squibb) + """, + left_table=trials, + right_table=pharma_companies, + merge_on_left="sponsor", + merge_on_right="company", +) +``` + +## What you get back + +A DataFrame with all left table columns plus matched right table columns. Rows that don't match get nulls for the right columns (like a left join). + +## Why fuzzy matching fails + +Even when there *is* some string overlap, thresholds don't work: + +- Threshold > 0.9: Misses "Dr. Ioana Colfescu" ↔ "Colfi" (same person, GitHub username) +- Threshold < 0.7: Matches "John Smith" ↔ "Jane Smith" (different people) + +There's no good threshold because matching is contextual. "Colfi" as a GitHub handle should match the full name. "John" and "Jane" shouldn't match even though they're 80% similar. + +## Parameters + +| Name | Type | Description | +|------|------|-------------| +| `task` | str | How to match the tables | +| `left_table` | DataFrame | Primary table (all rows kept) | +| `right_table` | DataFrame | Table to match from | +| `merge_on_left` | str | Column in left table | +| `merge_on_right` | str | Column in right table | +| `session` | Session | Optional, auto-created if omitted | + +## Performance + +| Size | Time | Cost | +|------|------|------| +| 100 × 50 | ~3 min | ~$2 | +| 2,000 × 50 | ~8 min | ~$9 | +| 1,000 × 1,000 | ~12 min | ~$15 | + +## Case studies + +- [Software Supplier Matching](https://futuresearch.ai/software-supplier-matching/) — 2,000 products to 50 vendors, 91% accuracy, zero false positives +- [HubSpot Contact Merge](https://futuresearch.ai/merge-hubspot-contacts/) — 99.9% recall despite GitHub handles, typos, and partial emails +- [CRM Merge Workflow](https://futuresearch.ai/crm-merge-workflow/) — joining fund-level and contact-level data diff --git a/docs/RANK.md b/docs/RANK.md new file mode 100644 index 00000000..8bc2d30a --- /dev/null +++ b/docs/RANK.md @@ -0,0 +1,86 @@ +# Rank + +Score rows based on criteria you can't put in a database field. + +## The problem + +Traditional lead scoring uses firmographic data—employee count, industry code, funding stage. But "likelihood to need data integration tools" isn't in any database. + +Ultramain Systems and Ukraine International Airlines both show up as "Aviation" in your CRM. One sells software to airlines (simple ops, unified systems). The other *is* an airline (complex ops, dozens of data sources, legacy integrations everywhere). Their actual needs are opposite, but they look identical on paper. + +## How it works + +You describe what you want to score in plain English. For each row, agents research the company and assign a score with reasoning. + +```python +from everyrow.ops import rank + +result = await rank( + task="Score by likelihood to need data integration solutions", + input=leads_dataframe, + field_name="integration_need_score", +) +``` + +The task can be as specific as you want: + +```python +result = await rank( + task=""" + Score 0-100 by likelihood to adopt research tools in the next 12 months. + + High scores: teams actively publishing, hiring researchers, or with + recent funding for R&D. Low scores: pure trading shops, firms with + no public research output. + """, + input=investment_firms, + field_name="research_adoption_score", + ascending_order=False, # highest first +) +``` + +## Structured output + +If you want more than just a number, pass a Pydantic model: + +```python +from pydantic import BaseModel, Field + +class LeadScore(BaseModel): + score: float = Field(description="0-100, higher = more likely to need integration") + reasoning: str = Field(description="Why this score") + key_signal: str = Field(description="The single most important factor") + +result = await rank( + task="Score by data integration needs", + input=leads, + field_name="score", + response_model=LeadScore, +) +``` + +Now each row has `score`, `reasoning`, and `key_signal` columns. + +## Parameters + +| Name | Type | Description | +|------|------|-------------| +| `task` | str | What to score and how | +| `input` | DataFrame | Your data | +| `field_name` | str | Column name for the score | +| `response_model` | BaseModel | Optional structured output | +| `ascending_order` | bool | True = lowest first (default) | +| `session` | Session | Optional, auto-created if omitted | + +## Performance + +| Rows | Time | Cost | +|------|------|------| +| 100 | ~2 min | ~$1.50 | +| 1,000 | ~7 min | ~$13 | +| 5,000 | ~25 min | ~$60 | + +## Case studies + +- [Lead Scoring with Data Fragmentation](https://futuresearch.ai/lead-scoring-data-fragmentation/) — 1,000 B2B leads ranked by data fragmentation risk +- [Lead Scoring Without CRM](https://futuresearch.ai/lead-scoring-without-crm/) — 85 investment firms scored for $28 (Clay wanted $145) diff --git a/docs/SCREEN.md b/docs/SCREEN.md new file mode 100644 index 00000000..850e38e8 --- /dev/null +++ b/docs/SCREEN.md @@ -0,0 +1,104 @@ +# Screen + +Filter rows based on criteria that require research. + +## The problem + +You want to find S&P 500 companies that: +- Have >75% recurring revenue +- Would benefit from Taiwan tensions (CHIPS Act, defense, cybersecurity) +- Aren't dependent on Taiwan for manufacturing + +None of this is in a database. Bloomberg can filter on P/E ratio. It can't filter on "benefits from geopolitical tensions." + +Even simpler stuff fails with traditional methods. Filtering job postings for "remote-friendly" with regex gets 68% precision—hundreds of false positives to sort through manually. + +## How it works + +You describe what should pass. Agents research each row (pulling 10-Ks, earnings calls, news) and decide. + +```python +from everyrow.ops import screen +from pydantic import BaseModel, Field + +class ScreenResult(BaseModel): + passes: bool = Field(description="True if company meets criteria") + +result = await screen( + task=""" + Find companies with >75% recurring revenue that would benefit from + Taiwan tensions. Include CHIPS Act beneficiaries, defense contractors, + cybersecurity firms. Exclude companies dependent on Taiwan manufacturing. + """, + input=sp500, + response_model=ScreenResult, +) +``` + +Only passing rows come back. + +## Richer output + +Want to know *why* something passed? Add fields: + +```python +class VendorRisk(BaseModel): + approved: bool = Field(description="True if vendor is acceptable") + risk_level: str = Field(description="low / medium / high") + security_issues: str = Field(description="Any breaches or incidents") + recommendation: str = Field(description="Summary") + +result = await screen( + task=""" + Assess each vendor for enterprise use. Research: + 1. Security incidents in past 3 years + 2. Financial stability (layoffs, funding issues) + + Approve only low/medium risk with no unresolved critical incidents. + """, + input=vendors, + response_model=VendorRisk, +) +``` + +Now you get `risk_level`, `security_issues`, and `recommendation` for every row that passed. + +## The pass/fail field + +Your response model needs a boolean field. It can be named anything—`passes`, `approved`, `include`, whatever. The system figures out which field is the filter. + +```python +class Simple(BaseModel): + passes: bool + +class Detailed(BaseModel): + approved: bool # this is the filter + confidence: float + notes: str +``` + +## Parameters + +| Name | Type | Description | +|------|------|-------------| +| `task` | str | What should pass | +| `input` | DataFrame | Rows to screen | +| `response_model` | BaseModel | Must have a boolean field | +| `batch_size` | int | Parallel rows (default 10) | +| `session` | Session | Optional, auto-created if omitted | + +## Performance + +| Rows | Time | Cost | Precision | +|------|------|------|-----------| +| 100 | ~3 min | ~$0.70 | >90% | +| 500 | ~12 min | ~$3.30 | >90% | +| 1,000 | ~20 min | ~$6 | >90% | + +Compare: regex on "remote-friendly" job postings gets 68% precision. + +## Case studies + +- [Thematic Stock Screen](https://futuresearch.ai/thematic-stock-screening/) — 63 of 502 S&P 500 companies passed, $3.29 +- [Job Posting Screen](https://futuresearch.ai/job-posting-screening/) — >90% precision vs 68% for regex +- [Screening Workflow](https://futuresearch.ai/screening-workflow/) — iterate on criteria without rerunning everything diff --git a/thematic_screen_results_20260114_172950.csv b/thematic_screen_results_20260114_172950.csv deleted file mode 100644 index b692e65e..00000000 --- a/thematic_screen_results_20260114_172950.csv +++ /dev/null @@ -1,10 +0,0 @@ -company,cik,ticker,market_cap_usd_billion,gics_sector,gics_sub_industry,headquarters,founded,research,passes_screen -Oracle Corporation,1341439,ORCL,590.4561602560001,Information Technology,Systems Software,"Austin, Texas",1977,"{'screening_result': ""Oracle meets both criteria. Its recurring revenue from 'Cloud services and license support' accounts for approximately 85.7% of total revenue ($49.23B of $57.4B in FY2025). As a major US defense cloud provider (with significant DoD contracts) and a leader in sovereign cloud and cybersecurity, it would likely see increased strategic importance and revenue from escalating US-China tensions. Its exposure to China is relatively low as a percentage of total revenue, and it has been downsizing its presence there. [Bullfincher, Oracle FY2025/FY2026 Results, USASpending]"", 'passes_screen': ""Oracle meets both criteria. Its recurring revenue from 'Cloud services and license support' accounts for approximately 85.7% of total revenue ($49.23B of $57.4B in FY2025). As a major US defense cloud provider (with significant DoD contracts) and a leader in sovereign cloud and cybersecurity, it would likely see increased strategic importance and revenue from escalating US-China tensions. Its exposure to China is relatively low as a percentage of total revenue, and it has been downsizing its presence there. [Bullfincher, Oracle FY2025/FY2026 Results, USASpending]""}",True -Palantir Technologies,1321655,PLTR,417.391640576,Information Technology,Application Software,"Denver, Colorado",2003,"{'screening_result': ""Palantir operates a high-quality recurring revenue model (SaaS/subscriptions) with approximately 80% of revenue derived from long-term contracts. It is a major beneficiary of US-China tensions as a leading provider of AI-driven defense and intelligence software (the 'AI arsenal of the free world'). It has minimal exposure to China and is not dependent on Taiwan for manufacturing, making it a pure-play geopolitical software beneficiary. (Sources: Search result 1, 3, 22, 33, 35)"", 'passes_screen': ""Palantir operates a high-quality recurring revenue model (SaaS/subscriptions) with approximately 80% of revenue derived from long-term contracts. It is a major beneficiary of US-China tensions as a leading provider of AI-driven defense and intelligence software (the 'AI arsenal of the free world'). It has minimal exposure to China and is not dependent on Taiwan for manufacturing, making it a pure-play geopolitical software beneficiary. (Sources: Search result 1, 3, 22, 33, 35)""}",True -ServiceNow,1373715,NOW,172.721618944,Information Technology,Systems Software,"Santa Clara, California",2003,"{'screening_result': 'ServiceNow has an exceptionally high recurring revenue model, with subscription revenues accounting for approximately 94-95% of total revenue ($3.299 billion of $3.475 billion in Q3 2025). It is a strategic beneficiary of US-China tensions as companies invest in digital transformation, automation, and supply chain resilience (reshoring) to mitigate geopolitical risks. Its direct China exposure is minimal, with the entire Asia-Pacific and Other region representing only about 11% of revenue, and it lacks dependency on Taiwan manufacturing.', 'passes_screen': 'ServiceNow has an exceptionally high recurring revenue model, with subscription revenues accounting for approximately 94-95% of total revenue ($3.299 billion of $3.475 billion in Q3 2025). It is a strategic beneficiary of US-China tensions as companies invest in digital transformation, automation, and supply chain resilience (reshoring) to mitigate geopolitical risks. Its direct China exposure is minimal, with the entire Asia-Pacific and Other region representing only about 11% of revenue, and it lacks dependency on Taiwan manufacturing.'}",True -Verizon,732712,VZ,171.88257792000002,Communication Services,Integrated Telecommunication Services,"New York City, New York",1983 (1877),"{'screening_result': ""Verizon's wireless service revenue, which is inherently recurring (monthly subscriptions), comprised 76% of its 2024 revenue. The company is a key provider of domestic U.S. critical infrastructure and would likely benefit from increased strategic importance and domestic telecom upgrades as U.S.-China tensions rise (e.g., reshoring connectivity, domestic cybersecurity focus). It has negligible China revenue exposure (reported at 0.00% by some trackers). Sources: [https://www.fitchratings.com/research/corporate-finance/fitch-rates-verizon-new-senior-unsecured-notes-a-10-11-2025], [https://1792exchange.com/spotlight-reports/china-risk-database/company/705/], [https://www.verizon.com/about/sites/default/files/2024-Proxy-Statement.pdf]"", 'passes_screen': ""Verizon's wireless service revenue, which is inherently recurring (monthly subscriptions), comprised 76% of its 2024 revenue. The company is a key provider of domestic U.S. critical infrastructure and would likely benefit from increased strategic importance and domestic telecom upgrades as U.S.-China tensions rise (e.g., reshoring connectivity, domestic cybersecurity focus). It has negligible China revenue exposure (reported at 0.00% by some trackers). Sources: [https://www.fitchratings.com/research/corporate-finance/fitch-rates-verizon-new-senior-unsecured-notes-a-10-11-2025], [https://1792exchange.com/spotlight-reports/china-risk-database/company/705/], [https://www.verizon.com/about/sites/default/files/2024-Proxy-Statement.pdf]""}",True -Lockheed Martin,936468,LMT,104.551489536,Industrials,Aerospace & Defense,"North Bethesda, Maryland",1995,"{'screening_result': 'Lockheed Martin (LMT) is a premier defense contractor that generates substantially all of its revenue from long-term contracts with the U.S. government and international allies. A significant portion of its revenue (~30-50%) comes from sustainment and services, which are highly recurring as they cover the multi-decade maintenance of platforms like the F-35. As the producer of HIMARS and F-35 components, LMT is a direct beneficiary of increased defense spending and specific military sales to Taiwan (recently receiving a $328.5 million contract). China revenue is minimal (estimated at ~2%), and the company is a primary beneficiary of U.S. strategic reshoring in the aerospace sector. Sources: https://investors.lockheedmartin.com/static-files/850a403e-158b-41f2-bbfa-576f0375d6f1, https://www.reuters.com/business/aerospace-defense/pentagon-says-lockheed-martin-gets-3285-million-taiwan-military-sale-contract-2025-12-31/, https://dc.medill.northwestern.edu/blog/2020/07/17/heres-why-chinese-sanctions-against-lockheed-martin-are-a-paper-tiger/.', 'passes_screen': 'Lockheed Martin (LMT) is a premier defense contractor that generates substantially all of its revenue from long-term contracts with the U.S. government and international allies. A significant portion of its revenue (~30-50%) comes from sustainment and services, which are highly recurring as they cover the multi-decade maintenance of platforms like the F-35. As the producer of HIMARS and F-35 components, LMT is a direct beneficiary of increased defense spending and specific military sales to Taiwan (recently receiving a $328.5 million contract). China revenue is minimal (estimated at ~2%), and the company is a primary beneficiary of U.S. strategic reshoring in the aerospace sector. Sources: https://investors.lockheedmartin.com/static-files/850a403e-158b-41f2-bbfa-576f0375d6f1, https://www.reuters.com/business/aerospace-defense/pentagon-says-lockheed-martin-gets-3285-million-taiwan-military-sale-contract-2025-12-31/, https://dc.medill.northwestern.edu/blog/2020/07/17/heres-why-chinese-sanctions-against-lockheed-martin-are-a-paper-tiger/.'}",True -Southern Company,92122,SO,96.996327424,Utilities,Electric Utilities,"Atlanta, Georgia",1945,"{'screening_result': ""Southern Company is a regulated utility with essentially 100% recurring revenue from long-term service agreements with customers in its service territories. It is a prime beneficiary of the 'reshoring' trend and the CHIPS Act, as semiconductor and other high-tech manufacturing facilities (e.g., Hyundai, Qcells, and Absolics) relocate to the Southeast US, driving massive industrial load growth. As a domestic utility, it has virtually no exposure to China revenue or Taiwan manufacturing risks. [Southern Company IRP, Bloomberg, SIA]"", 'passes_screen': ""Southern Company is a regulated utility with essentially 100% recurring revenue from long-term service agreements with customers in its service territories. It is a prime beneficiary of the 'reshoring' trend and the CHIPS Act, as semiconductor and other high-tech manufacturing facilities (e.g., Hyundai, Qcells, and Absolics) relocate to the Southeast US, driving massive industrial load growth. As a domestic utility, it has virtually no exposure to China revenue or Taiwan manufacturing risks. [Southern Company IRP, Bloomberg, SIA]""}",True -Duke Energy,1326160,DUK,92.495028224,Utilities,Electric Utilities,"Charlotte, North Carolina",1904,"{'screening_result': 'Duke Energy operates a rate-regulated utility model which provides highly predictable, recurring revenue from a captive customer base (often likened to a long-term contract model). It is a direct beneficiary of US-China tensions as it serves the Carolinas, a major hub for reshored semiconductor manufacturing and data centers incentivized by the CHIPS Act. This reshoring is driving significant load growth for the utility. It has negligible China revenue risk. Sources: Seeking Alpha [https://seekingalpha.com/article/4669368-duke-energy-strong-long-term-potential-despite-macro-headwinds], Duke Energy Newsroom [https://news.duke-energy.com/releases/duke-energy-proposes-new-investments-in-north-carolina-to-boost-reliability-and-support-economic-growth-across-the-state].', 'passes_screen': 'Duke Energy operates a rate-regulated utility model which provides highly predictable, recurring revenue from a captive customer base (often likened to a long-term contract model). It is a direct beneficiary of US-China tensions as it serves the Carolinas, a major hub for reshored semiconductor manufacturing and data centers incentivized by the CHIPS Act. This reshoring is driving significant load growth for the utility. It has negligible China revenue risk. Sources: Seeking Alpha [https://seekingalpha.com/article/4669368-duke-energy-strong-long-term-potential-despite-macro-headwinds], Duke Energy Newsroom [https://news.duke-energy.com/releases/duke-energy-proposes-new-investments-in-north-carolina-to-boost-reliability-and-support-economic-growth-across-the-state].'}",True -General Dynamics,40533,GD,90.876616704,Industrials,Aerospace & Defense,"Reston, Virginia",1899,"{'screening_result': 'General Dynamics meets the recurring revenue criteria via long-term defense contracts and maintenance (70% of revenue from the Pentagon with a backlog exceeding $100B). It is a direct beneficiary of Taiwan tensions, selling M1 Abrams tanks to Taiwan and building submarines critical for Indo-Pacific deterrence. While its Gulfstream business has some China exposure, 86% of its revenue is from North America, and it is a key strategic player in US defense. (Sources: Search result 2, 4, 31, 37, 39)', 'passes_screen': 'General Dynamics meets the recurring revenue criteria via long-term defense contracts and maintenance (70% of revenue from the Pentagon with a backlog exceeding $100B). It is a direct beneficiary of Taiwan tensions, selling M1 Abrams tanks to Taiwan and building submarines critical for Indo-Pacific deterrence. While its Gulfstream business has some China exposure, 86% of its revenue is from North America, and it is a key strategic player in US defense. (Sources: Search result 2, 4, 31, 37, 39)'}",True -American Tower,1053507,AMT,82.26003353600001,Real Estate,Telecom Tower REITs,"Boston, Massachusetts",1995,"{'screening_result': 'American Tower is a REIT that generates over 98% of its revenue from long-term (5-10 year) tenant leases, fitting the definition of high-quality recurring revenue perfectly. As a primary owner of U.S. digital infrastructure, it is a beneficiary of reshoring efforts and the strategic drive for secure, domestic connectivity. Its China exposure is minimal compared to its massive U.S. and global tower footprint. Sources: [https://www.americantower.com/media/2791/download], [https://seekingalpha.com/article/4857967-american-tower-stock-is-boring-but-the-upside-potential-isnt]', 'passes_screen': 'American Tower is a REIT that generates over 98% of its revenue from long-term (5-10 year) tenant leases, fitting the definition of high-quality recurring revenue perfectly. As a primary owner of U.S. digital infrastructure, it is a beneficiary of reshoring efforts and the strategic drive for secure, domestic connectivity. Its China exposure is minimal compared to its massive U.S. and global tower footprint. Sources: [https://www.americantower.com/media/2791/download], [https://seekingalpha.com/article/4857967-american-tower-stock-is-boring-but-the-upside-potential-isnt]'}",True