Daily Search Pipeline

Daily search pipeline

The daily search (daily_run) is one task with these steps. Each one logs to the run's live log.

flowchart TD
  A[1. Scrape all sources] --> B[2. Drop aggregators and reposters]
  B --> C[3. Dedup against seen jobs and the tracker]
  C --> D[4. Triage: keep plausible roles]
  D --> E[5. Company size filter]
  E --> F[6. Fetch full posting text]
  F --> G[7. Score fit 0-100]
  G --> H[8. Pick the best]
  H --> I[9. Draft CV, cover letter, Why this role?]
  I --> J[10. Compile and verify]
  J --> K[Ready or Needs review]

Loading

Step	What happens	Who
1. Scrape	Company career boards, Instahyre, LinkedIn and freehire. See Job Sources.	Python
2. Blocklist	Any job whose company or link matches AGGREGATORS (Jobgether, Weekday, Turing, Crossover, Toptal, Uplers, staffing firms ...) is dropped.	Python
3. Dedup	Skips jobs already seen (job_scraper/seen_jobs.json, by link and by company + title) and roles already in the tracker.	Python
4. Triage	One cheap pass over titles, companies and locations, in batches of 150. Drops wrong role types, other-country listings, body shops and reposters. When unsure it keeps the job.	Claude (triage model)
5. Size	Drops companies below Minimum company size (default 500 employees). Sizes come from the source, the LinkedIn company page or a web search, cached for 90 days. Unknown sizes are dropped unless Keep unknown sizes is on.	Python + Claude
6. Details	Reads the full posting text. Instahyre postings older than 30 days are skipped here.	Python
7. Score	Batches of 12 postings scored against your profile and deal-breakers: a 0-100 fit score, a location verdict (PASS / FLAG / FAIL) and the gaps. Pay below your floor becomes a SALARY FAIL gap.	Claude (scoring model)
8. Select	Keeps scores at or above Minimum fit score (default 60) with a location PASS and no salary fail; soonest deadline first, then best score; at most Drafts per run (default 8).	Python
9. Draft	For each pick, 3 at a time: re-reads the posting (closed → EXPIRED, deal-breaker → VETO), saves it, tailors the closest existing CV and cover letter, and writes the "Why this role?" answer.	Claude (drafting model)
10. Verify	Compiles in the sandbox, then checks: CV exactly 2 pages, letter exactly 1, no orphaned headings, extractable text, no banned names. Any failure → Needs review with the reason in the notes.	Python

The run ends with a summary such as 3 ready, 0 need review, 27 scored, 188 new of 642 scraped and, if enabled, a phone alert.

Where the results go

• job_search_tracker.csv: one row per application (status, fit, notes, files, apply link, deadline).
• cv/main_<slug>.tex/.pdf, cover_letters/cover_<slug>.tex/.pdf
• documents/applications/<slug>/: job_posting.md, why_role.md, later interview_prep.md.

Tuning it

Want	Change
Fewer, better drafts	raise Minimum fit score, lower Drafts per run
Smaller companies too	lower Minimum company size (0 turns the filter off)
Different roles	your target roles in CLAUDE.md; queries in app/pipeline.py and app/instahyre.py
Different location	see Job Sources