Skip to content

Job‐Fetch Pipeline

Jump to bottom
Chazona Baum edited this page Jun 24, 2026 · 2 revisions

Relevant source files

The Job-Fetch Pipeline is Lodestar's automated engine for discovering, analyzing, and scoring job opportunities. It transforms raw career pages into structured, scored job entities within the user's vault. The pipeline is built on a durable, retryable task-queue model where every step is a discrete unit of work persisted to a SQLite database.

The "Queue-Tasks-Are-Steps" Model

Unlike traditional linear pipelines, Lodestar uses a state-machine approach where the queue tasks ARE the stepssrc-tauri/src/pipeline/steps.rs#1-5 Each successful step enqueues its successor, carrying the output payload (e.g., scraped HTML or LLM-structured JSON) forward. This ensures that a failure in a late stage (like LLM hallucination) does not require re-running expensive or rate-limited upstream stages (like web scraping).

The pipeline is driven by a background worker thread that executes the pump_once loop src-tauri/src/pipeline/steps.rs#14-17 claiming tasks from the SqliteQueue and projecting live progress to the UI via an EventSinksrc-tauri/src/pipeline/steps.rs#62-70

Run Kinds & Execution Chains

The pipeline orchestrates three primary "Run Kinds," each serving a specific stage of the job lifecycle:

Run Kind Trigger Sequence of Steps
job_check Discovery careers-scrape → structure-listings → finalize
job_detail Deep Analysis jd-scrape → structure-jd → gap-detect → research-gaps
job_scoring Fit Alignment fit-score → alignment

1. Discovery (job_check)

Triggered at the company level, this run scrapes a careers page, uses an LLM to identify open roles, filters them against user criteria, and writes new Job stubs to the vault src-tauri/src/pipeline/steps.rs#11-12 For details, see Discovery Prefilter.

2. Deep Analysis (job_detail)

Triggered for a specific job, this run scrapes the full Job Description (JD). It uses gap-detect to identify missing critical info (like salary or tech stack) and conditionally triggers a research-gaps step using LLM-powered web search src-tauri/src/pipeline/steps.rs#132-142 For details, see Gap Detection & Research.

3. Fit Alignment (job_scoring)

The final stage (often automatically triggered by job_detail) calculates a numerical fit score and generates a natural language "Alignment" narrative comparing the job to the user's profile src-tauri/src/pipeline/steps.rs#144-150

Pipeline Architecture & Code Mapping

The following diagram bridges the conceptual pipeline stages to the specific Rust modules and traits that implement them.

System Logic to Code Entity Mapping

flowchart LR
    subgraph subGraph2 ["Data Transformation"]
        Sanitize["sanitize.rs (HTML Cleaning)"]
        Filter["filter.rs (Prefilter)"]
        Prompts["prompts.rs (LLM Logic)"]
    end
    subgraph subGraph1 ["External Integration"]
        Scraper["Scraper Trait (ScrapingBee)"]
        LLM["Llm Trait (OpenRouter)"]
    end
    subgraph subGraph0 ["Orchestration Layer"]
        Runner["runner.rs (Step Execution)"]
        Steps["steps.rs (Chain Logic)"]
        Queue["SqliteQueue (Persistence)"]
    end
    Steps --> Queue
    Steps --> Runner
    Runner --> Scraper
    Runner --> LLM
    Scraper --> Sanitize
    LLM --> Prompts
    Steps --> Filter
Loading

Sources: src-tauri/src/pipeline/steps.rs#40-51src-tauri/src/pipeline/queue.rs#33-35src-tauri/src/pipeline/runner.rs#10-25

Pipeline Execution Flow

This diagram illustrates how a single job_detail run progresses through the system, highlighting the transition from raw web data to structured vault notes.

Job Detail Execution Flow

sequenceDiagram
    participant UI as Frontend (Svelte)
    participant W as Worker (worker.rs)
    participant Q as SqliteQueue
    participant S as Scraper (scraper.rs)
    participant L as LLM (llm.rs)
    participant V as Vault (note.rs)
    UI->>W: fetch_job_details(job_slug)
    W->>Q: enqueue("jd-scrape")
    Q->>W: claim_next()
    W->>S: scrape(url)
    S-->>W: raw_html
    W->>W: sanitize(raw_html)
    W->>Q: enqueue("structure-jd" | payload: sanitized)
    Q->>W: claim_next()
    W->>L: prompt(build_structure_jd_prompt)
    L-->>W: StructuredJd (JSON)
    W->>V: update_job_field(...)
    W->>Q: enqueue("gap-detect")
Loading

Sources: src-tauri/src/worker.rs#63-65src-tauri/src/pipeline/steps.rs#15-17src-tauri/src/pipeline/runner.rs#41-55

Sub-System Summaries

Pipeline Steps & Run Orchestration

Details the pump_once execution loop, the EventSink trait for real-time UI updates, and the specific state transitions for discovery, detail, and scoring runs. See Pipeline Steps & Run Orchestration for details.

Task Queue & Step Runner

Explains the durable SQLite-backed queue that handles task states (pending, claimed, done, dead) and the StepRunner which manages telemetry, error classification, and exponential backoff. See Task Queue & Step Runner for details.

Scraper & HTML Sanitization

Covers the Scraper trait, ScrapingBee integration with automatic proxy escalation (Premium to Stealth), and the sanitize.rs logic that strips noise from HTML to minimize LLM token costs. See Scraper & HTML Sanitization for details.

LLM Integration & Prompt Engineering

Describes the Llm trait, OpenRouter implementation, and the prompts.rs module which uses "DATA-fence" markers to protect against prompt injection during JD analysis. See LLM Integration & Prompt Engineering for details.

Gap Detection & Research

Explains the logic that inspects a structured Job for missing fields (e.g., visa_sponsorship) and triggers a targeted web search via the LLM to fill those gaps. See Gap Detection & Research for details.

Discovery Prefilter

Covers the high-recall filtering logic used during discovery to discard irrelevant roles based on title keywords and deduplicate against existing URLs in the vault. See Discovery Prefilter for details.

Sources:

Clone this wiki locally