Skip to content

Pipeline Steps & Run Orchestration

Jump to bottom
Chazona Baum edited this page Jun 24, 2026 · 1 revision

Relevant source files

The pipeline orchestration system manages the lifecycle of automated job discovery, data extraction, and fit analysis. It implements a "queue tasks ARE the steps" model where each stage of a pipeline run is a durable, retryable unit of work persisted in a SQLite-backed queue. This design ensures that failures in late stages (e.g., LLM analysis) do not require re-executing expensive upstream work (e.g., web scraping).

Execution Loop & Pump Mechanism

The pipeline is driven by a pump_once execution loop. In production, a background worker thread (Task-6) continuously calls pump_once until the queue is empty.

  1. Claim: The loop claims the next pending task from the SqliteQueuesrc-tauri/src/pipeline/steps.rs#15-17
  2. Dispatch: The task is matched to a specific execution handler based on its stage name src-tauri/src/pipeline/steps.rs#14-17
  3. Telemetry: Before execution, step_started is emitted to the EventSink. After execution, the outcome is recorded in the vault as a Check note and step_done is emitted src-tauri/src/pipeline/steps.rs#65-70
  4. Handoff: On success, the handler typically enqueues the next task in the chain, passing its output as the next task's payload src-tauri/src/pipeline/steps.rs#1-5

Run Orchestration Overview

The following diagram bridges the logical pipeline concepts to the Rust functions and structures that implement them.

Title: Pipeline Run Orchestration

flowchart LR
    subgraph subGraph2 ["Step Chains"]
        D1["careers-scrape"]
        D2["structure-listings"]
        D3["finalize"]
        J1["jd-scrape"]
        J2["structure-jd"]
        J3["gap-detect"]
        J4["research-gaps"]
        S1["fit-score"]
        S2["alignment"]
    end
    subgraph subGraph1 ["Worker Thread"]
        P["pump_once()"]
        T["QueuedTask"]
        D["dispatch()"]
        V["Vault: checks/*.md"]
        S["EventSink (TauriSink)"]
    end
    subgraph subGraph0 ["Entrypoints (Tauri Commands)"]
        A["fetch_jobs_for_company()"]
        Q["SqliteQueue"]
        B["fetch_job_details()"]
        C["rescore_job()"]
    end
    P --> T
    P --> D
    D --> V
    D --> S
    D --> D1
    D1 --> D2
    D2 --> D3
    D --> J1
    J1 --> J2
    J2 --> J3
    J3 --> J4
    D --> S1
    S1 --> S2
    J3 --> S1
    A --> Q
    B --> Q
    C --> Q
Loading

Sources: src-tauri/src/pipeline/steps.rs#1-26src-tauri/src/worker.rs#91-112src-tauri/src/worker.rs#121-157

The Discovery Chain (job_check)

The discovery chain is used to find new job listings from a company's careers page. It is initiated via start_discoverysrc-tauri/src/pipeline/steps.rs#108

Step Function/Logic Data Flow
careers-scrape run_scrape_step Scrapes the careers_url. Returns sanitized HTML.
structure-listings build_structure_listings_prompt LLM extracts a list of RawListing objects from HTML.
finalize prefilter Filters listings by title/URL; writes new Job stubs to the vault.

Scrape Failure Policy:

Sources: src-tauri/src/pipeline/steps.rs#11-13src-tauri/src/pipeline/steps.rs#19-26src-tauri/src/pipeline/steps.rs#98-108

The Detail & Scoring Chains

When a user requests details for specific roles, the system executes two distinct runs: job_detail followed by an automatic handoff to job_scoring.

1. Detail Chain (job_detail)

This chain focuses on extracting the full context of a single job.

2. Scoring Chain (job_scoring)

Once the JD is structured, the pipeline transitions to analysis.

3. The Handoff

When the job_detail run completes successfully, the finalize_job_detail step automatically triggers start_rescore_runsrc-tauri/src/worker.rs#13-16 This creates a new run ID but maintains the same subject (the job slug), allowing the UI to show a continuous progress strip src-tauri/src/worker.rs#29-40

Title: Detail to Scoring Handoff

sequenceDiagram
    participant Q as SqliteQueue
    participant P as pump_once
    participant V as Vault (Job Note)
    participant E as EventSink
    P->>P: Execute gap-detect/research-gaps
    P->>V: update_job_field (Finalize Detail)
    Note over P: Detail Run Complete
    P->>Q: start_rescore_run(job_slug)
    Q-->>P: New Run ID (job_scoring)
    P->>E: run_finished(detail_run | "complete")
    P->>Q: claim_next(fit-score)
    P->>E: step_started(scoring_run | "fit-score")
Loading

Sources: src-tauri/src/pipeline/steps.rs#1-9src-tauri/src/worker.rs#31-40src/lib/pipeline.ts#5-17

Abort & Cancellation Logic

The system supports both manual and system-driven cancellations.

Sources: src-tauri/src/worker.rs#136-156src-tauri/src/pipeline/steps.rs#143-145src/lib/pipeline.ts#56-59

EventSink & Frontend Integration

The EventSink trait abstracts the delivery of live progress updates.

  • TauriSink: The production implementation. It uses app.emit to send run:step and run:finished events to the Svelte frontend src-tauri/src/worker.rs#44-47
  • StepEvent Payload: Includes run_id, subject (job or company slug), stage, and status. The detail field is used for sub-phase info like "stealth" proxy retries src-tauri/src/worker.rs#29-40
  • Frontend Display: The phaseLabel function in TypeScript maps these internal stage names to human-readable strings (e.g., structure-jd → "Reading the JD…") src/lib/pipeline.ts#92-114

Sources: src-tauri/src/pipeline/steps.rs#62-70src-tauri/src/worker.rs#48-89src/lib/pipeline.ts#62-84

Clone this wiki locally