Gap Detection & Research

Relevant source files

• src-tauri/src/pipeline/gaps.rs
• src-tauri/src/pipeline/steps.rs
• src-tauri/src/prompts.rs
• src/lib/job.ts

The Gap Detection & Research subsystem is responsible for identifying missing information in a Job entity after the initial Job Description (JD) structuring and attempting to fill those holes using LLM-driven web research. This ensures that the fit scoring engine has high-fidelity data even when the primary JD text is sparse or missing key details like compensation or visa sponsorship.

Gap Detection Logic

The gap-detect stage follows the structure-jd stage in the job_detail pipeline src-tauri/src/pipeline/steps.rs#8-12 It uses a whitelist of fields defined in RESEARCHABLE_FIELDS to determine what data points are valuable enough to warrant an external search src-tauri/src/pipeline/gaps.rs#10-26

RESEARCHABLE_FIELDS Whitelist

The system specifically looks for gaps in these categories:

• Compensation:comp_low, comp_high, comp_currency, comp_period, comp_equitysrc-tauri/src/pipeline/gaps.rs#11-15
• Arrangement:remote, location_constraints, visa_sponsorship, relocationsrc-tauri/src/pipeline/gaps.rs#16-19
• Role Details:employment_type, yoe_min, tech_stack, reports_to, team, countriessrc-tauri/src/pipeline/gaps.rs#20-25

The detect_gaps function iterates through this list and checks if the corresponding field in the Job struct is None (for scalars) or empty (for vectors) src-tauri/src/pipeline/gaps.rs#30-38

Sources:src-tauri/src/pipeline/gaps.rs#1-60src-tauri/src/pipeline/steps.rs#137-142

Research Workflow & Validation

If gaps are detected, the pipeline enqueues a research-gaps task src-tauri/src/pipeline/steps.rs#442-446 This task utilizes an LLM with web-search capabilities to find the missing information.

Research Validation State Machine

The results from the LLM are not trusted blindly. They must pass through parse_and_validate_research, which enforces strict typing and value constraints src-tauri/src/prompts.rs#46-49

Entity	Role	Logic
ResearchedField	Successful Data	Contains the field name, the value (as a TypedValue), and the source URL src-tauri/src/prompts.rs#163-168
Rejection	Failed Data	Contains the field and a reason why it couldn't be found (e.g., "Not in JD or public site") src-tauri/src/prompts.rs#171-175
TypedValue	Type Safety	An enum ensuring values match expected types: String, Number (i64), or Array (Vec) src-tauri/src/prompts.rs#155-160

Sources:src-tauri/src/prompts.rs#155-175src-tauri/src/pipeline/steps.rs#462-475

Data Flow: From Gap to Vault

The following diagram illustrates how the system transitions from a structured JD with missing fields to a fully researched Job note.

Gap Detection to Research Data Flow

flowchart TD
    JOB_NOTE["Job Markdown File"]
    subgraph subGraph1 ["Code Entity Space"]
        JOB_STRUCT["Job (Rust Struct)"]
        GAP_DET["detect_gaps()"]
        PROMPT_BLDR["build_research_gaps_prompt()"]
        VALIDATOR["parse_and_validate_research()"]
        VAULT_WRITER["update_job_field() / set_job_list_field()"]
    end
    subgraph subGraph0 ["Natural Language Space"]
        WEB["Web / Google Search"]
        JD_TEXT["Raw JD Text"]
    end
    JD_TEXT --> JOB_STRUCT
    JOB_STRUCT --> GAP_DET
    GAP_DET --> PROMPT_BLDR
    PROMPT_BLDR --> WEB
    WEB --> VALIDATOR
    VALIDATOR --> VAULT_WRITER
    VAULT_WRITER --> JOB_NOTE

Loading

Sources:src-tauri/src/pipeline/gaps.rs#30-38src-tauri/src/pipeline/steps.rs#454-480src-tauri/src/prompts.rs#188-210

Pipeline Step Integration

The research logic is encapsulated in the research-gaps step within src-tauri/src/pipeline/steps.rs.

1. Payload Preparation: The ResearchGapsPayload carries the slug and the list of gaps identified by detect_gapssrc-tauri/src/pipeline/steps.rs#138-142
2. LLM Execution: The system calls build_research_gaps_prompt, which instructs the LLM to use its web tool to find the specific missing fields src-tauri/src/prompts.rs#188-210
3. Validation & Persistence:

• The LLM response is parsed into ResearchedWritesrc-tauri/src/prompts.rs#178-182
• Valid fields are written to the job note using update_job_field or set_job_list_fieldsrc-tauri/src/pipeline/steps.rs#493-518
• The names of the successfully researched fields are added to the researched metadata list on the Job note to track provenance src-tauri/src/pipeline/steps.rs#524-529

Step Execution Logic

sequenceDiagram
    participant Q as SqliteQueue
    participant S as steps.rs (research-gaps)
    participant L as OpenRouterLlm
    participant V as Vault (Markdown)
    Q->>S: Claim Task (ResearchGapsPayload)
    S->>L: Web Search + LLM Inference
    L-->>S: Raw JSON Response
    S->>S: parse_and_validate_research()
    S->>V: update_job_field()
    S->>V: set_job_list_field(field: "researched")
    S->>Q: Mark Task Done

Loading

Sources:src-tauri/src/pipeline/steps.rs#454-535src-tauri/src/prompts.rs#213-240