Skip to content

Discovery Prefilter

Jump to bottom
Chazona Baum edited this page Jun 24, 2026 · 1 revision

Relevant source files

The Discovery Prefilter is a high-recall, low-precision filtering stage within the job-fetch pipeline. It acts as the first gate for RawListing objects extracted during the discovery phase, ensuring that the system only processes jobs that are broadly relevant to the user's career interests while aggressively deduplicating to save on downstream LLM and API costs.

Purpose and Scope

The prefilter is implemented in src-tauri/src/pipeline/filter.rssrc-tauri/src/pipeline/filter.rs#1-5 Its primary goal is to narrow down the flood of raw scraped data to a set of "on-target" candidates.

Key characteristics of this stage include:

  • Recall-Oriented: It uses case-insensitive title matching to include any job that might be a fit.
  • Global Deduplication: It checks listings against URLs already persisted in the vault to avoid re-processing known roles.
  • Batch Deduplication: It handles pagination overlaps or double-parsing by collapsing duplicate URLs within the same scrape batch.
  • Intentional Permissiveness: It specifically omits filtering by work arrangement (e.g., Remote vs. Onsite) or compensation at this stage. These factors are handled by the soft-scoring fit engine later, allowing the user to decide if a high-quality role is worth a compromise src-tauri/src/pipeline/filter.rs#2-5

Data Flow and Code Entities

The prefilter consumes RawListing objects and TargetCriteria parsed from the user's vault.

Logic Flow: Discovery to Prefilter

The following diagram illustrates how raw scraped data moves from the natural language space of job boards into the structured RawListing struct and through the prefilter logic.

Prefilter Logic and Entity Mapping

flowchart LR
    KEEP["Include in Pipeline"]
    DROP["Discard"]
    subgraph subGraph2 ["Vault Configuration: profile/target_criteria.md"]
        TC["struct TargetCriteria"]
        MT["match_titles: #91;'ai engineer', 'software engineer'#93;"]
    end
    subgraph subGraph1 ["Code Entity Space: src-tauri/src/pipeline/filter.rs"]
        RL["struct RawListing"]
        RL_A["RawListing { title: 'Senior AI Engineer', ... }"]
        RL_B["RawListing { title: 'Real Estate Agent', ... }"]
        PF["fn prefilter()"]
        TM["fn title_matches()"]
    end
    subgraph subGraph0 ["Natural Language Space (Scraped Data)"]
        A["HTML Listing: 'Senior AI Engineer'"]
        B["HTML Listing: 'Real Estate Agent'"]
    end
    A --> RL_A
    B --> RL_B
    RL_A --> PF
    RL_B --> PF
    TC --> PF
    MT -.-> TM
    PF --> TM
    TM --> KEEP
    TM --> DROP
Loading

Sources: src-tauri/src/pipeline/filter.rs#12-43src-tauri/src/profile.rs#37-56

Key Functions

title_matches

This function performs a case-insensitive check to see if the job title contains any of the strings defined in the user's match_titles list src-tauri/src/pipeline/filter.rs#20-23

prefilter

The main entry point for filtering a batch of listings. It applies a chain of filters in a specific order to preserve input sequence while cleaning the data src-tauri/src/pipeline/filter.rs#31-43

Filter Step Logic Purpose
Global Dedup !existing_urls.contains(&l.url) Skip jobs already saved in the vault src-tauri/src/pipeline/filter.rs#39
Batch Dedup seen.insert(l.url.clone()) Ensure a URL only appears once in the current run src-tauri/src/pipeline/filter.rs#40
Title Match title_matches(&l.title, ...) Drop clear non-matches (e.g., "Nurse" for a "Developer" search) src-tauri/src/pipeline/filter.rs#41

Sources: src-tauri/src/pipeline/filter.rs#31-43

Deduplication Strategy

Lodestar treats the URL as the unique identity of a job posting src-tauri/src/pipeline/filter.rs#25-30

The deduplication happens in two layers:

  1. Cross-Run Deduplication: The pipeline provides a HashSet<String> of existing_urls pulled from the vault's current state src-tauri/src/pipeline/filter.rs#34
  2. Within-Batch Deduplication: Using a local HashSet named seen, the function ensures that if a scraper encounters the same URL twice (common in infinite-scroll or paginated results), only the first occurrence is kept src-tauri/src/pipeline/filter.rs#36-40

Entity Relationship: URL Identity

flowchart LR
    RL["RawListing"]
    subgraph subGraph1 ["Pipeline Execution"]
        BATCH["Vec"]
        SEEN["seen (HashSet)"]
    end
    subgraph subGraph0 ["Vault (Persistence)"]
        V["Job Notes (*.md)"]
        EU["existing_urls (HashSet)"]
    end
    BATCH --> EU
    BATCH --> SEEN
    SEEN --> RL
Loading

Sources: src-tauri/src/pipeline/filter.rs#25-43

Intentional Omissions

A critical design decision in filter.rs is the exclusion of work-arrangement filtering. Even if a user's TargetCriteria specifies "Remote" only, the prefilter will allow "Onsite" or "Hybrid" roles to pass through if the title matches src-tauri/src/pipeline/filter.rs#2-5

This is handled later by the arrangement_fit score in the scoring engine. This prevents the system from being too aggressive—if a "Dream Company" has an onsite role, the user should see it and decide if the location is a dealbreaker, rather than the system silently discarding it at the discovery stage src-tauri/src/pipeline/filter.rs#28-30

Sources: src-tauri/src/pipeline/filter.rs#1-5src-tauri/src/profile.rs#3-5

Clone this wiki locally