Scraper & HTML Sanitization

Relevant source files

• src-tauri/src/llm.rs
• src-tauri/src/pipeline/steps.rs
• src-tauri/src/sanitize.rs
• src-tauri/src/scraper.rs

This subsystem provides the bridge between the public internet and the internal pipeline by fetching raw HTML from careers pages or job descriptions and transforming it into a clean, LLM-ready text format. It handles the complexities of bot-detection, credit-based cost accounting, and defense-in-depth against prompt injection.

The Scraper Trait & Implementations

The Scraper trait src-tauri/src/scraper.rs#105-109 defines a uniform interface for fetching content. While FakeScraper is used for unit testing, the production implementation is ScrapingBeeScrapersrc-tauri/src/scraper.rs#135

ScrapingBee Integration

The scraper utilizes the ScrapingBee API to handle Javascript rendering and proxy rotation. Key features include:

• Proxy Tiers: Supports Premium (25 credits) and Stealth (75 credits) src-tauri/src/scraper.rs#36-43
• Cost Tracking: Credits consumed are extracted from the Spb-cost response header and returned via ScrapeResultsrc-tauri/src/scraper.rs#97-103
• Timeouts: Configured with a 90-second timeout src-tauri/src/scraper.rs#146-148 to accommodate slow ATS renders (e.g., Workday, Greenhouse).
• Percent Encoding: The percent_encode_target_url function src-tauri/src/scraper.rs#122-124 ensures target URLs are RFC-3986 compliant, preventing query parameters from leaking into ScrapingBee's own parameter parser.

Failure Classification

Scrape results are passed through classify_scrape_failuresrc-tauri/src/scraper.rs#76-94 which maps HTTP status codes and body content to a FailureClasssrc-tauri/src/scraper.rs#21-33:

FailureClass	Logic	Action
Terminal	404/410	Abort run; page is gone.
FixEncoding	500 + "Unknown arguments"	Re-issue with RFC-3986 encoding.
EscalateProxy	403 or generic 500	Retry once using ProxyTier::Stealth.
Transient	429, 503, or other	Bounded backoff retry (max 2 attempts).

Sources:src-tauri/src/scraper.rs#21-152

HTML Sanitization Layer

The sanitize.rs module provides a pure-function gate src-tauri/src/sanitize.rs#17-59 that converts raw HTML into "clean" text. This serves two purposes: reducing token waste and preventing LLM "jailbreaks" embedded in invisible HTML attributes.

Sanitization Process

1. DOM Walk: Uses scraper (html5ever) to walk the document tree src-tauri/src/sanitize.rs#21
2. Pruning: Subtrees for <script> and <style> are entirely removed src-tauri/src/sanitize.rs#29-30
3. Visibility Check: Elements with hidden, aria-hidden="true", or display:none/visibility:hidden styles are pruned src-tauri/src/sanitize.rs#61-73
4. Link Resolution: <a> tags have their href resolved to absolute URLs against the base_urlsrc-tauri/src/sanitize.rs#103-113 This prevents the LLM from fabricating relative paths src-tauri/src/sanitize.rs#35-43
5. Normalization: Whitespace is collapsed, and zero-width characters (e.g., \u{200b}) are stripped src-tauri/src/sanitize.rs#93-97
6. Fencing: The output is wrapped in <<<SCRAPED_DATA>>> and <<<END_SCRAPED_DATA>>> delimiters src-tauri/src/sanitize.rs#8-9

Sources:src-tauri/src/sanitize.rs#1-122

Data Flow & Entity Mapping

The following diagrams illustrate how raw web content moves from the internet into the Lodestar pipeline.

Scrape Execution Flow

This diagram shows the relationship between the Queue worker, the Scraper trait, and the failure classification logic.

flowchart LR
    subgraph subGraph2 ["Failure Handling"]
        F["scraper::classify_scrape_failure"]
        G["FailureClass"]
        H["Retry with percent_encode_target_url"]
        I["Retry with ProxyTier::Stealth"]
        J["Mark Task Dead"]
    end
    subgraph subGraph1 ["Scraper Entity Space"]
        C["Scraper::fetch"]
        D["ScrapingBeeScraper"]
        E["ScrapingBee API"]
    end
    subgraph subGraph0 ["Pipeline Runner"]
        A["steps::pump_once"]
        B["runner::run_scrape_step"]
    end
    A --> B
    B --> C
    C --> D
    D --> E
    E --> D
    D --> B
    B --> F
    F --> G
    G --> H
    G --> I
    G --> J

Loading

Sources:src-tauri/src/pipeline/steps.rs#14-51src-tauri/src/scraper.rs#76-138

Content Sanitization & LLM Preparation

This diagram bridges the raw HTML data to the sanitized text used by the LLM prompts.

flowchart TD
    subgraph subGraph2 ["Output Entity Space"]
        H["Fenced Text"]
        I["SANITIZED_OPEN / SANITIZED_CLOSE"]
        J["prompts::build_structure_listings_prompt"]
    end
    subgraph subGraph1 ["sanitize.rs Logic"]
        C["scraper::Html::parse_document"]
        D["DOM Walk"]
        E["script/style/hidden nodes"]
        F["resolve_href(a.href)"]
        G["strip_zero_width"]
    end
    subgraph subGraph0 ["Input Space"]
        A["Raw HTML String"]
        B["Base URL"]
    end
    A --> C
    C --> D
    D --> E
    D --> F
    D --> G
    G --> H
    H --> I
    I --> J

Loading

Sources:src-tauri/src/sanitize.rs#8-59src-tauri/src/pipeline/steps.rs#44-50

Implementation Details

Pipeline Integration

The scraper is invoked within the careers-scrape and jd-scrape steps src-tauri/src/pipeline/steps.rs#11-12 If a scrape fails with EscalateProxy, the task is re-enqueued with a ScrapePayload where tier is set to "stealth"src-tauri/src/pipeline/steps.rs#98-118

Runtime Constraints

The ScrapingBeeScraper uses reqwest::blocking. Because reqwest::blocking can panic if executed directly on a tokio runtime thread, the pipeline runner ensures these steps are executed on a dedicated worker thread (Task 6) or via a sync command src-tauri/src/scraper.rs#132-134

Sources:src-tauri/src/pipeline/steps.rs#1-42src-tauri/src/scraper.rs#132-138