feat(extraction): opt-in rs-trafilatura candidate (TRAWL_RS_TRAF)#83
Merged
Conversation
trawl now ships its own docker-compose.yml and scripts/deploy.sh so it can run independently of any single consumer and serve multiple MCP clients from one instance. Publishes 127.0.0.1:8765 and mounts ~/.trawl; LLM endpoints default to host.docker.internal llama-servers and are overridable via a .env file. HyDE stays off by default; its env (TRAWL_HYDE_URL) is left blank with a note that a relayed/offloaded utility LLM may live on a different port than the embedding/rerank servers. README documents the HTTP/compose deployment path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wires the already-shipped TRAWL_VLM_SLOT env var into the standalone docker-compose service (default slot 3) to avoid evicting other consumers' KV cache on the shared :8080 llama-server. TRAWL_HYDE_SLOT left as a commented placeholder since HyDE is off by default.
Scores rs-trafilatura (rs_text=main_content, rs_md=html_to_markdown of content_html) against a fresh Python-Trafilatura baseline (run.py's _TRAF_KWARGS) on the WCXB dev split with the vendored word-F1 evaluator. 2026-07-03 full-split result (trafilatura 2.0.0, rs-trafilatura 0.1.1): rs_text F1 0.844 / 0 fail / 10ms median vs baseline 0.750 / 15ms — +0.094, wins all 7 page types (forum +0.225, collection +0.202). Fresh baseline reproduces the recorded 0.750 exactly, confirming the 2026-04 number was config-consistent. Details in notes/rs-trafilatura-verification-outcome.md (local). Note: ExtractResult.content_markdown is always None in rs-trafilatura 0.1.1; the rs_md variant derives markdown from content_html instead.
Adds rs-trafilatura as an optional 5th entrant in extract_html()'s score-based candidate selection, gated behind TRAWL_RS_TRAF (default off, accepts 1/true/yes/on). The module is lazily imported and fully optional: a missing or failing package logs one debug line and the candidate is skipped permanently. Candidate output is html_to_markdown(extract(html).content_html) — the markdown route keeps headings/tables for the chunker. Appended (not inserted) so the newest candidate loses exact-score ties to established ones. Measurement (2026-07-03, rs-trafilatura 0.1.1, trafilatura 2.0.0): - WCXB dev split 1,497 pages: flag ON combined F1 0.809 vs recorded 3-way 0.777 (+0.032); rs markdown-variant alone 0.848 over-ok / 0.814 over-all vs same-session Trafilatura baseline 0.750. - Parity matrix 15/15 flag OFF and flag ON (fetch cache disabled for the ON run — cached extractions would otherwise mask the candidate). - agent_patterns coding shard 24/24 flag ON. - Missing-package behavior verified: extraction falls through to the existing candidates without error. Default stays OFF pending a default-flip decision. Also: gitignore benchmarks/wcxb/results/, .env.example entry, CHANGELOG Unreleased entry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds rs-trafilatura (Rust extractor, PyO3 bindings) as an optional entrant in
extract_html()'s score-based candidate selection, gated behindTRAWL_RS_TRAF(default OFF, accepts 1/true/yes/on). Lazy import, fully optional dependency: missing/failing package logs one debug line and the candidate is skipped permanently. Output route ishtml_to_markdown(extract(html).content_html)to keep headings/tables for the chunker.Gates (pre-registered, 2026-07-03)
Verification spike (full WCXB dev split, same harness/evaluator): rs markdown-variant F1 0.848 over-ok / 0.814 over-all vs same-session Trafilatura 2.0.0 baseline 0.750; rs wins all 7 page types. Details: notes/rs-trafilatura-verification-outcome.md (local).
Review
/code-review medium: 8 candidates -> 3 survived. Fixed: env-flag accepted set widened to match sibling flags; insert(3) magic index replaced with append + tie-break comment. Kept as-is (documented trade-off): module-global lazy loader is heavier than _readability()'s inline pattern but buys log-once and avoids re-running import machinery per page when the package is absent.
Not in this PR
_score_candidate.🤖 Generated with Claude Code