scrape: raise planet feed timeout, add User-Agent and retry - #3752
Merged
Conversation
Planet scraping fails intermittently because river's built-in HTTP timeout is 3s, which is too aggressive for many feeds, and some hosts reject requests with a default/empty User-Agent. Pass an explicit 30s ?timeout and a descriptive ?user_agent to River.fetch/seo_image/meta_description (added by tarides/river#16), and wrap the feed fetch in an exponential-backoff retry (3 attempts) to absorb transient network/DNS/5xx failures. River is temporarily pinned to the river#16 head via pin-depends; this should be replaced by a version constraint once river >= 0.5 is released to opam. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
The 5 residual (non-timeout) scrape failures noted above are tracked separately in #3753 — dead/moved feeds that this PR intentionally doesn't touch. |
PR ocaml#16 is merged. Repoint the temporary pin from the fork PR-head SHA to tarides/river master (28281fe, "Prepare release 0.5"), which carries the ?timeout/?user_agent API and drops the stale, unused ocamlnet dependency that made the intermediate ocaml#16 merge commit unsolvable. Replace with `"river" {>= "0.5"}` once river 0.5 is published to opam. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cuihtlauac
force-pushed
the
planet-scrape-timeout
branch
from
August 25, 2026 14:02
1e8303f to
4801f03
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
OCaml Planet scraping intermittently drops feeds because river's HTTP client
has a hardcoded 3s timeout and sends no descriptive User-Agent. Slow-but-alive
feeds get cut off, and transient network/DNS/5xx blips kill a source for the
whole run (the scraper catches the exception and records the source as an
error).
Change
tool/data-scrape/lib/blog_scraper.ml:?timeoutand a descriptive?user_agent(
ocaml.org planet aggregator (+https://ocaml.org)) toRiver.fetch/River.seo_image/River.meta_description. These optionalparameters were added upstream in Expose timeout and user-agent in public API tarides/river#16.
fetch_with_retry— 3 attempts with exponentialbackoff (2s, 4s) — so a transient failure no longer drops the source; the
final attempt still falls through to the existing per-source error handler.
tool/data-scrape/lib/dune: addunix(forUnix.sleepf).ocamlorg.opam: apin-dependson tarides/river master (which carries #16plus the 0.5 release prep).
Why a git pin and not a version constraint
ocaml.org builds against a frozen opam-repository snapshot (pinned to
584630e7inMakefile,Dockerfile, and the CI workflows includingrelease-scrapers.yml). A"river" {>= "0.5"}constraint would be unsolvablethere even after river 0.5 is published upstream, because that snapshot doesn't
contain it.
opamhonourspin-dependsregardless of the snapshot, andrelease-scrapers.ymlrunsopam install --deps-only ., so the git pin is themechanism that actually delivers the fix to the production scraper. The pin can
be dropped (and replaced by a version constraint) only once the
opam-repository pin is bumped to a commit that includes river 0.5 — see the
inline comment in
ocamlorg.opam.Validation
Upstream (river#16): merged, and verified with a self-contained
backward-compatibility harness (in-process HTTP server, no network) — unchanged
call sites still compile, the 3s default is preserved,
?timeoutcontrols thebound both ways, and
?user_agentis sent verbatim.This change (live A/B over 74 active sources):
The one timeout-class failure (
psellos,Http.Timeoutat 3s — responds in<30s) recovered. The 5 residual errors are orthogonal and un-fixable by a
timeout change: 3 malformed/non-feed sources, 1 DNS-dead host, 1 hard 404 —
tracked separately in #3753. Timeouts are stochastic, so this run is a floor on
the benefit; the retry additionally absorbs transient failures that didn't
happen to occur in this window. Cost: the run takes longer (~100s vs ~27s)
because dead feeds now wait out the 30s×retries instead of bailing at 3s —
acceptable for the nightly scrape.
🤖 Generated with Claude Code