Skip to content

scrape: raise planet feed timeout, add User-Agent and retry - #3752

Merged
cuihtlauac merged 2 commits into
ocaml:mainfrom
cuihtlauac:planet-scrape-timeout
Aug 25, 2026
Merged

scrape: raise planet feed timeout, add User-Agent and retry#3752
cuihtlauac merged 2 commits into
ocaml:mainfrom
cuihtlauac:planet-scrape-timeout

Conversation

@cuihtlauac

@cuihtlauac cuihtlauac commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Problem

OCaml Planet scraping intermittently drops feeds because river's HTTP client
has a hardcoded 3s timeout and sends no descriptive User-Agent. Slow-but-alive
feeds get cut off, and transient network/DNS/5xx blips kill a source for the
whole run (the scraper catches the exception and records the source as an
error).

Change

tool/data-scrape/lib/blog_scraper.ml:

  • Pass an explicit 30s ?timeout and a descriptive ?user_agent
    (ocaml.org planet aggregator (+https://ocaml.org)) to
    River.fetch / River.seo_image / River.meta_description. These optional
    parameters were added upstream in Expose timeout and user-agent in public API tarides/river#16.
  • Wrap the feed fetch in fetch_with_retry — 3 attempts with exponential
    backoff (2s, 4s) — so a transient failure no longer drops the source; the
    final attempt still falls through to the existing per-source error handler.
  • tool/data-scrape/lib/dune: add unix (for Unix.sleepf).

ocamlorg.opam: a pin-depends on tarides/river master (which carries #16
plus the 0.5 release prep).

Why a git pin and not a version constraint

ocaml.org builds against a frozen opam-repository snapshot (pinned to
584630e7 in Makefile, Dockerfile, and the CI workflows including
release-scrapers.yml). A "river" {>= "0.5"} constraint would be unsolvable
there even after river 0.5 is published upstream, because that snapshot doesn't
contain it. opam honours pin-depends regardless of the snapshot, and
release-scrapers.yml runs opam install --deps-only ., so the git pin is the
mechanism that actually delivers the fix to the production scraper. The pin can
be dropped (and replaced by a version constraint) only once the
opam-repository pin is bumped to a commit that includes river 0.5 — see the
inline comment in ocamlorg.opam.

Validation

Upstream (river#16): merged, and verified with a self-contained
backward-compatibility harness (in-process HTTP server, no network) — unchanged
call sites still compile, the 3s default is preserved, ?timeout controls the
bound both ways, and ?user_agent is sent verbatim.

This change (live A/B over 74 active sources):

errors
before (3s, no retry/UA) 6
after (30s + retry + UA) 5

The one timeout-class failure (psellos, Http.Timeout at 3s — responds in
<30s) recovered. The 5 residual errors are orthogonal and un-fixable by a
timeout change: 3 malformed/non-feed sources, 1 DNS-dead host, 1 hard 404 —
tracked separately in #3753. Timeouts are stochastic, so this run is a floor on
the benefit; the retry additionally absorbs transient failures that didn't
happen to occur in this window. Cost: the run takes longer (~100s vs ~27s)
because dead feeds now wait out the 30s×retries instead of bailing at 3s —
acceptable for the nightly scrape.

🤖 Generated with Claude Code

Planet scraping fails intermittently because river's built-in HTTP
timeout is 3s, which is too aggressive for many feeds, and some hosts
reject requests with a default/empty User-Agent.

Pass an explicit 30s ?timeout and a descriptive ?user_agent to
River.fetch/seo_image/meta_description (added by tarides/river#16), and
wrap the feed fetch in an exponential-backoff retry (3 attempts) to
absorb transient network/DNS/5xx failures.

River is temporarily pinned to the river#16 head via pin-depends; this
should be replaced by a version constraint once river >= 0.5 is
released to opam.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@cuihtlauac

Copy link
Copy Markdown
Collaborator Author

The 5 residual (non-timeout) scrape failures noted above are tracked separately in #3753 — dead/moved feeds that this PR intentionally doesn't touch.

PR ocaml#16 is merged. Repoint the temporary pin from the fork PR-head SHA
to tarides/river master (28281fe, "Prepare release 0.5"), which carries
the ?timeout/?user_agent API and drops the stale, unused ocamlnet
dependency that made the intermediate ocaml#16 merge commit unsolvable.

Replace with `"river" {>= "0.5"}` once river 0.5 is published to opam.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@cuihtlauac
cuihtlauac force-pushed the planet-scrape-timeout branch from 1e8303f to 4801f03 Compare August 25, 2026 14:02
@cuihtlauac
cuihtlauac merged commit 965e726 into ocaml:main Aug 25, 2026
4 checks passed
@cuihtlauac
cuihtlauac deleted the planet-scrape-timeout branch August 25, 2026 16:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant