Skip to content

One guarded fetch path, and it checks where it lands - #339

Merged
WaylandYang merged 2 commits into
devfrom
fix/one-guarded-fetch
Sep 5, 2026
Merged

One guarded fetch path, and it checks where it lands#339
WaylandYang merged 2 commits into
devfrom
fix/one-guarded-fetch

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

Closes #330.

ingest_sources.rs built a fresh reqwest::Client in six places, each with a timeout and a user agent and nothing else. None of them asked where the request was going, reqwest followed up to ten redirects on its own, and fetch_page ended in resp.bytes() with no cap.

Who chose the address decides how strict it is

  • Reach::Operator — a person with editor rights typed this host (url, rss, jira, custom, webdav). Private addresses stay reachable, because an internal wiki is what this product is for; sync_custom already had a loopback branch that turned the proxy off, and that judgement is now in one place instead of one source kind.
  • Reach::Content — the address came out of fetched content. Public only. Nothing uses it from a feed yet; api.github.com uses it because a fixed public host has no reason to be looser, and Add guarded full-content hydration for new RSS items #326's article fetching is the case it was written for.

Both get: one resolution whose every address is checked, resolve_to_addrs pinning those addresses so a second lookup cannot move the target, redirects walked by hand with each hop re-checked, no https→http downgrade, a byte cap on the body, and a deadline across the whole trip.

A redirect may not widen the reach: if a hop landed on a public address, the next one may not land inside the network. The operator chose that host, not its willingness to forward.

The address list

is_public_ip refuses loopback, private, link-local, CGNAT, benchmark, documentation, multicast and unspecified ranges, and unwraps v4-mapped, NAT64 and 6to4 addresses before judging what is inside. It deliberately does not treat all of 192.0.0.0/16 as reserved — only 192.0.0.0/24 and 192.0.2.0/24 are, and over-blocking is a site a user cannot reach.

Verified

Unit tests over the address table, both directions, including the wrapped forms (::ffff:127.0.0.1, 64:ff9b::7f00:1, 2002:7f00:1::) and the two addresses that must stay allowed inside otherwise-reserved space. Three tests against a real wiremock server: a loopback host is refused for Content and fetched for Operator — the same URL, and the whole distinction in one assertion; a body over the cap is refused where the old code read it all; a redirect loop ends at the limit instead of being followed by the library.

cargo fmt --check, cargo clippy --all-targets clean, 150 server tests green.

The shape comes from the fetcher @J-i-K wrote for #326 — this lifts it out to a path every source shares, which should shrink that PR rather than compete with it.

🤖 Generated with Claude Code

WaylandYang and others added 2 commits September 5, 2026 09:16
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <wayland0916@gmail.com>
@WaylandYang
WaylandYang merged commit db4f02d into dev Sep 5, 2026
4 checks passed
@WaylandYang
WaylandYang deleted the fix/one-guarded-fetch branch September 5, 2026 01:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Every fetch path builds its own client, and none of them checks where it lands

1 participant