fix(websearch): harden SSRF guard in ContentFetcher (#383) - #429
Conversation
Three vectors were unguarded: 1. Only loopback addresses were blocked — private RFC 1918 ranges, link-local (169.254.x.x), CGNAT (100.64.x.x), and other reserved networks were passed through. 2. Decimal-integer-encoded IPv4 addresses (e.g. 2130706433 == 127.0.0.1) were not caught because ipaddress.ip_address() only parses dotted decimal; the integer form is parsed by a second int() coercion. 3. follow_redirects=True meant a legitimate initial URL could silently redirect to a private address after the initial check passed. Fix: replace _is_loopback_url with _is_safe_url (module-level) that checks all private/reserved/non-global flags explicitly, handles decimal-integer IPs, and blocks non-HTTP(S) schemes. Redirect following is now manual (follow_redirects=False) so every hop is validated before the next request is sent. Tests: 39 cases covering private ranges, decimal encoding, non-HTTP schemes, redirect-to-private-IP, safe redirect following, and redirect chain length cap.
📝 WalkthroughWalkthroughThis PR hardens the ChangesSSRF Hardening with Manual Redirect Validation
Sequence DiagramsequenceDiagram
participant Client
participant ContentFetcher
participant SafetyCheck
participant TargetServer
Client->>ContentFetcher: fetch URL
loop Redirect loop
ContentFetcher->>SafetyCheck: _is_safe_url(current_url)
SafetyCheck-->>ContentFetcher: valid?
alt URL unsafe
ContentFetcher-->>Client: return None
end
ContentFetcher->>TargetServer: GET (follow_redirects=False)
TargetServer-->>ContentFetcher: response
alt Redirect (301/302/etc)
ContentFetcher->>ContentFetcher: update current_url from Location
alt Too many redirects
ContentFetcher-->>Client: return None
end
else Non-redirect
ContentFetcher-->>Client: process & return content
end
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
What was wrong
What was changed
Tests39 cases: all private/reserved address ranges, decimal-integer encoding, non-HTTP schemes, redirect-to-private-IP (redirect target never contacted), safe redirect chains, and redirect chain length cap. Note on DNS rebindingFull DNS rebinding protection (where a hostname resolves to a public IP at check time then to a private IP at connect time) requires a custom transport layer and is out of scope here. The per-hop redirect validation and IP-literal checks cover the practical attack vectors from a compromised search provider. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
openrag/components/websearch/test_content_fetcher.py (1)
66-68: ⚡ Quick winAvoid asserting that any regular hostname is safe.
This test encodes the current bypass behavior. Please replace/extend it with a DNS-resolution-aware case (e.g., mocked
getaddrinfo) to assert that hostnames resolving to blocked ranges are rejected.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@openrag/components/websearch/test_content_fetcher.py` around lines 66 - 68, The test test_allows_regular_hostname currently asserts _is_safe_url("https://example.com/page") is True without checking DNS resolution; update it to be DNS-resolution aware by mocking socket.getaddrinfo (or the resolver used by _is_safe_url) and add two assertions: one where getaddrinfo returns an address in an allowed/public range and assert _is_safe_url(...) is True, and another where getaddrinfo returns an IP in a blocked/private range and assert _is_safe_url(...) is False; keep references to the _is_safe_url function and the test_allows_regular_hostname test when implementing the mocks so the change targets the correct logic.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@openrag/components/websearch/content_fetcher.py`:
- Around line 84-85: The hostname-only branch in _is_safe_url currently returns
True without DNS resolution; change it to perform A and AAAA lookups for the
parsed hostname and reject if any resolved IP falls into blocked ranges
(localhost 127.0.0.0/8, IPv6 loopback, private RFC1918 ranges, link-local,
etc.); do this check per redirect hop (same place where redirects are
re-validated) and only return True after all DNS-resolved addresses pass the
existing ip-in-blocklist tests used elsewhere in _is_safe_url.
---
Nitpick comments:
In `@openrag/components/websearch/test_content_fetcher.py`:
- Around line 66-68: The test test_allows_regular_hostname currently asserts
_is_safe_url("https://example.com/page") is True without checking DNS
resolution; update it to be DNS-resolution aware by mocking socket.getaddrinfo
(or the resolver used by _is_safe_url) and add two assertions: one where
getaddrinfo returns an address in an allowed/public range and assert
_is_safe_url(...) is True, and another where getaddrinfo returns an IP in a
blocked/private range and assert _is_safe_url(...) is False; keep references to
the _is_safe_url function and the test_allows_regular_hostname test when
implementing the mocks so the change targets the correct logic.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 6218a3c8-41cf-4445-8925-af4a2bf51971
📒 Files selected for processing (2)
openrag/components/websearch/content_fetcher.pyopenrag/components/websearch/test_content_fetcher.py
| # Regular hostname — passes initial check; every redirect hop is re-validated. | ||
| return True |
There was a problem hiding this comment.
Resolve hostnames before treating them as SSRF-safe.
On Line 84, regular hostnames are accepted without checking what they resolve to. A malicious/misbehaving provider can return a hostname that DNS-resolves to 127.0.0.1, 10.0.0.0/8, link-local, or other internal ranges, which bypasses this guard.
Please resolve A/AAAA records and reject if any resolved IP is blocked (and keep this check per redirect hop).
Suggested direction
+import socket
+
+def _hostname_resolves_to_blocked_ip(host: str) -> bool:
+ try:
+ infos = socket.getaddrinfo(host, None, proto=socket.IPPROTO_TCP)
+ except socket.gaierror:
+ return True # fail closed for untrusted external URLs
+
+ for family, _, _, _, sockaddr in infos:
+ ip_str = sockaddr[0]
+ if _is_blocked_address(ipaddress.ip_address(ip_str)):
+ return True
+ return FalseThen use this before the final return True path in _is_safe_url.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@openrag/components/websearch/content_fetcher.py` around lines 84 - 85, The
hostname-only branch in _is_safe_url currently returns True without DNS
resolution; change it to perform A and AAAA lookups for the parsed hostname and
reject if any resolved IP falls into blocked ranges (localhost 127.0.0.0/8, IPv6
loopback, private RFC1918 ranges, link-local, etc.); do this check per redirect
hop (same place where redirects are re-validated) and only return True after all
DNS-resolved addresses pass the existing ip-in-blocklist tests used elsewhere in
_is_safe_url.
2 participants
Why
The web search fetcher fetches URLs returned by an external provider. A compromised or misbehaving provider could return URLs pointing at internal infrastructure. Three vectors were unguarded.
What was wrong
2130706433==127.0.0.1) were not caught becauseipaddress.ip_address()only accepts dotted-decimal notation.follow_redirects=Truemeant a legitimate initial URL could silently redirect to a private address after the initial SSRF check passed.What changed
_is_loopback_urlreplaced by_is_safe_url(module-level), which checks all private/reserved/non-global IP flags explicitly, handles decimal-integer IPv4 encoding, and rejects non-HTTP(S) schemes.follow_redirects=False): every redirect target is validated by_is_safe_urlbefore the next request is sent, with a hard cap of 10 hops.Tests
39 cases covering: all private/reserved ranges, decimal-integer encoding, non-HTTP schemes, redirect-to-private-IP (redirect target never contacted), safe redirect chains, and redirect limit.
Summary by CodeRabbit