web-fetch-http SSRF guard rejects every fetch under transparent-proxy fake-ip DNS (198.18.0.0/15) - no resolver/allowlist escape hatch

[eddiedon]

Description

Title: web-fetch-http SSRF guard rejects every fetch under transparent-proxy fake-ip DNS (198.18.0.0/15) — no resolver/allowlist escape hatch

Summary

Since v0.1.2, the new default public fetch backend (@deepseek-ai/dsh-web-fetch-http) rejects all fetches to foreign domains when DSH runs on a network whose DNS answers with fake-ip addresses (198.18.0.0/15) — the standard DNS mode of transparent proxies (OpenClash / Clash / mihomo TUN), which is an extremely common home/office deployment in China. The domain resolves via the system resolver to a reserved-range IP, the SSRF guard correctly classifies it as non-public, and the whole answer set is rejected (WEB_BLOCKED_URL). There is no configuration to point the provider at a different resolver or to allowlist specific domains, so the shipped web_fetch tool degrades to "domestic direct domains only".

Environment

• DeepSeek Harness 0.1.2-alpha.2 (Windows 11, host-side — unrelated to the file sandbox)
• Router runs OpenClash in fake-ip mode (hijacks UDP/53); workstation DNS → router
• Resolve-DnsName example.com → 198.18.21.38, github.com → 198.18.0.7 (fake-ip range), while CN-direct domains (e.g. www.baidu.com → 180.101.51.73) resolve to real public IPs

Reproduction

web_fetch({ url: "https://example.com" })
→ Error: URL hostname "example.com" resolves to a non-public IP address (WEB_BLOCKED_URL)

web_fetch({ url: "https://www.baidu.com" })   // real-IP domain → works
→ HTTP 200

Any tool bypassing the guard (e.g. Node fetch inside the DSH sandbox) reaches the same URLs fine — the network path is healthy; only the guard's view of DNS is "poisoned" by fake-ip.

Root cause (from 0.1.2-alpha.2 sources)

@deepseek-ai/dsh-web-fetch-http (src/network.ts, built lib/index.js ~L54-78):

1. Resolution uses the system resolver: lookup(hostname, { all: true, order: "verbatim" }) from node:dns/promises (getaddrinfo) → returns the fake-ip.
2. isPublicIpAddress() = ipaddr.js range() === "unicast"; 198.18.0.0/15 (RFC 2544 benchmarking, used by Clash-family fake-ip) classifies as reserved → rejected.
3. Whole-answer-set rejection: if any resolved address is non-public, the entire result is rejected and the connection is pinned to the validated set — a sound anti-DNS-rebinding design.
4. Config surface has no escape hatch: no custom resolver / DoH option, no per-domain allowlist, no allowPrivate. README documents the caps (timeouts/bytes/chars) as the only knobs.

Impact

• For every deployment behind a fake-ip transparent proxy (the default Clash TUN mode across a very large user base), the shipped web_fetch is effectively limited to domains that resolve to real public IPs (typically CN-direct domains). Foreign domains — the primary fetch targets (GitHub, docs, APIs) — always fail with WEB_BLOCKED_URL.
• The failure is a clean structured error, so agents can fall back to other channels, but the flagship "public WebFetch enabled by default" feature (v0.1.2 changelog) is moot for this deployment class.

Suggested directions (any one would fix this without weakening the SSRF guarantee)

1. Custom resolver for the fetch provider (preferred): a config field like resolver: <ip> or a DoH URL on web-fetch-http. Resolution then returns real public IPs, the existing whole-set validation and connection pinning stay byte-for-byte identical — only the resolver is swappable. This is the most surgical fix and matches how other HTTP clients expose DNS overrides.
2. Explicit per-deployment allowlist: e.g. allowDomains: ["github.com", "*.githubusercontent.com"] — an operator-asserted trust list that skips the non-public rejection for matching hostnames (still pinning to whatever the resolver returned). Clear, scoped, and loud in config.
3. Minimum: document the incompatibility in the README known-limits ("fake-ip transparent-proxy DNS breaks public-address validation; use a resolver override / exclude DSH from fake-ip").

Workarounds in the wild (for other users hitting this)

• Exclude the needed domains from fake-ip at the proxy (OpenClash fake-ip-filter / dnsmasq whitelist) so they resolve to real IPs;
• Pin real IPs in the OS hosts file;
• Use Node fetch/MCP-based fetching tools instead of the shipped web_fetch.

Happy to test a build or provide more traces.

---

中文版

标题建议： web-fetch-http 的 SSRF 防护在透明代理 fake-ip DNS（198.18.0.0/15）环境下拒绝一切外域抓取，且无 resolver/allowlist 配置口

一句话：0.1.2 默认启用的公网 WebFetch，在路由器 OpenClash/Clash fake-ip 模式（国内最常见的透明代理 DNS 部署）下，外域全部报 WEB_BLOCKED_URL——域名经系统解析拿到 198.18 假 IP，SSRF 校验按保留段拒绝整组结果。国内直连域名（真实 IP）不受影响。

根因：dsh-web-fetch-http 用系统解析器（getaddrinfo）+ ipaddr.js range() === "unicast" 判定 + 任一地址非公网即整组拒绝。这是正确的防 DNS rebinding 设计，但 fake-ip 部署下解析结果永远不可能通过校验，而配置面上没有任何换 resolver / 域名白名单的口子。

建议（任一即可，均不削弱 SSRF 保证）：

1. 给 fetch 后端加自定义 resolver / DoH 配置（最优）：换解析源后现有整组校验与连接钉死逻辑原样生效；
2. 显式 allowDomains 信任清单：按域名豁免非公网拒绝，运营者显式断言；
3. 至少在 README 已知限制里写明与 fake-ip 透明代理不兼容。

临时绕行：代理侧 fake-ip-filter 放行所需域名 / 本机 hosts 固定真实 IP / 用 node fetch 或 MCP 类抓取工具。

[argszero]

源码侧全部核实（本地 0.1.2-alpha.1 = cd5ef81；alpha.2 仅版本号变动，web-fetch-http 源码与你的引用一致）。你的四条根因论断逐条都能对上，且这组文件里有一个你可能没注意到的现成注入点，让修复比想象中更顺。

核实结果（packages/web/web-fetch-http/）：

• 系统解析器：network.ts:9 lookup from 'node:dns/promises'，:82 以 { all: true, order: 'verbatim' } 调用——正是 getaddrinfo，fake-ip 部署下返回 198.18.x.x；
• ipaddr 分类：isPublicIpAddress（network.ts:53-63）= parsed.range() === 'unicast'。我拉取了实际依赖版本（package.json 钉 ipaddr.js@^2.5.0）的源码确认：lib/ipaddr.js:207 的保留段表里就有 [new IPv4([198, 18, 0, 0]), 15] → 198.18.0.0/15 分类为 reserved，非 unicast → 必被拒。你的"Clash fake-ip 段命中保留段"论断精确；
• 整组拒绝：resolvePublicAddresses（:74-109）任一地址非公网即抛 WEB_BLOCKED_URL（:99-100），通过后的地址集由 createPinnedLookup（:211-236）钉死给 Undici——防 DNS rebinding 的设计确实如你所述；
• 配置面无逃生口：index.ts:32-51 的 Config schema 只有 maxResponseBytes / maxBodyChars / timeoutMs / maxRedirects / userAgent 五个字段，无 resolver / allowlist / allowPrivate。

增量发现（对修复评估重要）：解析器注入点其实已经存在——HttpFetchProvider 构造函数第二参 resolveAddresses?: HttpFetchResolver（provider.ts:45-48），且 AddressResolver 类型（network.ts:33）与测试（tests/fetch-http.spec.ts:125-171 大量用注入 resolver 测 NAT64/rebinding/空结果）证明"可换解析"本就是设计意图。缺的只是把它从构造函数参数接到 Config schema 上——这正是你建议 1 的落点，改动面很小。

三条建议的评估：

1. 自定义 resolver（最优，同意）：Node 侧实现成本比想象低——dns.Resolver 类自带 setServers([ip])（systemd 下不读 /etc/resolv.conf 被劫持的问题也绕开），或 DoH 用现成 undici 实现；包装成现有 AddressResolver 形状（{ all: true, order: 'verbatim' } → LookupAddress[]）约十几行。解析返回真实公网 IP 后，整组校验与连接钉死逐字节不变——守卫语义零削弱，这是它优于其他方案的根本点；
2. allowDomains 白名单：可行但要小心一个 rebinding 变体——白名单跳过的若是"整个非公网拒绝"，那么该域名一旦经 DNS rebinding 解析到真内网段（RFC1918 等，非 fake-ip），连接就被钉到内网 IP 上，守卫对该域名完全失效。建议实现时收窄：白名单只豁免 fake-ip 保留段（198.18.0.0/15 等运营商/基准测试段），对 RFC1918/loopback/link-local 仍保持拒绝——这样"显式信任该域名经代理可达"与"永不连内网"两条性质同时成立；
3. README 记录：README.md:130 已有 "Known Limitations and Deferred Work" 一节（目前仅 2 条），补一条 fake-ip 不兼容说明是零成本，建议无论 1/2 做不做都先补。

另：不建议把 198.18.0.0/15 直接当公网段放行（第三种隐式变体）——那等于对所有 DNS 答案开放该段，信任面是隐式的且无配置痕迹，不如显式配置可审计。

结论：方向 1 先行（外科手术式、守卫语义不变、有现成注入点），方向 2 作为配置面补充并注意收窄豁免范围，方向 3 立即补。PR 通道重开后这是可落地的候选；若先出插件形态（在 web-fetch-http 外注册包装 provider）也能验证思路，但配置口最终应在 in-tree。

[eddiedon]

感谢逐条核实，特别是 resolveAddresses 注入点的确认——方向 1 的成本比预想低得多。两点补充：

1. resolver 建议直接支持 DoH 形态（而非仅 UDP 指定服务器）。 真实受害部署里，OpenClash/Clash TUN 普遍劫持 UDP53：dns.Resolver.setServers(['223.5.5.5']) 这类查询仍会被代理 DNS 抢答 fake-ip，方案在这类场景会静默失效。DoH 走 443 不受劫持（undici 有现成实现），建议作为首选形态，或在 README 明确此限制。

2. allowDomains 收窄方案同意。 rebinding 反例成立——"仅豁免 198.18.0.0/15 等保留段、RFC1918/loopback/link-local 照拒"是正确形态。

[YiHui-Liu]

相关讨论还存在于：#4893 #3051

[argszero]

Re-anchored against alpha.2 (0a53fb55be) just now — the state is unchanged from my earlier verification, which makes the three threads YiHui-Liu linked a clean triangle:

• [Security] DeepSeek Harness `web_fetch` 工具与命令沙箱缺乏 SSRF/私网出口过滤，可被诱导访问云元数据与内网服务 #3051 = why the guard exists (SSRF to cloud metadata / private nets).
• 建议为 web_fetch 增加可配置的非公网 CIDR 白名单，以支持 Fake-IP 与特殊路由环境 #4893 = the escape-hatch feature request (configurable non-public CIDR allowlist) — Yurzi hit the same fake-ip wall from the feature side.
• web-fetch-http SSRF guard rejects every fetch under transparent-proxy fake-ip DNS (198.18.0.0/15) - no resolver/allowlist escape hatch #5202 = the over-blocking bug report (foreign-domain fetches all rejected).

The fix direction satisfies #5202 and #4893 at once. Verified current alpha.2: provider.ts:47 still injects resolveAddresses: HttpFetchResolver = publicHttpNetwork.resolve as a constructor parameter, and index.ts:32-54 still has only maxResponseBytes / maxBodyChars / timeoutMs / maxRedirects / userAgent — no resolver / allowlist field on the Config schema. So the seam is still there, still unwired.

On the DoH point (eddiedon) — agree on the threat model, one precision:

The UDP53-hijack concern is real: under OpenClash/Clash TUN the transparent proxy intercepts plain UDP53, so dns.Resolver.setServers(['223.5.5.5']) still gets answered with fake-ip. DoH over 443 is the right answer there.

One precision: "undici 有现成实现" is slightly optimistic — undici's Agent accepts a custom connect.lookup, but it does not ship a DoH resolver itself. What makes this cheap is the seam already there: HttpFetchResolver / AddressResolver accepts any function, so an RFC 8484 JSON-over-HTTPS DoH client (plain fetch to https://<doh-endpoint>/resolve?name=…&type=A, parse Answer) can be passed as the injected resolver without touching undici at all. That DoH client would be a small separate module, and wiring it to Config (resolver: { mode: 'system' | 'doh', url?: string } or an allowList CIDR array) is the same schema change either way.

On allowDomains narrowing — agreed, and it's the constraint that keeps the triangle closed: exempt only the fake-ip range (198.18.0.0/15) and any user-declared CIDRs; keep RFC1918 / loopback / link-local rejected so #3051's security intent is preserved. That is also exactly what #4893 is asking for — a configurable allowlist that the SSRF guard still filters through.

The test pattern is already there too (tests/fetch-http.spec.ts injects resolvers for NAT64/rebinding/empty-result cases), so a DoH resolver + allowlist would just be more cases through the same harness.