Web profile: dsh-tool-web (web_fetch) disabled by default -> agents fall back to bash+curl with default UA and get blocked by WAF/CDN

[maxwell-feng]

Web profile: dsh-tool-web (web_fetch) is disabled by default → agents fall back to bash+curl with a fingerprintable default UA and get blocked by WAF/CDN

In a real session on the web profile, asking the agent to fetch a web page produced this chain:

1. web_search (deepseek-official provider) fails — DEEPSEEK_API_KEY is not set. Configuring API keys on the user side is understood to be the design; the problem is what happens after that failure.
2. web_fetch is not in the agent's tool list at all — even though the tool exists in the codebase.
3. The agent falls back to the bash tool and runs curl https://... directly → default curl/x.x User-Agent, no browser fingerprint → the request is blocked by WAF (reproduced on Baidu's search page) → the agent then improvised UA spoofing with -A "Mozilla/5.0 ... Chrome/125..." (fragile), and repeated blocked requests risk IP bans.

The effective web-profile config shows why:

- id: tool-web
  name: '@deepseek-ai/dsh-tool-web'
  config:
    fetch: false
  disabled: true

Requests:

1. Enable web_fetch by default in the web profile. Plain HTTP fetch does not need an API key — it runs through the local ctx.web fetch provider. If it is intentionally off by default, please document the override in the README; right now the only way to find it is dumping the config:

   # ~/.dsh/profiles/web/cordis.patch.yml
   - id: tool-web
     enabled: true
     config:
       fetch: true

2. Make the fetch User-Agent configurable (and default to something less fingerprintable). This is a product-quality ask, not an anti-bot one: an obviously non-browser UA is exactly what gets requests blocked and IPs banned on WAF/CDN-fronted sites.
3. When web tools are unavailable (search key missing / fetch disabled), surface it in the runtime context or UI instead of leaving the agent to silently fall back to bash+curl with default headers.

Happy to provide the config/UX details if useful.

[weijiafu14]

这个场景可以先用现成 Pi Web 工具把“没有 web_fetch → 模型退化到 bash + curl”这条链拆掉，而且现在安装不需要写 Cordis overlay：

dsh plugin --profile web add pi2dsh@0.11.0
dsh plugin --profile web add @juicesharp/rpiv-web-tools

装完重启后会向 DSH 注册 web_search 和 web_fetch 两个正常工具。搜索可选多种后端；例如自建 SearXNG：

export WEB_SEARCH_PROVIDER=searxng
export SEARXNG_URL=https://your-searxng.example.com

这条路径已经做过真实运行验证：web_search 经 SearXNG 返回结果，web_fetch 直接抓取公开 HTTPS 页面，而不是让 agent 临时拼 curl。桥接项目是 pi2dsh，Pi 包保持原样运行。

它不能保证绕过所有站点的 WAF，但能直接补上原帖缺失的正式 fetch/search 工具，并避免因为工具不存在而自动滑向默认 UA 的裸 curl。