Skip to content

Memory embedding fails with HTTP 400 when using SiliconFlow (fetchWithSsrFGuard incompatibility) #91396

Closed as not planned

Description

@xiangcaoni

Environment

  • OpenClaw version: 2026.6.1 (2e08f0f)
  • OS: WSL2 Ubuntu 24.04 on Windows 11
  • Node.js: v24.15.0
  • Install method: npm global install

Configuration

Memory search configured to use SiliconFlow as an OpenAI-compatible embedding provider:

{
  "agents": {
    "defaults": {
      "memorySearch": {
        "provider": "openai",
        "model": "BAAI/bge-large-zh-v1.5",
        "remote": {
          "baseUrl": "https://api.siliconflow.cn/v1",
          "apiKey": "sk-..."
        }
      }
    }
  }
}

SiliconFlow model added to models.providers:

{
  "siliconflow": {
    "api": "openai-completions",
    "apiKey": "sk-...",
    "baseUrl": "https://api.siliconflow.cn/v1",
    "models": [
      {
        "api": "openai-completions",
        "id": "BAAI/bge-large-zh-v1.5",
        "name": "BGE Large Chinese 1.5",
        "input": ["text"],
        "contextWindow": 512,
        "maxTokens": 512,
        "reasoning": false
      }
    ]
  }
}

Expected Behavior

openclaw memory index --force --agent main should successfully index memory files using SiliconFlow's embedding API.

Actual Behavior

Memory indexing fails with:

Memory index failed (main): openai embeddings failed: HTTP 400
{"code":20015,"message":"The parameter is invalid. Please check again.","data":null}

Key Findings from Debugging

1. Request Looks Correct

I added debug logging to dist/embedding-provider-CtNPrLQj.js (the createOpenAiEmbeddingProvider function) and dist/memory-core-host-engine-embeddings-BB4b09Jx.js (the postJson function). The debug output confirms:

[POSTJSON-DEBUG] url=https://api.siliconflow.cn/v1/embeddings
[POSTJSON-DEBUG] body={"model":"BAAI/bge-large-zh-v1.5","input":["...valid text..."]}
[POSTJSON-DEBUG] headers={"Content-Type":"application/json","Authorization":"Bearer sk-..."}
[POSTJSON-DEBUG] ssrfPolicy={"allowedHostnames":["api.siliconflow.cn"]}
[POSTJSON-DEBUG] res.status=400

The URL, headers, body, and model are all correct and match what works via direct HTTP clients.

2. Same Request Works Outside OpenClaw

The exact same HTTP request (same URL, same headers, same JSON body) sent via curl or Node.js fetch (undici) returns HTTP 200 with valid embeddings:

# curl test — SUCCEEDS (200)
curl -s -w "\nHTTP_CODE:%{http_code}" \
  "https://api.siliconflow.cn/v1/embeddings" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-..." \
  -d '{"model":"BAAI/bge-large-zh-v1.5","input":["# MEMORY.md content..."]}'
# → HTTP_CODE:200 ✅
// Node.js undici test — SUCCEEDS (200)
const res = await fetch('https://api.siliconflow.cn/v1/embeddings', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer sk-...'
  },
  body: JSON.stringify({ model: 'BAAI/bge-large-zh-v1.5', input: ['test'] })
});
// → Status: 200 ✅

3. Intermittent Success Pattern

During testing with a local HTTP proxy intercepting requests:

  • Some requests succeed (200), some fail (400) — often the first request fails
  • Content that fails via openclaw succeeds when replayed via curl
  • This suggests the issue is at the HTTP transport layer, not the API or content level

4. Request Flow

The call chain is:

  1. MemoryIndexManager.loadProviderResult() → createEmbeddingProvider()
  2. → openAiMemoryEmbeddingProviderAdapter.create() → createOpenAiEmbeddingProvider()
  3. → fetchRemoteEmbeddingVectors() → postJson() → withRemoteHttpResponse()
  4. → fetchWithSsrFGuard() (in dist/fetch-guard-B8N1GCBS.js)

The SSRF guard's fetchWithSsrFGuardInternal adds complex connection handling (dispatcher selection, proxy checks, DNS pinning). The hypothesis is that something in this guarded fetch pipeline modifies the HTTP request in a way that SiliconFlow's API rejects.

5. Also Fails with openai-compatible Provider

Setting memorySearch.provider to "siliconflow" (which auto-resolves to "openai-compatible") also fails with the same 400 error. The openai-compatible code path in dist/registry-BVye-IRt.js has its own postEmbeddingRequest function that uses fetchWithSsrFGuard directly.

Workaround

Setting memorySearch.provider to "none" (FTS-only mode, no vector embeddings) allows memory indexing to succeed:

{
  "memorySearch": {
    "provider": "none"
  }
}

This provides keyword-based memory search but lacks semantic/vector search capabilities.

Possible Root Cause Hypothesis

The fetchWithSsrFGuard in dist/fetch-guard-B8N1GCBS.js may be:

  1. Adding/modifying HTTP headers that SiliconFlow rejects
  2. Using HTTP/2 connection reuse that causes state issues
  3. Applying a dispatcher/proxy agent that modifies the TCP/TLS connection behavior
  4. Setting transfer-encoding: chunked vs content-length in a way SiliconFlow doesn't expect

Steps to Reproduce

  1. Install OpenClaw 2026.6.1
  2. Configure memorySearch with provider: "openai", model BAAI/bge-large-zh-v1.5, and remote.baseUrl pointing to https://api.siliconflow.cn/v1
  3. Create a memory file in ~/.openclaw/workspace/memory/
  4. Run openclaw memory index --force --agent main
  5. Observe HTTP 400 error from SiliconFlow

Activity

  1. clawsweeper commented on Jun 8, 2026

    @clawsweeper
    Contributor

    Codex review: this still needs some work.

    Summary
    The reporter’s corrected input-limit diagnosis remains actionable: neither embedding route declares this model’s capacity, and the proposed fix closed unmerged; newer chunking prevents claiming a verified current SiliconFlow failure.

    Reproducibility: Source confirms missing capacity declarations, but no current failing input establishes that the reported SiliconFlow HTTP 400 persists after chunking changes; no tests or live requests were run.

    Maintainer decision needed

    • Question: Should compatible embedding input budgets derive from matching contextWindow metadata or use a dedicated embedding-capacity setting?
    • Recommendation: Reuse matching context capacity: Use contextWindow only for the selected model and destination, applying the existing conservative splitter without repurposing maxTokens.
    • Why: Both routes omit configured capacity, but existing documentation does not establish how chat-model metadata should govern embedding input budgets.

    Next step
    Establish a failing current-version indexing input and choose the capacity contract before repairing both embedding adapters.

    Review details

    Best possible solution:

    Apply one agreed, destination-bound input-budget contract across both adapters while retaining native model limits and shared chunk enforcement.

    Do we have a high-confidence way to reproduce the issue?

    Source confirms missing capacity declarations, but no current failing input establishes that the reported SiliconFlow HTTP 400 persists after chunking changes; no tests or live requests were run.

    Is this the best way to solve the issue?

    The earlier adapter-only proposal is incomplete and prefers an output-token field; a shared capacity contract tested through both real provider routes is safer.

    AGENTS.md: found and applied where relevant.

    Remaining risk / open question:

    • The exact historical SiliconFlow failure has not been reproduced after the CJK-aware chunking changes.
    • Configured embedding input-capacity precedence and destination matching lack an established contract; maxTokens describes output capacity.

    Codex review notes: model internal, reasoning medium; reviewed against 5462f7f14643.

    Label changes

    Label changes:

    No label changes.

    Label justifications:

    • P2: The report concerns semantic indexing with a particular compatible embedding model in an advanced configuration workflow.
    • impact:auth-provider: The affected boundary is embedding provider/model configuration and input capacity.
    Evidence reviewed

    What I checked:

    • Reporter corrected the root cause: xiangcaoni withdrew the guarded-fetch hypothesis and attributed the 2026.6.1 failures to inputs exceeding the model’s 512-token capacity; successful curl tests had used shorter inputs.
    • OpenAI adapter omits the BGE capacity: The adapter assigns maxInputTokens only through a three-model native OpenAI map and does not resolve configured model contextWindow. (extensions/openai/embedding-provider.ts:63, 5462f7f14643)
    • Custom-provider route also omits capacity: Configured OpenAI-compatible APIs resolve to the generic adapter, whose returned provider contains model, optional dimensions, and embedding operations but no maxInputTokens. (src/plugins/openai-compatible-embedding-provider.ts:505, 5462f7f14643)
    • Fallback and enforcement use byte approximations: Undeclared remote providers receive an 8192 budget; enforcement compares UTF-8 bytes, and existing regression tests establish byte splitting and hard-cap behavior rather than model-token counting. (packages/memory-host-sdk/src/host/embedding-model-limits.ts:5, 5462f7f14643)
    • Current chunking changes the historical reproduction: Markdown chunking uses the shared CJK-aware weighted character estimator, explicitly documented in source as a provider-independent heuristic rather than an exact tokenizer. (packages/normalization-core/src/cjk-chars.ts:1, 5462f7f14643)
    • Supported route and metadata semantics: The memory configuration reference supports configured OpenAI-compatible providers; ModelDefinitionConfig explicitly defines maxTokens as completion/output capacity, so the earlier proposal’s preferred field is unsuitable for input capacity. Public reference: https://docs.openclaw.ai/reference/memory-config. (src/config/types.models.ts:47, 5462f7f14643)
    How this review workflow works
    • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
    • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
    • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
    • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
    • Maintainers can also comment @clawsweeper review to request a fresh review only.
    • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge; those execution lanes are retired.
    • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

    Reviewed October 15, 2026, 10:57 PM ET / October 16, 2026, 02:57 UTC.

  2. added
    P2Normal backlog priority with limited blast radius.
    impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.
    clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.
    on Jun 8, 2026
  3. xiangcaoni commented on Jun 8, 2026

    @xiangcaoni
    Author

    更正:根因不是 fetchWithSsrFGuard

    经过深入调试,发现真正的根因是 token 限制配置缺失,而非 HTTP pipeline 兼容性问题。

    真实根因

    1. BAAI/bge-large-zh-v1.5 的 token 上限是 512 tokens(不是 8192)
    2. Openclaw 的 OPENAI_MAX_INPUT_TOKENS 只包含 OpenAI 原生模型(text-embedding-3-small、text-embedding-ada-002 等),不包含第三方兼容模型
    3. 当模型不在这个映射中时,maxInputTokens 不会被设置,导致 enforceEmbeddingMaxInputTokens 使用默认值 8192
    4. 超过 512 tokens 的文本被原样发送给 SiliconFlow → API 返回 400 "The parameter is invalid"

    为什么之前误判为 fetchWithSsrFGuard

    因为文本长度不一,有的 ≤512 tokens(成功),有的 >512 tokens(失败),表现为「间歇性」失败,且 curl 测试时恰好用的都是短文本。

    本地临时修复

    在 OPENAI_MAX_INPUT_TOKENS 中添加:

    "BAAI/bge-large-zh-v1.5": 1000,  // ~512 tokens for Chinese text

    建议上游修复方案

    1. 方案 A(快速):在 OPENAI_MAX_INPUT_TOKENS 中添加常见第三方 embedding 模型的 token 限制
    2. 方案 B(通用):让 resolveEmbeddingMaxInputTokens 回退时读取模型的 contextWindow/maxTokens 配置,而不是硬编码 8192
    3. 方案 C(容错):当 API 返回 400 时,自动降低 maxInputTokens 并重试

    环境

    • OpenClaw 2026.6.1
    • Node.js v24.15.0
    • SiliconFlow API + BAAI/bge-large-zh-v1.5
  4. added
    issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.
    clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.
    clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.
    clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.
    and removed
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    on Jun 15, 2026
  5. 17 remaining items

  6. added
    issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.
    clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.
    and removed
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    on Sep 4, 2026
  7. added
    clawsweeper:needs-infoClawSweeper needs more reporter information before it can verify this issue.
    issue-rating: 🦐 gold shrimpDecent issue quality, but reproduction details are still incomplete.
    and removed
    clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.
    issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.
    on Oct 7, 2026
  8. openclaw-barnacle commented on Oct 23, 2026

    @openclaw-barnacle

    This issue has been automatically marked as stale after 7 days without qualifying human activity.
    Without new human activity, Barnacle may close it as not planned 7 days after this message was posted. This is administrative inactivity cleanup, not a claim that the problem is fixed.
    A human update restarts the inactivity clock; a new PR head also restarts it. A maintainer can add no-stale for a durable hold, or bad-barnacle to suppress Barnacle. Remove the hold when work can resume.

  9. steipete commented on Oct 24, 2026

    @steipete
    Contributor

    Thanks for filing this. We're closing older issues that no longer look actionable on current main.

    The root cause turned out to be the model's 512-token input limit, and embedding chunking has changed since 2026.6.1; we couldn't confirm the HTTP 400 still happens on current main. If SiliconFlow embeddings still fail on a recent release, please open a fresh issue with the request size and error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:needs-infoClawSweeper needs more reporter information before it can verify this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.issue-rating: 🦐 gold shrimpDecent issue quality, but reproduction details are still incomplete.staleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions