Skip to content

fix(telemetry): stop clipping the per-attempt error message at 256 chars#782

Merged
jarvis9443 merged 2 commits into
mainfrom
fix/telemetry-error-message-cap-1065
Jul 17, 2026
Merged

fix(telemetry): stop clipping the per-attempt error message at 256 chars#782
jarvis9443 merged 2 commits into
mainfrom
fix/telemetry-error-message-cap-1065

Conversation

@jarvis9443

@jarvis9443 jarvis9443 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Problem

The error_message an operator reads on the /logs page is truncated before it is ever stored. attempt_error_message caps the string at 256 chars — a bound copied from sanitize_tag, which exists to bound short provider tags (branded_provider, pk_label), not error text.

That cap sits on top of a bound that had already made the string safe to store: an UpstreamStatus message is bounded by the bridge at MAX_UPSTREAM_ERROR_MESSAGE_BYTES (1 KiB) in capture_upstream_error_http. So the 256-char cap was a second, tighter truncation with no budget of its own to defend.

Real upstream errors exceed it. A content-filter refusal — the shape that provoked the issue — runs ~260 chars on its own, so what reached telemetry ended mid-URL:

upstream returned HTTP 400: The response was filtered because the prompt triggered
the provider's content management policy. Please modify your prompt and retry. To
learn more about the content filtering policies that apply here, read the docum

— dropping exactly the part that tells the operator what to do about it.

Implementation

Raise the cap to 2048 chars (MAX_ATTEMPT_ERROR_MESSAGE_CHARS), above the bridge's 1 KiB byte budget, so the bridge's bound is the only limit that fires and the operator sees the whole message the bridge kept. Chars-vs-bytes matters here: a 1 KiB ASCII message is 1024 chars, so a 1024-char cap would still have clipped the prefix upstream returned HTTP 400: worth of tail.

The cap stays as a backstop — BridgeError variants that carry an unbounded string (e.g. Config) still can't write unbounded telemetry.

Baseline. Mainstream gateways bound this field an order of magnitude looser than we did: the closest comparable applies a 2048-char DB-storage cap to the stored error message (env-tunable), and truncates prompts on the same budget. 2048 matches that. None treat a short-tag-sized bound as adequate for error text.

Behavior change

Failed-attempt error_message telemetry can now carry up to ~1 KiB (whatever the bridge kept) instead of 256 chars. Only failed attempts populate the field, and cp-api stores it in a text column with no cap of its own; per-org usage_retention_days pruning is unchanged.

Tests

crates/aisix-proxy/src/attempt.rs — the two regression tests fail on the old cap and pass on the new one; verified by reverting the constant locally:

  • long_upstream_message_is_not_clipped — a long refusal survives to its trailing doc URL. The fixture is prose ending in a URL rather than generated filler, because what has to survive is the END of a realistically long message.
  • cap_clears_the_bridge_message_bound — a message already at the bridge's byte bound reaches telemetry whole.
  • pathological_message_still_hits_the_backstop — an unbounded Config string is still capped.
  • control_chars_are_stripped — unchanged sanitisation.

The paired dashboard fix (the detail panel renders this field with a truncate class, clipping it to one line regardless of length) ships separately in api7/AISIX-Cloud#1068.

Fixes api7/AISIX-Cloud#1065

The `error_message` an operator reads on the /logs page is truncated
before it is ever stored: `attempt_error_message` caps at 256 chars,
a bound copied from `sanitize_tag` — which exists to bound short
provider tags (`branded_provider`, `pk_label`), not error text.

An `UpstreamStatus` message is already bounded by the bridge at
`MAX_UPSTREAM_ERROR_MESSAGE_BYTES` (1 KiB), so the 256-char cap was a
second, tighter truncation applied on top of a bound that had already
made the string safe to store. Real upstream errors exceed it: Azure's
content-management-policy message is ~260 chars on its own, so what
reached telemetry ended mid-URL at "...read our documentation:
https://go.m" — dropping exactly the part that says what to do about it.

Raise the cap to 2048 chars, above the bridge's 1 KiB byte budget, so
the bridge's bound is the only one that fires and the operator sees the
whole message the bridge kept. The cap stays as a backstop for variants
carrying an unbounded string (e.g. `Config`).

Fixes api7/AISIX-Cloud#1065
@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Per-attempt error telemetry now permits up to 2048 characters after control-character removal. Tests cover upstream messages, bridge-bounded messages, unbounded errors, and newline or tab sanitization.

Changes

Attempt error telemetry

Layer / File(s) Summary
Error message backstop and validation
crates/aisix-proxy/src/attempt.rs
attempt_error_message uses a 2048-character cap while preserving control-character stripping, with tests covering bounded and unbounded error messages and sanitized output.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: moonming

🚥 Pre-merge checks | ✅ 5 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
E2e Test Quality Review ⚠️ Warning Blocking: the PR adds only unit tests for a helper in attempt.rs; it doesn’t exercise the real API→routing→logs path required for E2E coverage. Add an integration/e2e case that sends a failing request through the real flow and asserts the stored /logs error_message keeps the full tail.
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The PR addresses #1065 by making logged error messages more complete and adding regression tests for long and bounded messages.
Out of Scope Changes check ✅ Passed The changes stay focused on error-message telemetry and related tests, with no obvious unrelated additions.
Security Check ✅ Passed The change only widens capped attempt telemetry; it still strips controls, and I found no new secret/logging, auth, ownership, TLS, or DB issues.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: increasing per-attempt telemetry error message clipping from 256 chars.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/telemetry-error-message-cap-1065

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/aisix-proxy/src/attempt.rs`:
- Around line 203-225: Update the long-message regression test and its comment
to use provider-neutral wording and a generated message longer than 256
characters, while preserving the assertions that the full body and tail remain
intact. Rename long_upstream_message_is_not_clipped if needed to remove
provider-specific terminology, and eliminate the Azure/product names and
documentation URL from the fixture.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b64cbc72-e5e5-439d-a6b3-6482e54231f6

📥 Commits

Reviewing files that changed from the base of the PR and between 3b9253e and fa2e598.

📒 Files selected for processing (1)
  • crates/aisix-proxy/src/attempt.rs

Comment thread crates/aisix-proxy/src/attempt.rs Outdated
The regression fixture quoted an upstream vendor's refusal prose
verbatim, doc URL and all, inside a provider-neutral crate. Only its
LENGTH was load-bearing — the brand was decorative, which is what
AGENTS.md §7 rules out for shipped artifacts.

Keep the shape that makes the fixture worth having (prose ending in a
URL, so what must survive is the END of a realistically long message)
and write it generically. Still fails on the old 256-char cap.
@jarvis9443
jarvis9443 merged commit 90df23f into main Jul 17, 2026
11 checks passed
@jarvis9443
jarvis9443 deleted the fix/telemetry-error-message-cap-1065 branch July 17, 2026 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant