fix(ai-gateway): strip NUL bytes before microdollar_usage insert#2670
Merged
marius-kilocode merged 5 commits intomainfrom Apr 24, 2026
Merged
fix(ai-gateway): strip NUL bytes before microdollar_usage insert#2670marius-kilocode merged 5 commits intomainfrom
marius-kilocode merged 5 commits intomainfrom
Conversation
Postgres `text` columns reject NUL bytes with `22021 invalid byte sequence for encoding "UTF8": 0x00`, which crashes the microdollar_usage CTE insert and silently leaves the request un-billed. The observed rate is ~1.9k silent billing-write failures/day across FIM, chat, and responses paths. The realistic source is JSON-body-derived fields (prompt content, model IDs, upstream response fields like message_id / finish_reason) where NULs can pass through client/upstream JSON without HTTP-header-level rejection. Sanitizing at the DB boundary in `toInsertableDbUsageRecord` closes the bleed for every LLM path at once. - Add `stripNulBytesInPlace` helper that mutates string fields and records sanitized field names for observability. - Apply to both `core` and `metadata` before returning from `toInsertableDbUsageRecord`. - Emit a 5%-sampled `captureMessage` with the sanitized field list so we can identify the upstream source and fix it at the origin. - Export `stripNulBytesInPlace`, `toInsertableDbUsageRecord`, and `extractUsageContextInfo` for unit testing. - Add Jest tests covering: the pure helper, realistic body-sourced NULs, defensively-constructed header-sourced NULs, and the clean no-NUL path. Refs: Sentry KILOCODE-WEB-1G3Z
Contributor
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Files Reviewed (1 files)
Reviewed by gpt-5.4-20260305 · 290,758 tokens |
chrarnoldus
reviewed
Apr 23, 2026
…ureMessage for NUL-byte diagnostic Per chrarnoldus review: this is a one-off source-attribution probe, not an issue to triage in Sentry. console.warn lands in Axiom with full structured context, which is queryable and doesn't eat Sentry quota. Drops the 5% sampling gate since quota is no longer a concern, so the dominant source field will surface faster.
…r-usage-nul-bytes
MicrodollarUsageContext gained a required ttfb_ms field in #2734 after this branch diverged. Merged main and set ttfb_ms: null in the test helper so the new NUL-byte sanitization tests satisfy the type.
chrarnoldus
approved these changes
Apr 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Postgres
textcolumns reject NUL bytes with22021 invalid byte sequence for encoding "UTF8": 0x00, which crashes themicrodollar_usageCTE insert intoInsertableDbUsageRecordand silently leaves the request un-billed. Observed rate: ~1.9k silent billing-write failures/day across FIM, chat, and responses paths.stripNulBytesInPlacehelper, apply it to bothcoreandmetadataright before returning fromtoInsertableDbUsageRecord.captureMessagewith the sanitized field list so we can identify the upstream source and fix it at the origin.Sentry: KILOCODE-WEB-1G3Z — 1,363 events over ~5 weeks, multiple LLM paths.
Why sanitize at the DB boundary
The realistic NUL source is JSON-body-derived fields — prompt content (
system_prompt_prefix,user_prompt_prefix), LLM response fields (model,inference_provider,message_id,finish_reason,upstream_id,requested_model). HTTP header-sourced fields (http_user_agent,machine_id,session_id, etc.) can't realistically carry NULs because Node'sHeadersconstructor rejects them per RFC 7230 — documented in the test.Sanitizing at the
toInsertableDbUsageRecordboundary closes the bleed for every LLM path at once with a single chokepoint, and defensively covers header-sourced fields in case a future code path bypassesHeadersvalidation.Once the sampled
captureMessagesurfaces the actual source field(s), the plan is to sanitize upstream and remove the defensive sanitizer.Blast radius
/api/fim/completions(Mistral Codestral) — 66%/api/openrouter/responses(grok-code-fast-1:free) — 26%/api/openrouter/chat/completions— 6%/api/gateway/chat/completions— 2%microdollar_usagetable also feeds abuse detection and the 'first usage' PostHog event — downstream analytics are under-counting.Risk
Low. Sanitizer is a one-shot walk over two flat objects, only modifies strings that actually contain a NUL (early-exits via
indexOf), and the behavior is exercised by unit tests. No DB or schema changes.Test plan
pnpm --filter @kilocode/web exec jest src/lib/ai-gateway/processUsage.test.ts— all 27 tests pass (6 new).pnpm --filter @kilocode/web typecheck— clean.pnpm -w exec oxlint --config .oxlintrc.json apps/web/src/lib/ai-gateway/processUsage.ts apps/web/src/lib/ai-gateway/processUsage.test.ts— clean.Follow-up
captureMessagein Sentry for a few days to identify which field is the dominant NUL source.extractPromptInfoor an upstream body parser) and remove this defensive sanitizer.