fix(sandbox): emit small dep-metrics lines (pipeline drops large ones)#4275
Merged
Conversation
Measured in prod: the log pipeline stores only small lines from sandbox
pods (max ~765B observed across namespaces); the ~15KB chunked lines from
the previous fix were rejected outright ("missing _msg field"), so the
panel got zero data. The earlier ~250B per-dep lines survived fine.
Drop MAX_LINE_BYTES 15500 -> 600 so each line stays well inside the
proven-surviving range (~13 deps/line, ~27 lines for a big install). Many
small lines may be burst-sampled by the collector; the Grafana panel sums
sample_rate to correct for that. Tests updated to assert the small cap.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
1 issue found across 2 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
…ines The chunker budgets deps against the measured envelope, but repoName/branch were unbounded — a very long branch name could inflate the envelope past the 600B cap on its own, oversizing every line (incl. the always-emitted first dep) and getting the whole install's data dropped. Clip both to 80 chars before measuring and emitting. Adds a 500-char-branch test.
decocms Bot
pushed a commit
that referenced
this pull request
Jul 2, 2026
PR: #4275 fix(sandbox): emit small dep-metrics lines (pipeline drops large ones) Bump type: patch - @decocms/sandbox (packages/sandbox/package.json): 1.9.2 -> 1.9.3 Deploy-Scope: both
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause (measured in prod, not inferred)
The previous chunked format emitted ~15KB
sandbox.depslines. The log pipeline rejects large lines from sandbox pods — confirmed:sandbox.depsappears nowhere in VictoriaLogs (0 hits, any form), while VL logs 53missing _msg fieldingestion rejections, one from a 1.9.2 pod._msg= 765 bytes; nothing >2KB exists.So the 16KB truncation we saw earlier was the generous end; in practice only small lines land. My chunked cap (15.5KB) was ~20× too big → every line dropped → panel empty.
Fix
MAX_LINE_BYTES15500 → 600. Each line stays well inside the proven-surviving range (~13 deps/line → ~27 lines for a big install, vs the 350 one-per-dep lines that got burst-sampled). Small enough to survive ingestion, few enough to cut sampling.Residual burst-sampling is handled in the panel:
stats by (deps) sum(sample_rate)(unsampled → sample_rate=1 → sum=count; sampled → reversed to an estimate).Tests
Updated to assert the small cap: every emitted line
< 700bytes and the full set intact exactly once, across typical (391), long-name (~203-char), and 4000-short-dep inputs. 6 pass, typecheck clean.Honest caveat
I've had three wrong iterations on this chasing undocumented pipeline limits (16KB truncation → burst sampling → large-line drop). This cap is grounded in the measured 765B ceiling + proven 250B survival, but I can only fully confirm end-to-end once it deploys and a sandbox installs. I'll verify against real
sandbox.depslines after roll before calling it done.🤖 Generated with Claude Code
Summary by cubic
Emit small
sandbox.depslog lines (~600B) and clip long repo/branch metadata so they survive ingestion; previous ~15KB lines were dropped, leaving the panel with no data.repoNameandbranchto 80 chars so long metadata can’t oversize lines; chunking budgets against the clipped values.sample_rate.Written for commit 4ef12d1. Summary will update on new commits.