feat(observability): enrich Sentry error events with root-cause fingerprinting#3298
Merged
migmartri merged 1 commit intoJul 19, 2026
Merged
Conversation
There was a problem hiding this comment.
All reported issues were addressed across 4 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
migmartri
force-pushed
the
feat/sentry-error-enrichment
branch
from
July 19, 2026 12:47
422cb49 to
49720d5
Compare
All server errors funnel through LogAndMaskErr, so every Sentry event
carries the same capture-site stacktrace and a generic *fmt.wrapError
outermost type: issues are indistinguishable.
Add a BeforeSend hook in pkg/servicelogger, wired into both the
controlplane and artifact-cas binaries, that analyzes the captured
error's unwrap chain (depth- and visit-bounded, cycle- and
errors.Join-safe) and enriches the event:
- Fingerprint: ['{{ default }}', discriminator] where discriminator is
SQLSTATE, gRPC code, or kratos reason — extends Sentry's stack-based
default grouping with structured root-cause data. Formatted error
text is never used for fingerprinting (it embeds request-specific
values). Errors without a discriminator keep default grouping.
- Tags: error.root_type, error.sqlstate, error.grpc_code,
error.kratos_reason, error.chain_depth, error.multi_error
- Generic wrapper exception types (*fmt.wrapError et al.) on the
primary exception rewritten to the root-cause type so issue titles
identify the failure.
One hook covers all capture paths (LogAndMaskErr, auditor, future
CaptureException calls) with zero callsite changes.
Also removes a pre-existing leading-newline lint violation in
app/controlplane/cmd/main.go.
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>
migmartri
force-pushed
the
feat/sentry-error-enrichment
branch
from
July 19, 2026 13:17
49720d5 to
18fc01b
Compare
jiparis
approved these changes
Jul 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
All server errors funnel through
pkg/servicelogger.LogAndMaskErr, which callssentry.CaptureExceptionwithAttachStacktrace: true. Every event therefore carries the same capture-site stacktrace (servicelogger in LogAndMaskErr) and — because biz errors are wrapped withfmt.Errorf("...: %w")— the same generic outermost type*fmt.wrapError. The issue list is homogeneous and grouping depends on variable message text (IDs, constraint names), while the actually discriminating info (root-cause type, SQLSTATE, gRPC code) sits unindexed in the exception chain.Change
Adds a
BeforeSendhook inpkg/servicelogger, wired into both the controlplane and artifact-cassentry.Init, that analyzes the captured error's unwrap chain and enriches every outgoing error event:["{{ default }}", discriminator]where discriminator is SQLSTATE, gRPC code, or kratos reason. This extends Sentry's stack-based default grouping with structured root-cause data — it preserves call-site separation while subdividing by failure class. Formatted error text is never used for fingerprinting because it embeds request-specific values (emails, UUIDs, paths) that would cause unbounded issue cardinality. Errors without a structured discriminator keep Sentry's default grouping.error.root_type,error.sqlstate,error.grpc_code,error.kratos_reason,error.chain_depth,error.multi_error— searchable root-cause metadata (types and codes only, never raw messages).*fmt.wrapError,*fmt.withMessage, pkg/errors wrappers) on the primary exception are rewritten to the root-cause Go type, so issue titles read e.g.*pgconn.PgError: creating version: ...instead of*fmt.wrapError.Chain analysis is a depth- and visit-bounded DFS over both
Unwrap() errorandUnwrap() []error(matching sentry-go's traversal). It deliberately avoidserrors.As(no cycle protection in stdlib), collects discriminators via per-node type assertions, and skips fingerprinting forerrors.Joinmulti-errors, where any single-root key would mislead.One hook covers all capture paths (
LogAndMaskErr, the auditor, futureCaptureExceptioncalls) with zero callsite changes.Also removes a pre-existing leading-newline lint violation in
app/controlplane/cmd/main.goflagged by the scoped lint run.Design notes
{{ default }}+ discriminator, not a custom fingerprint: Sentry's default grouping already uses stack traces to separate by call site. Prefixing{{ default }}preserves that while adding a structured subdivision. A fully custom fingerprint would lose Sentry's stack-based grouping improvements and require maintaining grouping logic in code. See Sentry fingerprint rules.fmt.Errorf("error finding user %s: %w", email, err)embed request-specific values. Regex scrubbing covers known shapes (emails, UUIDs, numbers) but cannot handle arbitrary%s/%vcontent (paths, names, opaque tokens). Structured discriminators avoid this entirely.operationErrorwrapper carrying a stable operation constant at the capture boundary would be the right approach — tracked in spec 114 as a deferred item.Test plan
pkg/servicelogger/sentry_enrich_test.go: SQLSTATE/gRPC/kratos/plain/nested/joined/cyclic chains, discriminator precedence, nil status, default-grouping fallbacks, recovered panics (error and non-error), existing-tag preservation, nil event/hint.go test ./pkg/servicelogger/,go build+go vet(controlplane, artifact-cas, pkg), andgolangci-lint runon all touched packages: clean.🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri