Skip to content

ref(detectors): Improve logging of span-first/legacy detector mismatches - #120685

Merged
lobsterkatie merged 20 commits into
masterfrom
kmclb-improve-span-first-detector-mismatch-logging
Jul 28, 2026
Merged

ref(detectors): Improve logging of span-first/legacy detector mismatches#120685
lobsterkatie merged 20 commits into
masterfrom
kmclb-improve-span-first-detector-mismatch-logging

Conversation

@lobsterkatie

Copy link
Copy Markdown
Member

This PR includes a number of changes to the way we log instances of span-first detector results not matching legacy detector results, based on my experiences trying to parse our existing logs given the limitations of our logging product.

  • Stop relying on the rollout comparator to do the comparison and logging. (I was finding I was needing to work around more and more of its functionality, to the point where it ended up being more hindrance than help.) Instead, just use functions which actually do exactly what we need. This means we'll stop logging information we don't need, and have more control over how the information we do need is presented.

  • Strip duplicate and/or unneeded information before logging. (For example, parent_span_ids, cause_span_ids, and offender_span_ids all appear in two places in each PerformanceProblem object, cluttering the log.)

  • To make typing easier, and to avoid modifying the PerformanceProblem objects we're handling when doing said stripping, add a separate type for the dict version of a performance problem, and make PerformanceProblem.to_dict() produce a shallow copy of its internal collections rather than including the original collections themselves in the resulting dict.

  • When recording places in which span-first and legacy results differ, structure the diff as a dict of {place-where-there's-a-difference: [fingerprints of problems which differ in that spot]} rather than {fingerprint: [spots in which problems with that fingerprint differ]}, to get around the fact that the current way creates a different column name in the logs table for each fingerprint.

  • Fix all the tests to match the new logging and add a few missing tests.

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Jul 27, 2026
@lobsterkatie
lobsterkatie marked this pull request as ready for review July 28, 2026 18:04
@lobsterkatie
lobsterkatie requested review from a team as code owners July 28, 2026 18:04
@lobsterkatie
lobsterkatie merged commit 6744f51 into master Jul 28, 2026
71 checks passed
@lobsterkatie
lobsterkatie deleted the kmclb-improve-span-first-detector-mismatch-logging branch July 28, 2026 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants