ref(detectors): Improve logging of span-first/legacy detector mismatches - #120685
Merged
lobsterkatie merged 20 commits intoJul 28, 2026
Conversation
lobsterkatie
marked this pull request as ready for review
July 28, 2026 18:04
roggenkemper
approved these changes
Jul 28, 2026
lobsterkatie
deleted the
kmclb-improve-span-first-detector-mismatch-logging
branch
July 28, 2026 23:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR includes a number of changes to the way we log instances of span-first detector results not matching legacy detector results, based on my experiences trying to parse our existing logs given the limitations of our logging product.
Stop relying on the rollout comparator to do the comparison and logging. (I was finding I was needing to work around more and more of its functionality, to the point where it ended up being more hindrance than help.) Instead, just use functions which actually do exactly what we need. This means we'll stop logging information we don't need, and have more control over how the information we do need is presented.
Strip duplicate and/or unneeded information before logging. (For example,
parent_span_ids,cause_span_ids, andoffender_span_idsall appear in two places in eachPerformanceProblemobject, cluttering the log.)To make typing easier, and to avoid modifying the
PerformanceProblemobjects we're handling when doing said stripping, add a separate type for the dict version of a performance problem, and makePerformanceProblem.to_dict()produce a shallow copy of its internal collections rather than including the original collections themselves in the resulting dict.When recording places in which span-first and legacy results differ, structure the diff as a dict of
{place-where-there's-a-difference: [fingerprints of problems which differ in that spot]}rather than{fingerprint: [spots in which problems with that fingerprint differ]}, to get around the fact that the current way creates a different column name in the logs table for each fingerprint.Fix all the tests to match the new logging and add a few missing tests.