Skip to content

Conversation

@aliu39
Copy link
Member

@aliu39 aliu39 commented Nov 23, 2025

Adds specialized rpcs for getting a list of log/metric attribute dicts filtered by trace_id and optional substring. This is done with a single snuba rpc (not counting short id lookup) and is a lot more performant than the table queries we were doing earlier.

Return format is list of attr key-value dicts. This is different from the current tool output format which includes type, tbd if we still need to include that


GetTrace experiment:

Using https://snuba-admin.getsentry.net/#rpc-endpoints, I made requests to EndpointGetTrace and EndpointTraceItemTable for the same purpose - filter logs by a trace (2h old) in Seer project. Time range is last 30d, and I selected the same columns. The table query has an additional LIKE filter on sentry.body (message) - this isn't supported for GetTrace so we'd have to post-process in sentry or seer

EndpointGetTrace - 1318904 progressBytes (bytes processed by clickhouse)
EndpointTraceItemTable - 52792914551 progressBytes. 40k times more, prob scanning from the start of the time range. This is evidence the GetTrace endpoint is way better optimized for trace_id filters.

The requests:

GetTrace:

{
  "meta": {
    "organizationId": "1",
    "referrer": "seer.rpc",
    "projectIds": [
      "6178942"
    ],
    "startTimestamp": "2025-10-24T04:06:08.199596Z",
    "endTimestamp": "2025-11-23T04:06:08.199596Z",
    "traceItemType": "TRACE_ITEM_TYPE_LOG",
    "downsampledStorageConfig": {
      "mode": "MODE_NORMAL"
    }
  },
  "traceId": "983c9c1267364ce9b3ef9fc14c4b25a5",
  "items": [
    {
      "attributes": [
        {
          "type": "TYPE_STRING",
          "name": "sentry.item_id"
        },
        {
          "type": "TYPE_DOUBLE",
          "name": "sentry.timestamp_precise"
        },
        {
          "type": "TYPE_STRING",
          "name": "sentry.timestamp"
        },
        {
          "type": "TYPE_STRING",
          "name": "sentry.severity_text"
        },
        {
          "type": "TYPE_STRING",
          "name": "sentry.trace.parent_span_id"
        },
        {
          "type": "TYPE_STRING",
          "name": "sentry.body"
        }
      ],
      "itemType": "TRACE_ITEM_TYPE_LOG"
    }
  ]
}

TraceItemTable

{
  "meta": {
    "organizationId": "1",
    "referrer": "api.auth-token.events",
    "projectIds": [
      "6178942"
    ],
    "startTimestamp": "2025-10-24T04:06:08.199596Z",
    "endTimestamp": "2025-11-23T04:06:08.199596Z",
    "traceItemType": "TRACE_ITEM_TYPE_LOG",
    "downsampledStorageConfig": {
      "mode": "MODE_NORMAL"
    }
  },
  "columns": [
    {
      "key": {
        "type": "TYPE_STRING",
        "name": "sentry.item_id"
      },
      "label": "id"
    },
    {
      "key": {
      "type": "TYPE_DOUBLE",
          "name": "sentry.timestamp_precise"
        },
      "label": "timestamp_precise"
    },
    {
      "key": {
      "type": "TYPE_DOUBLE",
          "name": "sentry.timestamp"
        },
      "label": "timestamp"
    },
    {
      "key": {
          "type": "TYPE_STRING",
          "name": "sentry.severity_text"
        },
      "label": "severity"
    },
    {
      "key": {
          "type": "TYPE_STRING",
          "name": "sentry.trace.parent_span_id"
        },
      "label": "parent_span_id"
    },
    {
      "key":{
          "type": "TYPE_STRING",
          "name": "sentry.body"
        },
      "label": "message"
    }
  ],
  "orderBy": [
    {
      "column": {
        "key": {
          "type": "TYPE_DOUBLE",
          "name": "sentry.timestamp"
        },
        "label": "timestamp"
      },
      "descending": true
    }
  ],
  "pageToken": {
    "offset": "0"
  },
  "filter": {
    "andFilter": {
      "filters": [
        {
          "comparisonFilter": {
            "key": {
              "type": "TYPE_STRING",
              "name": "sentry.body"
            },
            "op": "OP_LIKE",
            "value": {
              "valStr": "%POST%"
            }
          }
        },
        {
          "comparisonFilter": {
            "key": {
              "type": "TYPE_STRING",
              "name": "sentry.trace_id"
            },
            "op": "OP_EQUALS",
            "value": {
              "valStr": "983c9c1267364ce9b3ef9fc14c4b25a5"
            }
          }
        }
      ]
    }
  }
}

@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Nov 23, 2025
@codecov
Copy link

codecov bot commented Nov 23, 2025

Codecov Report

❌ Patch coverage is 87.27273% with 14 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/sentry/seer/explorer/tools.py 87.27% 14 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master   #103875      +/-   ##
===========================================
+ Coverage   80.40%    80.61%   +0.21%     
===========================================
  Files        9316      9318       +2     
  Lines      397607    397563      -44     
  Branches    25398     25365      -33     
===========================================
+ Hits       319677    320486     +809     
+ Misses      77478     76625     -853     
  Partials      452       452              

@aliu39 aliu39 changed the title Aliu/trace query feat(explorer): rpcs for getting log/metric attrs for a trace id + substring Nov 25, 2025
Copy link
Member

@roaga roaga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approving to unblock

for field in self.default_fields:
assert field in log, field

def test_get_log_attributes_for_trace(self) -> None:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should probably test with some custom attribute as well, not just the build in ones

also should add a test for metrics

@roaga
Copy link
Member

roaga commented Nov 25, 2025

You mentioned in the other PR: "The trace waterfall tables can also be updated to call these, for faster performance"

How could we extend these to do so? don't think they'd work as is. I'm also interested if we can get those tables without a trace time range because that would enable us to show metrics/logs in a trace even if there are no spans (see https://linear.app/getsentry/issue/AIML-1697/allow-viewing-traces-without-spans)

@aliu39
Copy link
Member Author

aliu39 commented Nov 25, 2025

You mentioned in the other PR: "The trace waterfall tables can also be updated to call these, for faster performance"

How could we extend these to do so? don't think they'd work as is. I'm also interested if we can get those tables without a trace time range because that would enable us to show metrics/logs in a trace even if there are no spans (see https://linear.app/getsentry/issue/AIML-1697/allow-viewing-traces-without-spans)

Smart yeah this can be done, if there's no spans we can just try the whole 90d range, should still be performant

@aliu39 aliu39 enabled auto-merge (squash) November 26, 2025 17:34
@aliu39 aliu39 merged commit f5b3cda into master Nov 26, 2025
67 checks passed
@aliu39 aliu39 deleted the aliu/trace-query branch November 26, 2025 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants