feat(ai): Infer gen_ai_operation_type from span.op #5129

vgrozdanic · 2025-09-10T11:59:37Z

Description

In relay, we want to infer the gen_ai.operation.type attribute (it's a newly added attribute) from gen_ai.operation.name.

This will be used in UI to easier query and plot data based on operation type. Currently we have a long list of attributes that we query for one operation type, but as that list grows, it is becoming harder and harder to pass all the parameters as query param in a request to EAP.

Implementation plan

PRs will be separated to make it easier to review. The mapping will be defined in sentry, and it will be propagated to relays as a part of global config to make it easier for us to change and extend the mappings.

(in relay):

add ai_operation_type_map to global config (feat(ai): Add ai_operation_type_map to global config #5125)
implement inference of the operation type from name and add tests (this PR)

(in sentry):

during global config generation, generate ai_operation_type_map from defined list of attributes

Part of TET-1092: Introduce gen_ai.operation.type attribute

linear · 2025-09-10T12:00:58Z

TET-1092 Introduce `gen_ai.operation.type` attribute

relay-event-normalization/src/normalize/mod.rs

vgrozdanic · 2025-09-10T12:23:17Z

CHANGELOG.md

 - Add gen_ai_cost_total_tokens attribute and double write total tokens cost. ([#5121](https://github.com/getsentry/relay/pull/5121))
 - Change mapping of incoming OTLP spans with `ERROR` status to Sentry's `internal_error` status. ([#5127](https://github.com/getsentry/relay/pull/5127))
- Add `ai_operation_type_map` to global config ([#5125](https://github.com/getsentry/relay/pull/5125))
+- Add `ai_operation_type_map` to global config. ([#5125](https://github.com/getsentry/relay/pull/5125))


added dot at the end of the change log entry

vgrozdanic · 2025-09-10T12:24:25Z

relay-event-normalization/src/event.rs

+use crate::normalize::AiOperationTypeMap;
 use crate::normalize::request;
-use crate::span::ai::enrich_ai_span_data;
+use crate::span::ai::enrich_ai_event_data;


this is a function rename - it was called enrich_ai_span_data but it was operating on an event that had a lot of spans.

Also a new function enrich_ai_span_data is now introduced that operates only on span, and it is called inside of enrich_ai_event_data to prevent code duplication

vgrozdanic · 2025-09-10T12:25:19Z

relay-event-normalization/src/event.rs

    pub ai_model_costs: Option<&'a ModelCosts>,

+    /// Configuration for mapping AI operation types from span.op to gen_ai.operation.type
+    pub ai_operation_type_map: Option<&'a AiOperationTypeMap>,


Just like with ai_model_costs we don't have access to global config object in a code path where we would need it, so we add this map to NormalizationConfig which is accessible where we need it

vgrozdanic · 2025-09-10T12:32:45Z

relay-event-normalization/src/normalize/span/ai.rs

this file is a bit refactored:

extract_ai_data is no longer a pub function - there is no need for it

map_ai_measurements_to_data is no longer a pub function - there is no need for it

is_ai_span is no longer a pub function - there is no need for it

to prevent logic duplication, we now have only 2 pub functions enrich_ai_event_data and enrich_ai_span_data (also called within enrich_ai_event_data) and only those two unctions are publicly exposed

vgrozdanic · 2025-09-10T12:33:56Z

relay-server/src/services/processor/span/processing.rs

-    if let Some(model_costs_config) = ai_model_costs {
-        extract_ai_data(span, model_costs_config);
-    }
+    enrich_ai_span_data(span, ai_model_costs, ai_operation_type_map);


this is a benefit of the refactor. next time we want to extend AI enrichment, we only need to change enrich_ai_span_data function, and not worry about updating all of the other places where it could be called

jjbayer

Pressing request changes because of the .unwrap().

jjbayer · 2025-09-10T12:48:26Z

relay-event-normalization/src/normalize/mod.rs

+        let exact_key = Pattern::new(span_op).ok()?;
+        if self.operation_types.contains_key(&exact_key) {
+            return self.operation_types.get(&exact_key).map(String::as_str);
+        }


I don't understand this part. Why would we set the type to the key of the hash map instead of the value?

The value of the hashmap is String, so if possible, we do only a hash lookup with Pattern value of span_op, if not we will iterate over all of the keys to search for a fallback.

I don't really understand the question: "Why would we set the type to the key of the hash map instead of the value?"

If it helps and adds more context why it is done like this, we do the similar thing for model cost hash map:

relay/relay-event-normalization/src/normalize/mod.rs

Line 264 in 38e6099

pub fn cost_per_token(&self, model_id: &str) -> Option<ModelCostV2> {

We can do this without creating a Pattern for the key if you add a Borrow implementation to Pattern:

impl Borrow<str> for Pattern { fn borrow(&self) -> &str { &self.pattern } }

From the docs:

The key may be any borrowed form of the map’s key type, but Hash and Eq on the borrowed form must match those for the key type.

Suggested change

let exact_key = Pattern::new(span_op).ok()?;

if self.operation_types.contains_key(&exact_key) {

return self.operation_types.get(&exact_key).map(String::as_str);

}

if let Some(value) = self.operation_types.get(span_op) {

return Some(value.as_str());

}

Same for cost_per_token.

Did it for get_operation_type, for cost_per_token i'll do it as a follow up because i need to change a bit more code (the reference will be returned instead the owned object), and it will have nicer commit associated with it

relay-event-normalization/src/normalize/mod.rs

jjbayer · 2025-09-10T12:51:24Z

relay-event-normalization/src/normalize/span/ai.rs

+        return;
+    };
+
+    if let Some(operation_type) = operation_type_map.get_operation_type(span.op.value().unwrap()) {


This could panic because of the .unwrap().

good catch, while i was debugging this, i just put unwrap to quickly iterate on it 😬

Removed it now

jjbayer

We can optimize a little (see comment) but overall this looks good!

jjbayer · 2025-09-11T07:19:59Z

relay-event-normalization/src/normalize/mod.rs

+        let exact_key = Pattern::new(span_op).ok()?;
+        if self.operation_types.contains_key(&exact_key) {
+            return self.operation_types.get(&exact_key).map(String::as_str);
+        }


We can do this without creating a Pattern for the key if you add a Borrow implementation to Pattern:

impl Borrow<str> for Pattern { fn borrow(&self) -> &str { &self.pattern } }

From the docs:

The key may be any borrowed form of the map’s key type, but Hash and Eq on the borrowed form must match those for the key type.

Suggested change

let exact_key = Pattern::new(span_op).ok()?;

if self.operation_types.contains_key(&exact_key) {

return self.operation_types.get(&exact_key).map(String::as_str);

}

if let Some(value) = self.operation_types.get(span_op) {

return Some(value.as_str());

}

Same for cost_per_token.

jjbayer

Please check comments before merging.

relay-event-normalization/src/normalize/span/ai.rs

relay-server/src/services/processor.rs

relay-server/src/services/processor/span/processing.rs

Based on the comment (#5129 (comment)) this PR improves the code for AI cost calculation.

vgrozdanic force-pushed the vg/map-operation-type branch from cb4caba to 2ceb394 Compare September 10, 2025 12:07

vgrozdanic commented Sep 10, 2025

View reviewed changes

relay-event-normalization/src/normalize/mod.rs Show resolved Hide resolved

vgrozdanic force-pushed the vg/map-operation-type branch from 2ceb394 to 129ad7a Compare September 10, 2025 12:22

vgrozdanic commented Sep 10, 2025

View reviewed changes

vgrozdanic force-pushed the vg/map-operation-type branch from 129ad7a to a5ea007 Compare September 10, 2025 12:29

feat(ai): Infer gen_ai_operation_type from span.op

7c18e64

vgrozdanic force-pushed the vg/map-operation-type branch from a5ea007 to 7c18e64 Compare September 10, 2025 12:31

vgrozdanic commented Sep 10, 2025

View reviewed changes

vgrozdanic marked this pull request as ready for review September 10, 2025 12:34

vgrozdanic requested a review from a team as a code owner September 10, 2025 12:34

This comment was marked as outdated.

Sign in to view

jjbayer requested changes Sep 10, 2025

View reviewed changes

remove unwrap

32a9914

This comment was marked as outdated.

Sign in to view

fix lint

a78afb3

vgrozdanic requested a review from jjbayer September 10, 2025 13:35

jjbayer reviewed Sep 11, 2025

View reviewed changes

make it more performant

5b774c2

vgrozdanic requested a review from jjbayer September 11, 2025 09:10

jjbayer approved these changes Sep 11, 2025

View reviewed changes

relay-event-normalization/src/normalize/span/ai.rs Outdated Show resolved Hide resolved

relay-server/src/services/processor.rs Outdated Show resolved Hide resolved

relay-server/src/services/processor/span/processing.rs Outdated Show resolved Hide resolved

feedback

beae00d

vgrozdanic added this pull request to the merge queue Sep 11, 2025

Merged via the queue into master with commit fee8f0f Sep 11, 2025
28 checks passed

vgrozdanic deleted the vg/map-operation-type branch September 11, 2025 11:38

vgrozdanic mentioned this pull request Sep 11, 2025

ref(ai): Improve AI cost calculation code #5132

Merged

github-merge-queue bot pushed a commit that referenced this pull request Sep 12, 2025

ref(ai): Improve AI cost calculation code (#5132)

588ed67

Based on the comment (#5129 (comment)) this PR improves the code for AI cost calculation.

vgrozdanic mentioned this pull request Oct 15, 2025

fix(ai-insights): llm calls count getsentry/sentry#101509

Merged

feat(ai): Infer gen_ai_operation_type from span.op #5129

feat(ai): Infer gen_ai_operation_type from span.op #5129

Uh oh!

Conversation

vgrozdanic commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Implementation plan

Uh oh!

linear bot commented Sep 10, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

jjbayer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vgrozdanic Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jjbayer Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

jjbayer left a comment

Choose a reason for hiding this comment

Uh oh!

jjbayer Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jjbayer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vgrozdanic commented Sep 10, 2025 •

edited

Loading

vgrozdanic Sep 10, 2025 •

edited

Loading

jjbayer Sep 11, 2025 •

edited

Loading

jjbayer Sep 11, 2025 •

edited

Loading