feat(ourlogs): Add OTLP endpoint for logs #5130

AbhiPrasad · 2025-09-10T18:37:50Z

resolves #5111

Building on top of #5128, this PR adds an OTLP endpoint for logs. Much of this approach was inspired by #4223.

To use this endpoint, you'll need to provide the following environmental variables (where DSN_PROJECT_ID and DSN_PUBLIC_KEY are grabbed from the DSN) when setting up your OTEL setup. An integration test was added to tests/integration/test_otlp_logs.py to show this working.

OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="https://{HOST}/api/{DSN_PROJECT_ID}/otlp/v1/logs" 
OTEL_EXPORTER_OTLP_HEADERS="x-sentry-auth=sentry sentry_key={DSN_PUBLIC_KEY}"

jjbayer

Looks good AFAICT. A test case along the lines of

relay/tests/integration/test_nel.py

Line 8 in b044a69

def test_nel_converted_to_logs(mini_sentry, relay):

would be nice.

relay-server/src/services/processor/otel_logs.rs

jjbayer · 2025-09-12T07:59:37Z

relay-server/src/envelope/item.rs

+            // NOTE: semantically wrong, but too expensive to parse.
+            ItemType::OtelLogsData => smallvec![
+                (DataCategory::LogByte, self.len().max(1)),
+                (DataCategory::LogItem, item_count)


Suggested change

// NOTE: semantically wrong, but too expensive to parse.

ItemType::OtelLogsData => smallvec![

(DataCategory::LogByte, self.len().max(1)),

(DataCategory::LogItem, item_count)

ItemType::OtelLogsData => smallvec![

(DataCategory::LogByte, self.len().max(1)),

(DataCategory::LogItem, item_count) // NOTE: semantically wrong, but too expensive to parse.

Done with 68491f4

jjbayer · 2025-09-12T07:59:52Z

relay-server/src/endpoints/mod.rs

 mod minidump;
 mod monitor;
 mod nel;
+mod otel_log;


Should we rename this otlp_logs and rename traces to otlp_traces to match the endpoint routes?

Agree, done with d30110c

jjbayer · 2025-09-12T08:04:20Z

relay-server/src/services/processor.rs

 mod event;
 mod metrics;
 mod nel;
+mod otel_log;


I would name the helper module log and rename the function to convert_otel_logs to be consistent with:

relay/relay-server/src/services/processor.rs

Line 2214 in ed78b72

span::convert_otel_traces_data(managed_envelope);

I went with ourlog, done with cfe9765

jjbayer · 2025-09-12T08:08:40Z

relay-server/src/services/processor.rs

+                // Convert OTLP logs data to individual log items
+                otel_log::convert_to_logs(&mut managed_envelope);


Question for @Dav1dde for when you're back: Is this something that should move into LogsProcessor::prepare_envelope eventually?

Done in #5136.

Maybe we want to refine this in the future, prepare_envelope was meant to be very very very lightweight, no processing at all. So it has to happen in the expansion. But this is good because we will apply everything we can before the expensive expansion (e.g. if we could inbound filters). It also means we don't have to deserialize -> serialize -> deserialize.

relay-server/src/services/processor/otel_log.rs

jjbayer · 2025-09-12T08:16:25Z

relay-server/src/services/processor/otel_log.rs

+    if let Ok(()) = ItemContainer::from(logs).write_to(&mut item) {
+        envelope.envelope_mut().add_item(item);
+    }


I see this was copied from nel, but I would do at least .inspect_err() here. See

relay/relay-server/src/processing/spans/mod.rs

Lines 311 to 313 in ed78b72

ItemContainer::from(self.spans)

.write_to(&mut item)

.inspect_err(|err| relay_log::error!("failed to serialize spans: {err}"))?;

Good suggestion! I elected to pattern match and track a DiscardReason outcome. 334df53

jjbayer · 2025-09-12T08:30:39Z

relay-server/src/envelope/item.rs

            ItemType::Span => true,
            ItemType::OtelSpan => true,
            ItemType::OtelTracesData => true,
+            ItemType::OtelLogsData => true,


I believe this should be false (also OtelTracesData in the line above). The otel types are already custom container types, so I don't believe they will ever be nested within an ItemContainer. cc @Dav1dde

Yeah I had thought about this too, but I elected to follow the existing pattern with OtelTracesData in case there was some part of the processing pipeline that relied on this (or future work that was planned).

Changed with 8f451d7

Yup correct, this should be false. The otel format is already a container 👍

jjbayer

Please keep us in the loop when you roll out the feature flag, & check if all the necessary ops changes were made (see traces).

…rocessor (#5136) The implementation in #5130 added the expansion right into the processor service, not the logs processor (e.g. how the old implementation did it #5082). This not only means we have to serialize and deserialize an additional time, it also goes against the design where we want to go towards. This was most likely caused by my 'NEL' hack, which I now also separated out again. We can implement a proper NEL processor once we pick up the NEL work again. The implementation also did not deal with outcomes correctly. This implementation fixes the issues and also introduces `RecordKeeper::modify_by` to track changes in outcome quantities exactly, instead of just `lenient`ly. We can also work towards moving all occurrences of `lenient` to `modify_by` in follow-up PRs.

This is a sanity check integration test to make sure that we are recording outcomes properly for otlp logs. The intial set of integration tests were added in #5130

AbhiPrasad self-assigned this Sep 10, 2025

jjbayer reviewed Sep 11, 2025

View reviewed changes

Base automatically changed from abhi-relay-otel-pkg to master September 11, 2025 11:39

feat(ourlogs): Add OTLP endpoint for logs

9242f66

AbhiPrasad force-pushed the abhi-relay-otel-endpoint branch from b044a69 to 9242f66 Compare September 11, 2025 14:12

AbhiPrasad added 4 commits September 11, 2025 10:29

changelog entry

d690c9f

add integration test

e3473e9

rename file to match what it's doing

8a0fb69

lint fix

10b687c

AbhiPrasad marked this pull request as ready for review September 11, 2025 16:28

AbhiPrasad requested a review from a team as a code owner September 11, 2025 16:28

This comment was marked as outdated.

Sign in to view

seer-by-sentry bot reviewed Sep 11, 2025

View reviewed changes

relay-server/src/services/processor/otel_logs.rs Outdated Show resolved Hide resolved

remove byte size outcome

ed78b72

jjbayer reviewed Sep 12, 2025

View reviewed changes

AbhiPrasad added 5 commits September 12, 2025 10:39

use otlp prefix for files

d30110c

move around comment for env item

68491f4

mark otel env item as not part of item container

8f451d7

namespace ourlog processing code more clearly

cfe9765

better error handling for writing to log envelope

334df53

jjbayer approved these changes Sep 15, 2025

View reviewed changes

AbhiPrasad added this pull request to the merge queue Sep 15, 2025

Merged via the queue into master with commit 6e360df Sep 15, 2025
28 checks passed

AbhiPrasad deleted the abhi-relay-otel-endpoint branch September 15, 2025 18:48

Dav1dde mentioned this pull request Sep 16, 2025

fix(otel-logs): Fix some issues with otel logs, move expansion into processor #5136

Merged

AbhiPrasad mentioned this pull request Sep 19, 2025

test(ourlogs): Add integration test for otlp logs outcomes #5169

Merged

AbhiPrasad mentioned this pull request Sep 25, 2025

Add OTLP adaptor Logflare/logflare#2797

Closed

		// Convert OTLP logs data to individual log items
		otel_log::convert_to_logs(&mut managed_envelope);

	ItemContainer::from(self.spans)
	.write_to(&mut item)
	.inspect_err(\|err\| relay_log::error!("failed to serialize spans: {err}"))?;

feat(ourlogs): Add OTLP endpoint for logs #5130

feat(ourlogs): Add OTLP endpoint for logs #5130

Uh oh!

Conversation

AbhiPrasad commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jjbayer left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Dav1dde Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jjbayer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AbhiPrasad commented Sep 10, 2025 •

edited

Loading

Dav1dde Sep 16, 2025 •

edited

Loading