Enable displaying trace and span id on logs

[BrynCooke]

To enable correlation between trace and logs trace_id and span_id should be included on the log messages.

Json:

{"timestamp":"2024-03-19T15:37:41.516453239Z","level":"INFO","trace_id":"54ac7e5f0e8ab90ae67b822e95ffcbb8","span_id":"9b3f88c602de0ceb","message":"Supergraph GraphQL response".....

Text:

2024-03-19T15:14:46.040435Z INFO trace_id: bbafc3f048b6137375dd78c10df18f50 span_id: 40ede28c5df1b5cc router{

Configuration options have been added in logging for text and json formats defaulting to true for json and false for text.

Json:

   exporters:
     logging:
       stdout:
         format:
           json:
             display_span_id: true
             display_trace_id: true

Text:

  exporters:
    logging:
      stdout:
        format:
          text:
            display_span_id: true
            display_trace_id: true

---

Checklist

Complete the checklist (and note appropriate exceptions) before the PR is marked ready-for-review.

• Changes are compatible¹
• Documentation² completed
• Performance impact assessed and acceptable
• Tests added and passing³
  • Unit Tests
  • Integration Tests
  • Manual Tests

Exceptions

Note any exceptions here

Notes

Footnotes

1. It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this. ↩

2. Configuration is an important part of many changes. Where applicable please try to document configuration examples. ↩

3. Tick whichever testing boxes are applicable. If you are adding Manual Tests, please document the manual testing (extensively) in the Exceptions. ↩

[router-perf]

CI performance tests

• reload - Reload test over a long period of time at a constant rate of users
• events_big_cap_high_rate_callback - Stress test for events with a lot of users, deduplication enabled and high rate event with a big queue capacity using callback mode
• events_without_dedup_callback - Stress test for events with a lot of users and deduplication DISABLED using callback mode
• large-request - Stress test with a 1 MB request payload
• const - Basic stress test that runs with a constant number of users
• no-graphos - Basic stress test, no GraphOS.
• step-jemalloc-tuning - Clone of the basic stress test for jemalloc tuning
• events - Stress test for events with a lot of users and deduplication ENABLED
• events_callback - Stress test for events with a lot of users and deduplication ENABLED in callback mode
• events_big_cap_high_rate - Stress test for events with a lot of users, deduplication enabled and high rate event with a big queue capacity
• events_without_dedup - Stress test for events with a lot of users and deduplication DISABLED
• xxlarge-request - Stress test with 100 MB request payload
• xlarge-request - Stress test with 10 MB request payload
• step - Basic stress test that steps up the number of users over time

[Geal]

this looks fine but do we want to activate by default 2 new high cardinality fields in the logs? Wouldn't that surprise users?

[BrynCooke]

| this looks fine but do we want to activate by default 2 new high cardinality fields in the logs? Wouldn't that surprise users?

I see your point, but for instance New Relic and Datadog expect trace attributes to be on log messages, and the otel spec details that they should be there https://opentelemetry.io/docs/specs/otel/compatibility/logging_trace_context/#json-formats.

I think this'll give a better experience for the majority of users, and for logs carnality doesn't really matter.

[maschwenk]

@BrynCooke trying to understand how this PR interacts with this one:

• Attach dd.trace_id to json formatted log messages #4764

[Samjin]

@BrynCooke How to work with existing trace_id with value that our service stacks already generated from upstream? Is there a way to reassign the existing value to otel generated trace_id so we can avoid multiple trace_ids for all logging/trace/metrics?

eg. I was trying to override trace_id in span with our own value, but it's expecting a boolean only.

[bnjjj]

@Samjin I think you should use trace propagation for this use case https://www.apollographql.com/docs/router/configuration/telemetry/exporters/tracing/overview#propagation

[Samjin]

@Samjin I think you should use trace propagation for this use case https://www.apollographql.com/docs/router/configuration/telemetry/exporters/tracing/overview#propagation

Just tried it out with request header of trace-id, but it doesn't update the default OTEL trace_id value in span attribute and logs. Instead it did the opposite. It updated original request trace-id header and past to downstream.

[Samjin]

@Samjin I think you should use trace propagation for this use case https://www.apollographql.com/docs/router/configuration/telemetry/exporters/tracing/overview#propagation

Create a ticket to override default trace_id. #5056