Tracing with OpenTelemetry

[astrojuanlu]

Quick 2 cents

• +1 on setting the default level of the Kedro logger to INFO (if just for backwards compatibility), and then having -q set it to WARNING and -qq to ERROR
  • I don't think the current logging.yml logic is the only way to achieve that though
  • I'm also ambivalent on whether we should change the global logging level to INFO
• -1 on keeping the current logging.yml logic - whoever wants fine grained control of logs, file logging, rotation etc should be using journald, supervisor, Datadog, or whatever other solution. this is not Kedro's responsibility
• +1 on directing users to current best-practices for logging, including creating official integrations and docs for Sentry, OpenTelemetry

Originally posted by @astrojuanlu in #3446 (comment)

Inspired by: mlflow/mlflow#11454

[astrojuanlu]

This is possible and not very hard:

However, there are some loose ends: #3978

[astrojuanlu]

Yesterday a MLE told me that basically everyone expects a production system to have observability, and that it makes no sense that we don't have a native otel integration by now.

[noklam]

What does a native otel integration mean? Does that mean a custom Kedro otel instrumentation, TraceProvider? What exactly is preventing someone to do this already?

[astrojuanlu]

Another platform supporting traces: https://www.traceloop.com/

from traceloop.sdk import Traceloop

Traceloop.init()

...

from traceloop.sdk.decorators import workflow

@workflow(name="suggest_answers")
def suggest_answers(question: str):
  ...

[astrojuanlu]

Another platform with OpenTelemetry integration https://betterstack.com/docs/logs/open-telemetry/

(image from https://betterstack.com/community/guides/logging/json-logging/)

[astrojuanlu]

Another platform built on OpenTelemetry https://signoz.io/docs/

(image from https://signoz.io/blog/json-logs/)