chore: Refactor OTel concerns in preparation for metric provider

[morgan-wowk]

Refactor OpenTelemetry instrumentation into modular package

Changes:

• Restructured OpenTelemetry code into a dedicated package with clear public API
• Split provider setup and FastAPI instrumentation into separate functions (setup_providers() and instrument_fastapi())
• Added configuration validation and error handling for OpenTelemetry settings
• Implemented provider state checking to conditionally enable instrumentation only when providers are configured
• Created internal modules for shared configuration resolution and provider state management
• Added comprehensive test coverage for all new modules and functions
• Updated main application files to use the new simplified API calls

[morgan-wowk]

• feat: Measure system errors using a counter #127
• feat: Setup metric provider for manual and auto-instrumented measurements #126
• chore!: Refactor OTel exporters to be signal-specific #125
• feat: Set service version on OTel data #124
• chore: Refactor OTel concerns in preparation for metric provider #123 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[yuechao-qin]

Only

[morgan-wowk]

@yuechao-qin Feedback addressed.

[maxy-shpfy]

Question: for local setup, should otel service be run separately? What would happen if no OTEL endpoint is configured (e.g. in case of e2e tests on CI)?

[morgan-wowk]

Great question.

I'll answer in two ways.

For local development, we have this example stack I added and have personally been using to test traces and metrics for OSS: https://github.com/TangleML/tangle/tree/master/examples/observability/otel-jaeger-prometheus

For writing tests, or adding new metrics, here are some important things to know:

• The OTel Python SDK that we consume, and even our FastAPI auto-instrumentation consumes, is intentionally designed so that every action is a NOOP if no providers are set.
  • It means, no exceptions from emitting metrics or opening traces when OTel is not configured.
• We only set providers on our end if they have filled out correctly shaped values for the environment variables.
• We only auto-instrument FastAPI if providers are set.
• For verfiying a metric, or trace, was used, via testing (whether its unit or e2e) that is possible and I'll follow up with instructions and examples in a separate PR. Consider this is a TODO. We will have to manipulate the type of reader / exporter we use at the start of tests.

[morgan-wowk]

Here is an issue: #132

[Ark-kun]

Updated main application files to use the new simplified API calls

Hmm. Are you sure we're simplifying things? It's hard to get more simple than the current 1-liner:

otel_tracing.setup_api_tracing(app)