Integrate OpenTelemetry and Prometheus metrics across services

[marcschaeferger]

Community Contribution License Agreement

By creating this pull request, I grant the project maintainers an unlimited,
perpetual license to use, modify, and redistribute these contributions under any terms they
choose, including both the AGPLv3 and the Fossorial Commercial license terms. I
represent that I have the right to grant this license for all contributed content.

Description

This pull request introduces comprehensive OpenTelemetry-based observability and metrics instrumentation for Gerbil, supporting both Prometheus-native and OTel/OTLP metrics export. It includes a pluggable backend architecture, detailed documentation, example configurations for both Prometheus and OTel Collector, and a full test suite for the metrics subsystem. The changes also add new dependencies for OpenTelemetry and Prometheus clients.

Observability and Metrics Architecture:

• Added a pluggable metrics backend system supporting "prometheus", "otel", and "none" (disabled), with clear configuration, validation, and documentation. This enables runtime selection of metrics export method and prepares for future tracing/logging integration. [1] [2]

Documentation and Examples:

• Added docs/observability.md detailing the observability design, configuration (CLI, env vars), metric catalog, and backend selection guidance.
• Updated README.md to summarize observability features and direct users to the new documentation.
• Added example configs: examples/prometheus.yml for Prometheus scraping and examples/otel-collector-config.yaml for OTel Collector pipelines, demonstrating both pull and push-based metric flows. [1] [2]

Testing and Validation:

• Added a comprehensive test suite in internal/metrics/metrics_test.go covering backend initialization, metric recording, and HTTP scraping for Prometheus, as well as validation for noop and invalid backends.

Dependency Updates:

• Updated go.mod to include OpenTelemetry and Prometheus client libraries, plus their transitive dependencies, enabling the new observability features.

How to test?

See docs/observability.md