You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Ticket ID:#1847 Priority: normal Group: Community Assistance Engineer Assignee: Sajarin
Original ticket description:
Context
Hello!
We are using Airbyte 0.39.42-alpha with Docker Compose, and are setting it up to send metrics using OpenTelemetry, using information from the following documentation and threads:
According to the documentation, we have updated the Docker Compose stack to:
setup the airbyte-metrics-reporter service for OpenTelemetry
setup the airbyte-worker service for OpenTelemetry
setup the opentelemetry-collector service to handle OTEL gRPC calls, and expose metrics using the Prometheus exporter
Additionally, we have setup:
Prometheus to scrape data from opentelemetry-collector
Grafana to display Prometheus metrics
Issue
When the airbyte-metrics-reporter-service emits metrics using the OpenTelemetry SDK, the following warning can be seen:
airbyte-metrics-reporter | Aug 08, 2022 3:21:29 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
airbyte-metrics-reporter | WARNING: Instrument oldest_running_job_age_secs has recorded multiple values for the same attributes.
airbyte-metrics-reporter | Aug 08, 2022 3:21:29 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
airbyte-metrics-reporter | WARNING: Instrument num_running_jobs has recorded multiple values for the same attributes.
airbyte-metrics-reporter | Aug 08, 2022 3:21:29 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
airbyte-metrics-reporter | WARNING: Instrument oldest_pending_job_age_secs has recorded multiple values for the same attributes.
When sync jobs are running, the gauges corresponding to the number of pending and running jobs do not seem to be updated accordingly, e.g. with two sync jobs running:
I tried bumping the version of the SDK to 1.16 using Airbyte’s deps.toml and rebuilding the Docker image for airbyte-metrics-reporter:
$ git clone https://github.com/airbytehq/airbyte
$ cd airbyte
$ vim deps.toml # set OTEL SDK version to 1.16.0
$ cd airbyte-metrics/reporter
$ ../../gradlew build
but observed the same behaviour: warning messages, gauges stuck to 0.
The following discussion may provide better insights as to why the emission of the latest value fails for Airbyte gauges:
Comment made from Zendesk by Sajarin on 2022-08-12 at 19:39:
Hey @virtualtam, we really appreciate this post. I made an issue relating to your question on Github, please add your thoughts and follow the discussion over there!
This Github issue is synchronized with Zendesk:
Ticket ID: #1847
Priority: normal
Group: Community Assistance Engineer
Assignee: Sajarin
Original ticket description:
The text was updated successfully, but these errors were encountered: