Airbyte Core: any way to get Airbyte sync metrics into Datadog besides querying the metadata DB directly?

[chanjoong_kim (brian-223134)]

What I'm trying to do

I'm running Airbyte Self-Managed Community (deployed via abctl, Helm chart V2) and want to monitor sync health in Datadog — things like sync success/failure counts, sync duration, and records/bytes moved.

My deployment uses an external Postgres (RDS) for the metadata DB and S3 for storage, so the bulk of useful state lives outside the cluster.

What I've found so far

From the docs and the chart, the picture for Community seems to be:

• OpenTelemetry metrics (Monitoring Airbyte) is flagged Self-Managed Enterprise only (global.edition: enterprise, requires a license). So the airbyte.syncs / airbyte.gb_moved / airbyte.sync_duration metrics aren't available to Community.
• The legacy DogStatsD path (METRIC_CLIENT=datadog / STATSD_HOST etc.) — the Datadog Airbyte integration docs still describe this, but the Airbyte docs now say that integration only works on legacy Docker deployments, not Kubernetes/abctl.
• The chart V2 still renders global.metrics.statsd.* / otlp.collectorEndpoint env vars even on Community, but it's unclear whether Community actually emits anything through them.

So the only path I've actually gotten working is pointing Datadog directly at the metadata DB and writing custom_queries against the internal jobs / attempts / sync_stats tables (plus Datadog's standard Postgres/DBM check). That works, but it relies on undocumented internal schema that could change between versions.

My questions

1. For Community, is querying the metadata DB directly the only supported/intended way to get Airbyte-level sync metrics into Datadog (or any external monitoring tool)?
2. Is the global.metrics (statsd/otlp) configuration in chart V2 expected to do anything on Community, or is metric emission gated behind edition: enterprise?
3. Is there any roadmap for exposing sync metrics (OTel/Prometheus) to Community/Core deployments, even in a limited form?
4. If the metadata-DB approach is the recommended route, is there a stable/public set of tables or views intended for this, so we're not depending on internal schema?

For context, I'm fine with metrics only (I don't need traces). Just trying to confirm I'm not missing an officially supported path before committing to querying internal tables.

Thanks!

[Airbyte Support Bot (airbyte-support-bot)]

Hi chanjoong_kim (@brian-223134), thank you for this thorough and well-researched question!

You have correctly assessed the current state of metrics support across Airbyte editions. Here is a summary to confirm your findings and provide some additional pointers:

1. OpenTelemetry metrics (OTel) are Enterprise-only

As documented in the OpenTelemetry metrics monitoring guide, the airbyte.syncs, airbyte.gb_moved, airbyte.sync_duration, and airbyte.api_requests metrics require global.edition: enterprise. Community deployments do not emit these metrics.

2. Legacy DogStatsD / METRIC_CLIENT=datadog

You are correct that this path is specific to legacy Docker-based deployments and is not supported on Kubernetes/abctl.

3. Helm chart V2 global.metrics on Community

While the chart does render the global.metrics.statsd.* / otlp.collectorEndpoint environment variables, metric emission is gated behind the Enterprise edition check. Setting these values on Community will not produce metrics.

4. Metadata DB approach

For Community deployments, querying the metadata database is currently the primary way to extract sync-level metrics. Airbyte does publish a Database Data Catalog that documents the schema for both the Config and Jobs databases. The relevant tables for sync metrics are:

• jobs — job status, scope (connection ID), timestamps
• attempts — per-attempt status, timestamps, failure summaries
• normalization_summaries — normalization timing and failures

While this catalog is published documentation (not just internal schema), it does not carry explicit stability guarantees across versions.

5. Airbyte API as an alternative

Depending on your needs, the Airbyte API (/api/public/v1/) is available on self-managed Community deployments and includes endpoints for listing jobs and their statuses. This could be used as a polling-based approach to feed metrics into Datadog, and it is a more stable interface than the database schema. Worth evaluating as a complement or alternative to direct DB queries.

We have escalated this to our team for investigation as a feature request — specifically around the possibility of exposing sync metrics (e.g., via OTel or Prometheus) to Community deployments, and/or providing a more stable metrics-oriented API or view for the metadata DB approach.

---

Need more help? Join Airbyte Community Slack for peer support, or if you're a Cloud customer, open a support ticket referencing this URL.

[chanjoong_kim (brian-223134)]

Thank you, this is a really clear and complete answer — it confirms everything and the two extra pointers are exactly what I was missing.

A few notes for anyone who finds this thread later:

• The Database Data Catalog link is the key piece I'd overlooked. Knowing that jobs / attempts are documented (even without explicit cross-version stability guarantees) makes the metadata-DB approach much more comfortable to commit to.
• For now I've gone with the Datadog Postgres integration + custom_queries against jobs / attempts / sync_stats, since it's self-contained in the agent and needs no extra moving parts. It's working well for sync status, duration, and records/bytes moved.
• I'll keep the Public API in mind as the more stable interface to migrate to (via a small poller pushing to Datadog) if the schema ever changes under me — good to know it's available on Community.

Really glad to hear the OTel/Prometheus-for-Community (and/or a stable metrics-oriented API/view) was escalated as a feature request — that would replace this workaround with a first-class path, so big +1 from me. Happy to test or give feedback if that ever lands.

Thanks again!

[Airbyte Support Bot (airbyte-support-bot)]

This has been escalated to our internal team for investigation: https://github.com/airbytehq/oncall/issues/12892