From ab2996716e563f92b2744b5ad14de31825c918e6 Mon Sep 17 00:00:00 2001 From: cgardens Date: Wed, 21 Jun 2023 15:27:24 -0700 Subject: [PATCH] deprecate otel docs --- docs/operator-guides/collecting-metrics.md | 43 ++++++++++++++++++++++ docs/operator-guides/scaling-airbyte.md | 41 --------------------- docs/release_notes/july_2022.md | 2 +- docusaurus/sidebars.js | 1 - 4 files changed, 44 insertions(+), 43 deletions(-) diff --git a/docs/operator-guides/collecting-metrics.md b/docs/operator-guides/collecting-metrics.md index 09406df8e5644..9d210329daa44 100644 --- a/docs/operator-guides/collecting-metrics.md +++ b/docs/operator-guides/collecting-metrics.md @@ -268,3 +268,46 @@ Visit [OssMetricsRegistry.java](https://github.com/airbytehq/airbyte-platform/bl ## Additional information Suppose you are looking for a non-production way of collecting metrics with dbt and Metabase, the tutorial [Airbyte Monitoring with dbt and Metabase](https://airbyte.com/blog/airbyte-monitoring-with-dbt-and-metabase) by accessing Airbyte's Postgres DB. The source code is open on [airbytehq/open-data-stack](https://github.com/airbytehq/open-data-stack). Think of it as an exploratory for data analysts and data engineers of building a dashboard on top of the existing Airbyte Postgres database versus the Prometheus more for DevOps engineers in production. + +# Scaling Airbyte + +## Metrics +Airbyte supports exporting built-in metrics to Datadog or OpenTelemetry. + +### Key Metrics + +| Key Metrics | Description | +|---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------| +| ``oldest_pending_job_age_secs`` | Shows how long a pending job waits before it is scheduled. If a job is in pending state for a long time, more workers may be required. | +| ``oldest_running_job_age_secs`` | Shows how long the oldest job has been running. A running job that is too large can indicate stuck jobs. This is relative to each job’s runtime. | +| ``job_failed_by_release_stage`` | Shows jobs that have failed in that release stage and is tagged as alpha, beta, or GA. | + +:::note + +Metrics with ``by_release_stage`` in their name are tagged by connector release stage (alpha, beta, or GA). These tags allow you to filter by release stage. Alpha and beta connectors are less stable and have a higher failure rate than GA connectors, so filtering by those release stages can help you find failed jobs. + +::: + +### Recommended Metrics + +| Recommended Metrics | Description | +|-----------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------| +| ``num_running_jobs & num_pending_jobs`` | Shows how many jobs are currently running and how many jobs are in pending state. These metrics help you understand the general system state. | +| ``job_succeeded_by_release_stage`` | Shows successful jobs in that release stage and is tagged as alpha, beta, or GA. | +| ``job_created_by_release_stage`` | Shows the jobs created in that release stage and is tagged as alpha, beta, or GA. | + +### Example +If a job was created for an Alpha source to a Beta destination and the outcome of the job is a success, the following metrics are displayed: + +``` +job_created_by_release_stage[“alpha”] = 1; +job_created_by_release_stage[“beta”] = 1; +job_failed_by_release_stage[“alpha”] = 1; +job_succeeded_by_release_stage[“beta”] = 1; +``` + +:::note + +Each job has a source and destination, so each metric is counted twice — once for source and once for destination. + +::: diff --git a/docs/operator-guides/scaling-airbyte.md b/docs/operator-guides/scaling-airbyte.md index 5019757f48609..062cbd33d715f 100644 --- a/docs/operator-guides/scaling-airbyte.md +++ b/docs/operator-guides/scaling-airbyte.md @@ -65,44 +65,3 @@ is capped by `SQL_MAX_CONNS`. The advice here is best-effort and by no means comprehensive. Please reach out on Slack if anything doesn't make sense or if something can be improved. If you've been running Airbyte in production and have more tips up your sleeve, we welcome contributions! - -## Metrics -Airbyte supports exporting built-in metrics to Datadog or [OpenTelemetry](https://docs.airbyte.com/operator-guides/collecting-metrics/) - -### Key Metrics - -| Key Metrics | Description | -|---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------| -| ``oldest_pending_job_age_secs`` | Shows how long a pending job waits before it is scheduled. If a job is in pending state for a long time, more workers may be required. | -| ``oldest_running_job_age_secs`` | Shows how long the oldest job has been running. A running job that is too large can indicate stuck jobs. This is relative to each job’s runtime. | -| ``job_failed_by_release_stage`` | Shows jobs that have failed in that release stage and is tagged as alpha, beta, or GA. | - -:::note - -Metrics with ``by_release_stage`` in their name are tagged by connector release stage (alpha, beta, or GA). These tags allow you to filter by release stage. Alpha and beta connectors are less stable and have a higher failure rate than GA connectors, so filtering by those release stages can help you find failed jobs. - -::: - -### Recommended Metrics - -| Recommended Metrics | Description | -|-----------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------| -| ``num_running_jobs & num_pending_jobs`` | Shows how many jobs are currently running and how many jobs are in pending state. These metrics help you understand the general system state. | -| ``job_succeeded_by_release_stage`` | Shows successful jobs in that release stage and is tagged as alpha, beta, or GA. | -| ``job_created_by_release_stage`` | Shows the jobs created in that release stage and is tagged as alpha, beta, or GA. | - -### Example -If a job was created for an Alpha source to a Beta destination and the outcome of the job is a success, the following metrics are displayed: - -``` -job_created_by_release_stage[“alpha”] = 1; -job_created_by_release_stage[“beta”] = 1; -job_failed_by_release_stage[“alpha”] = 1; -job_succeeded_by_release_stage[“beta”] = 1; -``` - -:::note - -Each job has a source and destination, so each metric is counted twice — once for source and once for destination. - -::: diff --git a/docs/release_notes/july_2022.md b/docs/release_notes/july_2022.md index 0cf73f7d4b7b6..0c6cbc35e0047 100644 --- a/docs/release_notes/july_2022.md +++ b/docs/release_notes/july_2022.md @@ -26,7 +26,7 @@ This page includes new features and improvements to the Airbyte Cloud and Airbyt * Improved Airbyte Open Source self-hosting by refactoring and publishing Helm charts according to best practices as we prepare to formally support Helm deployments. [#14794](https://github.com/airbytehq/airbyte/pull/14794) -* Improved Airbyte Open Source by supporting the [OpenTelemetry (OTEL) Collector](https://docs.airbyte.com/operator-guides/collecting-metrics/). Airbyte Open Source now sends telemetry data to the OTEL collector, and we included a set of [recommended metrics](https://docs.airbyte.com/operator-guides/scaling-airbyte/#metrics) to export to OTEL when running Airbyte Open Source at scale. [#12908](https://github.com/airbytehq/airbyte/issues/12908) +* Improved Airbyte Open Source by supporting the OpenTelemetry (OTEL) Collector. Airbyte Open Source now sends telemetry data to the OTEL collector, and we included a set of [recommended metrics](https://docs.airbyte.com/operator-guides/scaling-airbyte/#metrics) to export to OTEL when running Airbyte Open Source at scale. [#12908](https://github.com/airbytehq/airbyte/issues/12908) * Improved the [Airbyte Connector Development Kit (CDK)](https://airbyte.com/connector-development-kit) by enabling detailed bug logs from the command line. In addition to the preset CDK debug logs, you can also create custom debug statements and display custom debug logs in the command line. [#14521](https://github.com/airbytehq/airbyte/pull/14521) diff --git a/docusaurus/sidebars.js b/docusaurus/sidebars.js index 2680d93080310..0ac2bb40b14b3 100644 --- a/docusaurus/sidebars.js +++ b/docusaurus/sidebars.js @@ -383,7 +383,6 @@ const operatorGuide = { 'operator-guides/using-custom-connectors', 'operator-guides/scaling-airbyte', 'operator-guides/configuring-sync-notifications', - 'operator-guides/collecting-metrics', ], };