Skip to content

Commit

Permalink
deprecate otel docs (#27589)
Browse files Browse the repository at this point in the history
  • Loading branch information
cgardens committed Jun 21, 2023
1 parent 77ea559 commit bf52bcf
Show file tree
Hide file tree
Showing 4 changed files with 44 additions and 43 deletions.
43 changes: 43 additions & 0 deletions docs/operator-guides/collecting-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -268,3 +268,46 @@ Visit [OssMetricsRegistry.java](https://github.com/airbytehq/airbyte-platform/bl

## Additional information
Suppose you are looking for a non-production way of collecting metrics with dbt and Metabase, the tutorial [Airbyte Monitoring with dbt and Metabase](https://airbyte.com/blog/airbyte-monitoring-with-dbt-and-metabase) by accessing Airbyte's Postgres DB. The source code is open on [airbytehq/open-data-stack](https://github.com/airbytehq/open-data-stack). Think of it as an exploratory for data analysts and data engineers of building a dashboard on top of the existing Airbyte Postgres database versus the Prometheus more for DevOps engineers in production.

# Scaling Airbyte

## Metrics
Airbyte supports exporting built-in metrics to Datadog or OpenTelemetry.

### Key Metrics

| Key Metrics | Description |
|---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
| ``oldest_pending_job_age_secs`` | Shows how long a pending job waits before it is scheduled. If a job is in pending state for a long time, more workers may be required. |
| ``oldest_running_job_age_secs`` | Shows how long the oldest job has been running. A running job that is too large can indicate stuck jobs. This is relative to each job’s runtime. |
| ``job_failed_by_release_stage`` | Shows jobs that have failed in that release stage and is tagged as alpha, beta, or GA. |

:::note

Metrics with ``by_release_stage`` in their name are tagged by connector release stage (alpha, beta, or GA). These tags allow you to filter by release stage. Alpha and beta connectors are less stable and have a higher failure rate than GA connectors, so filtering by those release stages can help you find failed jobs.

:::

### Recommended Metrics

| Recommended Metrics | Description |
|-----------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|
| ``num_running_jobs & num_pending_jobs`` | Shows how many jobs are currently running and how many jobs are in pending state. These metrics help you understand the general system state. |
| ``job_succeeded_by_release_stage`` | Shows successful jobs in that release stage and is tagged as alpha, beta, or GA. |
| ``job_created_by_release_stage`` | Shows the jobs created in that release stage and is tagged as alpha, beta, or GA. |

### Example
If a job was created for an Alpha source to a Beta destination and the outcome of the job is a success, the following metrics are displayed:

```
job_created_by_release_stage[“alpha”] = 1;
job_created_by_release_stage[“beta”] = 1;
job_failed_by_release_stage[“alpha”] = 1;
job_succeeded_by_release_stage[“beta”] = 1;
```

:::note

Each job has a source and destination, so each metric is counted twice — once for source and once for destination.

:::
41 changes: 0 additions & 41 deletions docs/operator-guides/scaling-airbyte.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,44 +65,3 @@ is capped by `SQL_MAX_CONNS`.
The advice here is best-effort and by no means comprehensive. Please reach out on Slack if anything doesn't make sense or if something can be improved.

If you've been running Airbyte in production and have more tips up your sleeve, we welcome contributions!

## Metrics
Airbyte supports exporting built-in metrics to Datadog or [OpenTelemetry](https://docs.airbyte.com/operator-guides/collecting-metrics/)

### Key Metrics

| Key Metrics | Description |
|---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
| ``oldest_pending_job_age_secs`` | Shows how long a pending job waits before it is scheduled. If a job is in pending state for a long time, more workers may be required. |
| ``oldest_running_job_age_secs`` | Shows how long the oldest job has been running. A running job that is too large can indicate stuck jobs. This is relative to each job’s runtime. |
| ``job_failed_by_release_stage`` | Shows jobs that have failed in that release stage and is tagged as alpha, beta, or GA. |
:::note

Metrics with ``by_release_stage`` in their name are tagged by connector release stage (alpha, beta, or GA). These tags allow you to filter by release stage. Alpha and beta connectors are less stable and have a higher failure rate than GA connectors, so filtering by those release stages can help you find failed jobs.

:::

### Recommended Metrics

| Recommended Metrics | Description |
|-----------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|
| ``num_running_jobs & num_pending_jobs`` | Shows how many jobs are currently running and how many jobs are in pending state. These metrics help you understand the general system state. |
| ``job_succeeded_by_release_stage`` | Shows successful jobs in that release stage and is tagged as alpha, beta, or GA. |
| ``job_created_by_release_stage`` | Shows the jobs created in that release stage and is tagged as alpha, beta, or GA. |

### Example
If a job was created for an Alpha source to a Beta destination and the outcome of the job is a success, the following metrics are displayed:

```
job_created_by_release_stage[“alpha”] = 1;
job_created_by_release_stage[“beta”] = 1;
job_failed_by_release_stage[“alpha”] = 1;
job_succeeded_by_release_stage[“beta”] = 1;
```

:::note

Each job has a source and destination, so each metric is counted twice — once for source and once for destination.

:::
2 changes: 1 addition & 1 deletion docs/release_notes/july_2022.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ This page includes new features and improvements to the Airbyte Cloud and Airbyt

* Improved Airbyte Open Source self-hosting by refactoring and publishing Helm charts according to best practices as we prepare to formally support Helm deployments. [#14794](https://github.com/airbytehq/airbyte/pull/14794)

* Improved Airbyte Open Source by supporting the [OpenTelemetry (OTEL) Collector](https://docs.airbyte.com/operator-guides/collecting-metrics/). Airbyte Open Source now sends telemetry data to the OTEL collector, and we included a set of [recommended metrics](https://docs.airbyte.com/operator-guides/scaling-airbyte/#metrics) to export to OTEL when running Airbyte Open Source at scale. [#12908](https://github.com/airbytehq/airbyte/issues/12908)
* Improved Airbyte Open Source by supporting the OpenTelemetry (OTEL) Collector. Airbyte Open Source now sends telemetry data to the OTEL collector, and we included a set of [recommended metrics](https://docs.airbyte.com/operator-guides/scaling-airbyte/#metrics) to export to OTEL when running Airbyte Open Source at scale. [#12908](https://github.com/airbytehq/airbyte/issues/12908)

* Improved the [Airbyte Connector Development Kit (CDK)](https://airbyte.com/connector-development-kit) by enabling detailed bug logs from the command line. In addition to the preset CDK debug logs, you can also create custom debug statements and display custom debug logs in the command line. [#14521](https://github.com/airbytehq/airbyte/pull/14521)

Expand Down
1 change: 0 additions & 1 deletion docusaurus/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -383,7 +383,6 @@ const operatorGuide = {
'operator-guides/using-custom-connectors',
'operator-guides/scaling-airbyte',
'operator-guides/configuring-sync-notifications',
'operator-guides/collecting-metrics',
],
};

Expand Down

0 comments on commit bf52bcf

Please sign in to comment.