From d930b186338c6ba2a8382f5e4137d6c826d324c0 Mon Sep 17 00:00:00 2001 From: paomian Date: Mon, 19 May 2025 16:42:10 +0800 Subject: [PATCH] chore: copy otel collector docs to 0.14 --- .../greptimecloud/integrations/otlp.md | 70 +++++++++-- .../for-observability/opentelemetry.md | 117 ++++++++++++++---- .../greptimecloud/integrations/otlp.md | 68 +++++++++- .../for-observability/opentelemetry.md | 111 +++++++++++++---- 4 files changed, 308 insertions(+), 58 deletions(-) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-0.14/greptimecloud/integrations/otlp.md b/i18n/zh/docusaurus-plugin-content-docs/version-0.14/greptimecloud/integrations/otlp.md index 66a19d44d..484ed6241 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-0.14/greptimecloud/integrations/otlp.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-0.14/greptimecloud/integrations/otlp.md @@ -11,10 +11,10 @@ description: 介绍如何通过 OpenTelemetry Protocol (OTLP) 将指标数据发 要通过 OpenTelemetry SDK 库将 OpenTelemetry 指标发送到 GreptimeDB，请使用以下信息： -* URL: `https:///v1/otlp/v1/metrics` -* Headers: - * `X-Greptime-DB-Name`: `` - * `Authorization`: `Basic` 认证，是 `:` 的 Base64 编码字符串。更多信息，请参阅 HTTP API 中的[认证](https://docs.greptime.cn/nightly/user-guide/protocols/http#鉴权)。 +- URL: `https:///v1/otlp/v1/metrics` +- Headers: + - `X-Greptime-DB-Name`: `` + - `Authorization`: `Basic` 认证，是 `:` 的 Base64 编码字符串。更多信息，请参阅 HTTP API 中的[认证](https://docs.greptime.cn/nightly/user-guide/protocols/http#鉴权)。 由于请求中使用二进制 protobuf 编码的 payload，因此需要使用支持 `HTTP/protobuf` 的包。例如，在 Node.js 中，可以使用 [`exporter-trace-otlp-proto`](https://www.npmjs.com/package/@opentelemetry/exporter-trace-otlp-proto)；在 Go 中，可以使用 [`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`](https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp)；在 Java 中，可以使用 [`io.opentelemetry:opentelemetry-exporter-otlp`](https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-exporter-otlp)；在 Python 中，可以使用 [`opentelemetry-exporter-otlp-proto-http`](https://pypi.org/project/opentelemetry-exporter-otlp-proto-http/)。 @@ -24,10 +24,66 @@ description: 介绍如何通过 OpenTelemetry Protocol (OTLP) 将指标数据发 ## OpenTelemetry Collector -如果你使用单独的 OTel 收集器，我们推荐更加成熟的 [Grafana -Alloy](https://grafana.com/docs/alloy/latest/) +OpenTelemetry Collector 是 OpenTelemetry 的一个与厂商无关的实现，下面是一个导出到 GreptimeDB 的示例配置。你可以使用 [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) 将指标、日志和追踪数据发送到 GreptimeDB。 -一个简单输出到 GreptimeDB 的配置例子： +```yaml +extensions: + basicauth/client: + client_auth: + username: + password: + +receivers: + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 + http: + endpoint: 0.0.0.0:4318 + +exporters: + otlphttp/traces: + endpoint: 'https:///v1/otlp' + auth: + authenticator: basicauth/client + headers: + x-greptime-db-name: '' + x-greptime-pipeline-name: 'greptime_trace_v1' + otlphttp/logs: + endpoint: 'https:///v1/otlp' + auth: + authenticator: basicauth/client + headers: + x-greptime-db-name: '' + # x-greptime-log-table-name: "" + + otlphttp/metrics: + endpoint: 'https:///v1/otlp' + auth: + authenticator: basicauth/client + headers: + x-greptime-db-name: '' + +service: + extensions: [basicauth/client] + pipelines: + traces: + receivers: [otlp] + exporters: [otlphttp/traces] + logs: + receivers: [otlp] + exporters: [otlphttp/logs] + metrics: + receivers: [otlp] + exporters: [otlphttp/metrics] +``` + + +## Grafana Alloy + +如果你更倾向于使用 [Grafana Alloy](https://grafana.com/docs/alloy/latest/) 的 OpenTelemetry 导出器，可以使用如下配置来发送你的数据。 + +一个简单的配置示例如下： ``` otelcol.exporter.otlphttp "greptimedb" { diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-0.14/user-guide/ingest-data/for-observability/opentelemetry.md b/i18n/zh/docusaurus-plugin-content-docs/version-0.14/user-guide/ingest-data/for-observability/opentelemetry.md index 4bbf26b2b..cb3711947 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-0.14/user-guide/ingest-data/for-observability/opentelemetry.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-0.14/user-guide/ingest-data/for-observability/opentelemetry.md @@ -21,10 +21,10 @@ GreptimeDB 通过原生支持 [OTLP/HTTP](https://opentelemetry.io/docs/specs/ot 使用下面的信息通过 Opentelemetry SDK 库发送 Metrics 到 GreptimeDB： -* URL: `https:///v1/otlp/v1/metrics` -* Headers: - * `X-Greptime-DB-Name`: `` -* `Authorization`: `Basic` 认证，是 `:` 的 Base64 编码字符串。更多信息请参考 [鉴权](https://docs.greptime.cn/user-guide/deployments/authentication/static/) 和 [HTTP API](https://docs.greptime.cn/user-guide/protocols/http#authentication)。 +- URL: `https:///v1/otlp/v1/metrics` +- Headers: + - `X-Greptime-DB-Name`: `` +- `Authorization`: `Basic` 认证，是 `:` 的 Base64 编码字符串。更多信息请参考 [鉴权](https://docs.greptime.cn/user-guide/deployments/authentication/static/) 和 [HTTP API](https://docs.greptime.cn/user-guide/protocols/http#authentication)。 请求中使用 binary protobuf 编码 payload，因此你需要使用支持 `HTTP/protobuf` 的包。例如，在 Node.js 中，可以使用 [`exporter-trace-otlp-proto`](https://www.npmjs.com/package/@opentelemetry/exporter-trace-otlp-proto)；在 Go 中，可以使用 [`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`](https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp)；在 Java 中，可以使用 [`io.opentelemetry:opentelemetry-exporter-otlp`](https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-exporter-otlp)；在 Python 中，可以使用 [`opentelemetry-exporter-otlp-proto-http`](https://pypi.org/project/opentelemetry-exporter-otlp-proto-http/)。 @@ -45,12 +45,12 @@ GreptimeDB 通过原生支持 [OTLP/HTTP](https://opentelemetry.io/docs/specs/ot ```ts const auth = Buffer.from(`${username}:${password}`).toString('base64') const exporter = new OTLPMetricExporter({ - url: `https://${dbHost}/v1/otlp/v1/metrics`, - headers: { - Authorization: `Basic ${auth}`, - "X-Greptime-DB-Name": db, - }, - timeoutMillis: 5000, + url: `https://${dbHost}/v1/otlp/v1/metrics`, + headers: { + Authorization: `Basic ${auth}`, + 'X-Greptime-DB-Name': db, + }, + timeoutMillis: 5000, }) ``` @@ -122,7 +122,7 @@ OTLP 指标数据模型按照下方的规则被映射到 GreptimeDB 数据模型 - 所有的 Attribute，包含 resource 级别、scope 级别和 data_point 级别，都被作为 GreptimeDB 表的 tag 列。 - 数据点的时间戳被作为 GreptimeDB 的时间戳索引，列名 `greptime_timestamp`。 - Gauge/Sum 两种类型的数据点数据被作为 field 列，列名 `greptime_value`。 -- Summary 类型的每个 quantile 被作为单独的数据列，列名 `greptime_pxx`，其中 xx 是 quantile 的数据，如 90 / 99 等。 +- Summary 类型的每个 quantile 被作为单独的数据列，列名 `greptime_pxx`，其中 xx 是 quantile 的数据，如 90 / 99 等。 - Histogram 和 ExponentialHistogram 暂时未被支持，我们可能在后续版本中推出 Histogram 数据类型来原生支持这两种类型。 ## Logs @@ -133,14 +133,14 @@ GreptimeDB 是能够通过 [OTLP/HTTP](https://opentelemetry.io/docs/specs/otlp/ 要通过 OpenTelemetry SDK 库将 OpenTelemetry 日志发送到 GreptimeDB，请使用以下信息： -* **URL:** `https:///v1/otlp/v1/logs` -* **Headers:** - * `X-Greptime-DB-Name`: `` - * `Authorization`: `Basic` 认证，这是一个 Base64 编码的 `:` 字符串。更多信息，请参考 [鉴权](/user-guide/deployments/authentication/static.md) 和 [HTTP API](/user-guide/protocols/http.md#鉴权)。 - * `X-Greptime-Log-Table-Name`: ``（可选）- 存储日志的表名。如果未提供，默认表名为 `opentelemetry_logs`。 - * `X-Greptime-Log-Extract-Keys`: ``（可选）- 从属性中提取对应 key 的值到表的顶级字段。key 应以逗号（`,`）分隔。例如，`key1,key2,key3` 将从属性中提取 `key1`、`key2` 和 `key3`，并将它们提升到日志的顶层，设置为标签。如果提取的字段类型是数组、浮点数或对象，将返回错误。如果提供了 pipeline name，此设置将被忽略。 - * `X-Greptime-Log-Pipeline-Name`: ``（可选）- 处理日志的 pipeline 名称。如果未提供，将使用 `X-Greptime-Log-Extract-Keys` 来处理日志。 - * `X-Greptime-Log-Pipeline-Version`: ``（可选）- 处理日志的 pipeline 的版本。如果未提供，将使用 pipeline 的最新版本。 +- **URL:** `https:///v1/otlp/v1/logs` +- **Headers:** + - `X-Greptime-DB-Name`: `` + - `Authorization`: `Basic` 认证，这是一个 Base64 编码的 `:` 字符串。更多信息，请参考 [鉴权](/user-guide/deployments/authentication/static.md) 和 [HTTP API](/user-guide/protocols/http.md#鉴权)。 + - `X-Greptime-Log-Table-Name`: ``（可选）- 存储日志的表名。如果未提供，默认表名为 `opentelemetry_logs`。 + - `X-Greptime-Log-Extract-Keys`: ``（可选）- 从属性中提取对应 key 的值到表的顶级字段。key 应以逗号（`,`）分隔。例如，`key1,key2,key3` 将从属性中提取 `key1`、`key2` 和 `key3`，并将它们提升到日志的顶层，设置为标签。如果提取的字段类型是数组、浮点数或对象，将返回错误。如果提供了 pipeline name，此设置将被忽略。 + - `X-Greptime-Log-Pipeline-Name`: ``（可选）- 处理日志的 pipeline 名称。如果未提供，将使用 `X-Greptime-Log-Extract-Keys` 来处理日志。 + - `X-Greptime-Log-Pipeline-Version`: ``（可选）- 处理日志的 pipeline 的版本。如果未提供，将使用 pipeline 的最新版本。 请求使用二进制 protobuf 编码负载，因此您需要使用支持 `HTTP/protobuf` 的包。 @@ -152,7 +152,8 @@ GreptimeDB 是能够通过 [OTLP/HTTP](https://opentelemetry.io/docs/specs/otlp/ ### 示例代码 -请参考 [Alloy 文档](alloy.md#日志)中的示例代码，了解如何将 OpenTelemetry 日志发送到 GreptimeDB。 +请参考 [opentelemetry-collector](#opentelemetry-collector) 中的示例代码，里面包含了如何将 OpenTelemetry 日志发送到 GreptimeDB。 +也可参考 [Alloy 文档](alloy.md#日志)中的示例代码，了解如何将 OpenTelemetry 日志发送到 GreptimeDB。 ### 数据模型 @@ -200,15 +201,15 @@ GreptimeDB 支持直接写入 OpenTelemetry 协议的 traces 数据，并内置 要通过 OpenTelemetry SDK 库将 OpenTelemetry 的 traces 数据发送到 GreptimeDB，请使用以下信息： -* URL: `http{s}:///v1/otlp/v1/traces` -* Headers: headers 与 [Logs](#Logs) 部分相同，你可以参考 [Logs](#Logs) 部分获取更多信息。 +- URL: `http{s}:///v1/otlp/v1/traces` +- Headers: headers 与 [Logs](#Logs) 部分相同，你可以参考 [Logs](#Logs) 部分获取更多信息。 默认地，GreptimeDB 会将 traces 数据写入到 `public` 数据库中的 `opentelemetry_traces` 表中。如果想要将 traces 数据写入到不同的表中，你可以使用 `X-Greptime-DB-Name` 和 `X-Greptime-Log-Table-Name` 头部信息来指定数据库和表名。 GreptimeDB 会接受 **protobuf 编码的 traces 数据** 通过 **HTTP 协议** 发送，其中对 HTTP header 有如下要求： -- `content-type` 应配置为 `application/x-protobuf`； -- `x-greptime-pipeline-name` 应配置为 `greptime_trace_v1`； +- `content-type` 应配置为 `application/x-protobuf`； +- `x-greptime-pipeline-name` 应配置为 `greptime_trace_v1`； ### 示例代码 @@ -255,3 +256,71 @@ OTLP traces 数据模型根据以下规则映射到 GreptimeDB 数据模型： ### Append 模式 通过此接口创建的表，默认为[Append 模式](/user-guide/administration/design-table.md#何时使用-append-only-表). + +# 使用 OpenTelemetry Collector 将数据发送到 GreptimeDB + +OpenTelemetry Collector 是一个可扩展的服务，用于接收、处理和导出 OpenTelemetry 数据。它可以作为数据的中间层，将数据从不同的源发送到 GreptimeDB。 +以下是使用 OpenTelemetry Collector 将数据发送到 GreptimeDB 的配置示例。 + +```yaml +extensions: + basicauth/client: + client_auth: + username: + password: + +receivers: + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 + http: + endpoint: 0.0.0.0:4318 + +exporters: + otlphttp/traces: + endpoint: 'http://127.0.0.1:4000/v1/otlp' + # auth: + # authenticator: basicauth/client + headers: + # x-greptime-db-name: '' + x-greptime-pipeline-name: 'greptime_trace_v1' + tls: + insecure: true + otlphttp/logs: + endpoint: 'http://127.0.0.1:4000/v1/otlp' + # auth: + # authenticator: basicauth/client + headers: + # x-greptime-db-name: '' + # x-greptime-log-table-name: "" + tls: + insecure: true + + otlphttp/metrics: + endpoint: 'http://127.0.0.1:4000/v1/otlp' + # auth: + # authenticator: basicauth/client + headers: + # x-greptime-db-name: '' + tls: + insecure: true + +service: + # extensions: [basicauth/client] + pipelines: + traces: + receivers: [otlp] + exporters: [otlphttp/traces] + logs: + receivers: [otlp] + exporters: [otlphttp/logs] + metrics: + receivers: [otlp] + exporters: [otlphttp/metrics] +``` + +在上面的配置中，我们定义了一个接收器 `otlp`，它可以接收来自 OpenTelemetry 的数据。我们还定义了三个导出器 `otlphttp/traces`、`otlphttp/logs` 和 `otlphttp/metrics`，它们将数据发送到 GreptimeDB 的 OTLP 路径。 +在 otlphttp 协议的基础上，我们增加了一些 header 用来指定一些参数，比如 `x-greptime-pipeline-name` 和 `x-greptime-log-table-name`，`x-greptime-pipeline-name` 用来指定要使用的 pipeline 名称，`x-greptime-log-table-name` 用来指定数据将要写入 GreptimeDB 的表名。 +如果你在 GreptimeDB 设置了鉴权。则需要使用 `basicauth/client` 扩展来处理基本的身份验证。 +这里我们强烈建议使用不同的导出器来分别处理 traces、logs 和 metrics 数据，一方面是因为 GreptimeDB 会支持一些特定的 header 来自定义一些处理流程，另一方面也可以做好数据隔离。 diff --git a/versioned_docs/version-0.14/greptimecloud/integrations/otlp.md b/versioned_docs/version-0.14/greptimecloud/integrations/otlp.md index 10f118518..2c19e37be 100644 --- a/versioned_docs/version-0.14/greptimecloud/integrations/otlp.md +++ b/versioned_docs/version-0.14/greptimecloud/integrations/otlp.md @@ -15,10 +15,10 @@ agents. To send OpenTelemetry Metrics to GreptimeDB through OpenTelemetry SDK libraries, use the following information: -* URL: `https:///v1/otlp/v1/metrics` -* Headers: - * `X-Greptime-DB-Name`: `` - * `Authorization`: `Basic` authentication, which is a Base64 encoded string of `:`. For more information, please refer to [Authentication](https://docs.greptime.com/nightly/user-guide/protocols/http#authentication) in HTTP API. +- URL: `https:///v1/otlp/v1/metrics` +- Headers: + - `X-Greptime-DB-Name`: `` + - `Authorization`: `Basic` authentication, which is a Base64 encoded string of `:`. For more information, please refer to [Authentication](https://docs.greptime.com/nightly/user-guide/protocols/http#authentication) in HTTP API. The request uses binary protobuf to encode the payload, so you need to use packages that support `HTTP/protobuf`. For example, in Node.js, you can use [`exporter-trace-otlp-proto`](https://www.npmjs.com/package/@opentelemetry/exporter-trace-otlp-proto); in Go, you can use [`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`](https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp); in Java, you can use [`io.opentelemetry:opentelemetry-exporter-otlp`](https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-exporter-otlp); and in Python, you can use [`opentelemetry-exporter-otlp-proto-http`](https://pypi.org/project/opentelemetry-exporter-otlp-proto-http/). @@ -28,8 +28,64 @@ The package names may change according to OpenTelemetry, so we recommend that yo ## OpenTelemetry Collector -We recommend [Grafana Alloy](https://grafana.com/docs/alloy/latest/) as OTel -collector if you use OpenTelemetry transformation or other advanced features. +OpenTelemetry Collector is a vendor-agnostic implementation of OpenTelemetry, below is a sample configuration for +exporting to GreptimeDB. You can use the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) to send metrics, logs, and traces to GreptimeDB. + +```yaml +extensions: + basicauth/client: + client_auth: + username: + password: + +receivers: + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 + http: + endpoint: 0.0.0.0:4318 + +exporters: + otlphttp/traces: + endpoint: 'https:///v1/otlp' + auth: + authenticator: basicauth/client + headers: + x-greptime-db-name: '' + x-greptime-pipeline-name: 'greptime_trace_v1' + otlphttp/logs: + endpoint: 'https:///v1/otlp' + auth: + authenticator: basicauth/client + headers: + x-greptime-db-name: '' + # x-greptime-log-table-name: "" + + otlphttp/metrics: + endpoint: 'https:///v1/otlp' + auth: + authenticator: basicauth/client + headers: + x-greptime-db-name: '' + +service: + extensions: [basicauth/client] + pipelines: + traces: + receivers: [otlp] + exporters: [otlphttp/traces] + logs: + receivers: [otlp] + exporters: [otlphttp/logs] + metrics: + receivers: [otlp] + exporters: [otlphttp/metrics] +``` + +## Grafana Alloy + +If you prefer to use [Grafana Alloy](https://grafana.com/docs/alloy/latest/)'s OpenTelemetry exporter, you can use the following configuration to send your data. A sample configuration for exporting to GreptimeDB: diff --git a/versioned_docs/version-0.14/user-guide/ingest-data/for-observability/opentelemetry.md b/versioned_docs/version-0.14/user-guide/ingest-data/for-observability/opentelemetry.md index fd86d8f32..8282c86e7 100644 --- a/versioned_docs/version-0.14/user-guide/ingest-data/for-observability/opentelemetry.md +++ b/versioned_docs/version-0.14/user-guide/ingest-data/for-observability/opentelemetry.md @@ -20,10 +20,10 @@ GreptimeDB is an observability backend to consume OpenTelemetry Metrics natively To send OpenTelemetry Metrics to GreptimeDB through OpenTelemetry SDK libraries, use the following information: -* URL: `http{s}:///v1/otlp/v1/metrics` -* Headers: - * `X-Greptime-DB-Name`: `` - * `Authorization`: `Basic` authentication, which is a Base64 encoded string of `:`. For more information, please refer to [Authentication](https://docs.greptime.com/user-guide/deployments/authentication/static/) and [HTTP API](https://docs.greptime.com/user-guide/protocols/http#authentication) +- URL: `http{s}:///v1/otlp/v1/metrics` +- Headers: + - `X-Greptime-DB-Name`: `` + - `Authorization`: `Basic` authentication, which is a Base64 encoded string of `:`. For more information, please refer to [Authentication](https://docs.greptime.com/user-guide/deployments/authentication/static/) and [HTTP API](https://docs.greptime.com/user-guide/protocols/http#authentication) The request uses binary protobuf to encode the payload, so you need to use packages that support `HTTP/protobuf`. For example, in Node.js, you can use [`exporter-trace-otlp-proto`](https://www.npmjs.com/package/@opentelemetry/exporter-trace-otlp-proto); in Go, you can use [`go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`](https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp); in Java, you can use [`io.opentelemetry:opentelemetry-exporter-otlp`](https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-exporter-otlp); and in Python, you can use [`opentelemetry-exporter-otlp-proto-http`](https://pypi.org/project/opentelemetry-exporter-otlp-proto-http/). @@ -44,12 +44,12 @@ Here are some example codes about how to setup the request in different language ```ts const auth = Buffer.from(`${username}:${password}`).toString('base64') const exporter = new OTLPMetricExporter({ - url: `https://${dbHost}/v1/otlp/v1/metrics`, - headers: { - Authorization: `Basic ${auth}`, - "X-Greptime-DB-Name": db, - }, - timeoutMillis: 5000, + url: `https://${dbHost}/v1/otlp/v1/metrics`, + headers: { + Authorization: `Basic ${auth}`, + 'X-Greptime-DB-Name': db, + }, + timeoutMillis: 5000, }) ``` @@ -132,14 +132,14 @@ GreptimeDB consumes OpenTelemetry Logs natively via [OTLP/HTTP](https://opentele To send OpenTelemetry Logs to GreptimeDB through OpenTelemetry SDK libraries, use the following information: -* URL: `http{s}:///v1/otlp/v1/logs` -* Headers: - * `X-Greptime-DB-Name`: `` - * `Authorization`: `Basic` authentication, which is a Base64 encoded string of `:`. For more information, please refer to [Authentication](/user-guide/deployments/authentication/static.md) and [HTTP API](/user-guide/protocols/http.md#authentication). - * `X-Greptime-Log-Table-Name`: `` (optional) - The table name to store the logs. If not provided, the default table name is `opentelemetry_logs`. - * `X-Greptime-Log-Extract-Keys`: `` (optional) - The keys to extract from the attributes. The keys should be separated by commas (`,`). For example, `key1,key2,key3` will extract the keys `key1`, `key2`, and `key3` from the attributes and promote them to the top level of the log, setting them as tags. If the field type is array, float, or object, an error will be returned. If a pipeline is provided, this setting will be ignored. - * `X-Greptime-Log-Pipeline-Name`: `` (optional) - The pipeline name to process the logs. If not provided, the extract keys will be used to process the logs. - * `X-Greptime-Log-Pipeline-Version`: `` (optional) - The pipeline version to process the logs. If not provided, the latest version of the pipeline will be used. +- URL: `http{s}:///v1/otlp/v1/logs` +- Headers: + - `X-Greptime-DB-Name`: `` + - `Authorization`: `Basic` authentication, which is a Base64 encoded string of `:`. For more information, please refer to [Authentication](/user-guide/deployments/authentication/static.md) and [HTTP API](/user-guide/protocols/http.md#authentication). + - `X-Greptime-Log-Table-Name`: `` (optional) - The table name to store the logs. If not provided, the default table name is `opentelemetry_logs`. + - `X-Greptime-Log-Extract-Keys`: `` (optional) - The keys to extract from the attributes. The keys should be separated by commas (`,`). For example, `key1,key2,key3` will extract the keys `key1`, `key2`, and `key3` from the attributes and promote them to the top level of the log, setting them as tags. If the field type is array, float, or object, an error will be returned. If a pipeline is provided, this setting will be ignored. + - `X-Greptime-Log-Pipeline-Name`: `` (optional) - The pipeline name to process the logs. If not provided, the extract keys will be used to process the logs. + - `X-Greptime-Log-Pipeline-Version`: `` (optional) - The pipeline version to process the logs. If not provided, the latest version of the pipeline will be used. The request uses binary protobuf to encode the payload, so you need to use packages that support `HTTP/protobuf`. @@ -151,7 +151,8 @@ For more information about the OpenTelemetry SDK, please refer to the official d ### Example Code -Please refer to the [Alloy documentation](alloy.md#logs) for example code on how to send OpenTelemetry logs to GreptimeDB. +Please refer to the sample code in [opentelemetry-collector](#opentelemetry-collector), which includes how to send OpenTelemetry logs to GreptimeDB. +You can also refer to the sample code in the [Alloy documentation](alloy.md#logs) to learn how to send OpenTelemetry logs to GreptimeDB. ### Data Model @@ -200,8 +201,8 @@ You can use [OpenTelemetry SDK](https://opentelemetry.io/docs/languages/) or oth To send OpenTelemetry traces data to GreptimeDB through OpenTelemetry SDK libraries, please use the following information: -* URL: `http{s}:///v1/otlp/v1/traces` -* Headers: The headers section is the same as the [Logs](#Logs) section, you can refer to the [Logs](#Logs) section for more information. +- URL: `http{s}:///v1/otlp/v1/traces` +- Headers: The headers section is the same as the [Logs](#Logs) section, you can refer to the [Logs](#Logs) section for more information. By default, GreptimeDB will write traces data to the `opentelemetry_traces` table in the `public` database. If you want to write traces data to a different table, you can use the `X-Greptime-DB-Name` and `X-Greptime-Trace-Table-Name` headers to specify the database and table name. @@ -254,3 +255,71 @@ By default, the table is partitioned into 16 uniform regions based on the `trace By default, log table created by OpenTelemetry API are in [append only mode](/user-guide/administration/design-table.md#when-to-use-append-only-tables). + +# Sending Data to GreptimeDB Using OpenTelemetry Collector + +OpenTelemetry Collector is an extensible service for receiving, processing, and exporting OpenTelemetry data. It can act as an intermediate layer to send data from different sources to GreptimeDB. +Below is a sample configuration for sending data to GreptimeDB using OpenTelemetry Collector. + +```yaml +extensions: + basicauth/client: + client_auth: + username: + password: + +receivers: + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 + http: + endpoint: 0.0.0.0:4318 + +exporters: + otlphttp/traces: + endpoint: 'http://127.0.0.1:4000/v1/otlp' + # auth: + # authenticator: basicauth/client + headers: + # x-greptime-db-name: '' + x-greptime-pipeline-name: 'greptime_trace_v1' + tls: + insecure: true + otlphttp/logs: + endpoint: 'http://127.0.0.1:4000/v1/otlp' + # auth: + # authenticator: basicauth/client + headers: + # x-greptime-db-name: '' + # x-greptime-log-table-name: "" + tls: + insecure: true + + otlphttp/metrics: + endpoint: 'http://127.0.0.1:4000/v1/otlp' + # auth: + # authenticator: basicauth/client + headers: + # x-greptime-db-name: '' + tls: + insecure: true + +service: + # extensions: [basicauth/client] + pipelines: + traces: + receivers: [otlp] + exporters: [otlphttp/traces] + logs: + receivers: [otlp] + exporters: [otlphttp/logs] + metrics: + receivers: [otlp] + exporters: [otlphttp/metrics] +``` + +In the above configuration, we define a receiver `otlp` that can receive data from OpenTelemetry. We also define three exporters: `otlphttp/traces`, `otlphttp/logs`, and `otlphttp/metrics`, which send data to the OTLP endpoint of GreptimeDB. +Based on the otlphttp protocol, we have added some headers to specify certain parameters, such as `x-greptime-pipeline-name` and `x-greptime-log-table-name`. The `x-greptime-pipeline-name` header is used to specify the pipeline name to use, and the `x-greptime-log-table-name` header is used to specify the table name in GreptimeDB where the data will be written. +If you have enabled authentication in GreptimeDB, you need to use the `basicauth/client` extension to handle basic authentication. +Here, we strongly recommend using different exporters to handle traces, logs, and metrics separately. On one hand, GreptimeDB supports some specific headers to customize processing flows; on the other hand, this also helps with data isolation.