Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporter/Logzioexporter] logs support #10821

Merged
merged 28 commits into from
Jun 28, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
da0c5a6
[logzioexporter] logs pipeline support
yotamloe Jun 7, 2022
5262bf2
Update exporters_test.go (CustomEndpoint -> Endpoint)
yotamloe Jun 7, 2022
9ab926f
go mod tidy
yotamloe Jun 7, 2022
53242ef
Merge branch 'main' into logzioexporter-logs-support
yotamloe Jun 7, 2022
f269cd7
Merge branch 'open-telemetry:main' into logzioexporter-logs-support
yotamloe Jun 7, 2022
1c7a4e3
update CHANGELOG.md
yotamloe Jun 7, 2022
9b142d0
go mod tidy
yotamloe Jun 8, 2022
1a59015
Merge branch 'main' into logzioexporter-logs-support
yotamloe Jun 8, 2022
4455867
make goporto + go.opentelemetry.io/collector v0.49.0 -> v0.52.0
yotamloe Jun 8, 2022
c615e17
go.opentelemetry.io/collector v0.52.0 -> v0.52.1-0.20220603175357-6fb…
yotamloe Jun 8, 2022
a0cfc1e
make gotidy
yotamloe Jun 8, 2022
5258a90
Restore legacy config options + `CheckAndWarnDeprecatedOptions` metho…
yotamloe Jun 13, 2022
b4c3fe7
Change warn log messages + CheckAndWarnDeprecatedOptions -> checkAndW…
yotamloe Jun 13, 2022
1f4cf60
Update changelog + Add opentelemetry links in README.md
yotamloe Jun 22, 2022
5e9c426
Rely on http client timeout instead of exporter helper
yotamloe Jun 23, 2022
2df3ee5
Merge branch 'main' into logzioexporter-logs-support
yotamloe Jun 23, 2022
1db79f3
go mod tidy + fix linting issues
yotamloe Jun 23, 2022
86f6fea
Merge remote-tracking branch 'origin/logzioexporter-logs-support' int…
yotamloe Jun 23, 2022
eebe6a6
Merge branch 'main' into logzioexporter-logs-support
yotamloe Jun 23, 2022
6e0e48b
Merge branch 'main' into logzioexporter-logs-support
yotamloe Jun 26, 2022
86c1777
make gotidy
yotamloe Jun 26, 2022
6828b1a
Fix default value of timeout ( 5s -> 30s )
yotamloe Jun 26, 2022
fbfef4c
Merge branch 'main' into logzioexporter-logs-support
yotamloe Jun 28, 2022
f7bf913
Move changes to unreleased (CHANGELOG.md)
yotamloe Jun 28, 2022
f593e93
Restore `replace` statement in go.mod
yotamloe Jun 28, 2022
d58df32
Make unnecessary public variables and functions private
yotamloe Jun 28, 2022
2504cd8
Make gotidy
yotamloe Jun 28, 2022
16704ec
make `CreateTracesExporter` and `CreateLogsExporter` private
yotamloe Jun 28, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,13 @@

### 🚩 Deprecations 🚩

- `logzioexporter`: Announcing `custom_endpoint`, `drain_interval`, `queue_capacity`, `queue_max_length` configuration options will be deprecated in upcoming releases (#10821)

### 🚀 New components 🚀

### 💡 Enhancements 💡

- `logzioexporter`: Add support for logs pipeline and support for exporterhelper (#10821)
- `flinkmetricsreceiver`: add attribute values to metadata #11520
- `prometheusreceiver`: Add `target_info` labels to resource attributes. (#11034)
- `saphanareceiver`: Fix component memory query, add better error handling (#11507)
Expand Down Expand Up @@ -108,6 +111,9 @@
- `couchbasereceiver`: Fully removed unimplemented Couchbase receiver (#10482)
- `hostmetricsreciever`: Fix Load Scraper to normalize 1m, 5m, and 15m averages independently (#8267)

### 🚩 Deprecations 🚩


### 🚀 New components 🚀

- `flinkmetricsreceiver`: Add implementation of Flink Metric Receiver (#10121)
Expand Down
6 changes: 0 additions & 6 deletions cmd/configschema/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,6 @@ require (
github.com/antonmedv/expr v1.9.0 // indirect
github.com/apache/thrift v0.16.0 // indirect
github.com/armon/go-metrics v0.3.10 // indirect
github.com/avast/retry-go v3.0.0+incompatible // indirect
github.com/aws/aws-sdk-go v1.44.43 // indirect
github.com/aws/aws-sdk-go-v2 v1.9.2 // indirect
github.com/aws/aws-sdk-go-v2/config v1.8.3 // indirect
Expand All @@ -78,7 +77,6 @@ require (
github.com/aws/aws-sdk-go-v2/service/sso v1.4.2 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.7.2 // indirect
github.com/aws/smithy-go v1.11.0 // indirect
github.com/beeker1121/goque v2.1.0+incompatible // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/blang/semver v3.5.1+incompatible // indirect
github.com/bmatcuk/doublestar/v3 v3.0.0 // indirect
Expand Down Expand Up @@ -215,8 +213,6 @@ require (
github.com/leoluk/perflib_exporter v0.1.0 // indirect
github.com/lib/pq v1.10.6 // indirect
github.com/linode/linodego v1.5.0 // indirect
github.com/logzio/jaeger-logzio v1.0.6 // indirect
github.com/logzio/logzio-go v1.0.6 // indirect
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
github.com/magiconair/properties v1.8.6 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
Expand All @@ -243,7 +239,6 @@ require (
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f // indirect
github.com/nginxinc/nginx-prometheus-exporter v0.8.1-0.20201110005315-f5a5f8086c19 // indirect
github.com/observiq/ctimefmt v1.0.0 // indirect
github.com/olivere/elastic v6.2.37+incompatible // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/alibabacloudlogserviceexporter v0.54.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awscloudwatchlogsexporter v0.54.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter v0.54.0 // indirect
Expand Down Expand Up @@ -459,7 +454,6 @@ require (
github.com/stretchr/objx v0.4.0 // indirect
github.com/subosito/gotenv v1.2.0 // indirect
github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635 // indirect
github.com/syndtr/goleveldb v1.0.0 // indirect
github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common v1.0.429 // indirect
github.com/tg123/go-htpasswd v1.2.0 // indirect
github.com/tidwall/gjson v1.12.1 // indirect
Expand Down
190 changes: 0 additions & 190 deletions cmd/configschema/go.sum

Large diffs are not rendered by default.

144 changes: 114 additions & 30 deletions exporter/logzioexporter/README.md
Original file line number Diff line number Diff line change
@@ -1,39 +1,115 @@
# Logzio Exporter

| Status | |
| ------------------------ |------------|
| Stability | [stable] |
| Supported pipeline types | traces |
| Distributions | [contrib] |

This exporter supports sending trace data to [Logz.io](https://www.logz.io)

The following configuration options are supported:

* `account_token` (Required): Your logz.io account token for your tracing account.
* `metrics_token` (Optional): This is deprecated, but may be used for the OpenSearch/ElasticSearch based Metrics backend.
* `region` (Optional): Your logz.io account [region code](https://docs.logz.io/user-guide/accounts/account-region.html#available-regions). Defaults to `us`. Required only if your logz.io region is different than US.
* `custom_endpoint` (Optional): Custom endpoint, mostly used for dev or testing. This will override the region parameter.
* `drain_interval` (Optional): Queue drain interval in seconds. Defaults to `3`.
* `queue_capacity` (Optional): Queue capacity in bytes. Defaults to `20 * 1024 * 1024` ~ 20mb.
* `queue_max_length` (Optional): Max number of items allowed in the queue. Defaults to `500000`.

Example:
| Status | |
| ------------------------ | --------------------- |
| Stability | traces [stable] |
| | logs [beta] |
| Supported pipeline types | traces, logs |
| Distributions | [contrib] |

This exporter supports sending trace and log data to [Logz.io](https://www.logz.io)

### The following configuration options are supported:
Logz.io exporter is utilizing opentelemetry [exporter helper](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md) for `retry_on_failure`,`sending_queue` and `timeout` settings
- `account_token` (Required): Your logz.io account token for your tracing or logs account.
- `region` Your logz.io account [region code](https://docs.logz.io/user-guide/accounts/account-region.html#available-regions). Defaults to `us`. Required only if your logz.io region is different than US.
- `endpoint` Custom endpoint, mostly used for dev or testing. This will override the region parameter.
- `retry_on_failure`
- `enabled` (default = true)
- `initial_interval`: Time to wait after the first failure before retrying; ignored if `enabled` is `false` (default = 5s)
- `max_interval`: Is the upper bound on backoff; ignored if `enabled` is `false` (default = 30s)
- `max_elapsed_time`: Is the maximum amount of time spent trying to send a batch; ignored if `enabled` is `false` (default = 300s)
- `sending_queue`
- `enabled` (default = true)
- `num_consumers`: Number of consumers that dequeue batches; ignored if `enabled` is `false` (default = 10)
- `queue_size`: Maximum number of batches kept in memory before dropping; ignored if `enabled` is `false`
User should calculate this as `num_seconds * requests_per_second` where:
- `num_seconds` is the number of seconds to buffer in case of a backend outage
- `requests_per_second` is the average number of requests per seconds.
- default = 5000
- `timeout`: Time to wait per individual attempt to send data to a backend. default = 30s

#### Tracing example:
* We recommend using `batch` processor. Batching helps better compress the data and reduce the number of outgoing connections required to transmit the data.

```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
jaeger:
protocols:
thrift_compact:
endpoint: "0.0.0.0:6831"
thrift_binary:
endpoint: "0.0.0.0:6832"
grpc:
endpoint: "0.0.0.0:14250"
thrift_http:
endpoint: "0.0.0.0:14268"
processors:
batch:
send_batch_size: 10000
timeout: 1s
exporters:
logzio:
logzio/traces:
account_token: "LOGZIOtraceTOKEN"
metrics_token: "LOGZIOmetricsTOKEN"
region: "eu"
drain_interval: 3
queue_capacity: 5000000
queue_max_length: 500000
region: "us"
service:
pipelines:
traces:
receivers: [ otlp,jaeger ]
processors: [ batch ]
exporters: [ logzio/traces ]
telemetry:
logs:
level: "debug"
```
In order to use the Prometheus backend you must use the standard prometheusremotewrite exporter as well. The following [regions](https://docs.logz.io/user-guide/accounts/account-region.html#supported-regions-for-prometheus-metrics) are supported and configured as follows. The Logz.io Listener URL for for your region, configured to use port 8052 for http traffic, or port 8053 for https traffic.
#### Logs example:
* We recommend using `batch` processor. Batching helps better compress the data and reduce the number of outgoing connections required to transmit the data.
* We recommend adding `type` attribute to classify your log records
* We recommend adding `resourcedetection` processor to add metadata to your log records

```yaml
receivers:
filelog:
include: [ "/private/var/log/*.log" ] # MacOs system logs
include_file_name: false
include_file_path: true
operators:
- type: move
from: attributes["log.file.path"]
to: attributes["log_file_path"]
attributes:
type: <<your-logzio-type>>
processors:
batch:
send_batch_size: 10000
timeout: 1s
resourcedetection/system:
detectors: [ "system" ]
system:
hostname_sources: [ "os" ]
exporters:
logzio/logs:
account_token: "LOGZIOlogsTOKEN"
region: "us"
service:
pipelines:
logs:
receivers: [filelog]
processors: [ resourcedetection/system, batch ]
exporters: [logzio/logs]
telemetry:
logs:
level: "debug"
```
#### Metrics:
In order to use the Prometheus backend you must use the standard prometheusremotewrite exporter as well. The following [regions](https://docs.logz.io/user-guide/accounts/account-region.html#supported-regions-for-prometheus-metrics) are supported and configured as follows. The Logz.io Listener URL for for your region, configured to use port 8052 for http traffic, or port 8053 for https traffic.
Example:

```yaml
exporters:
prometheusremotewrite:
Expand All @@ -60,7 +136,7 @@ receivers:
- targets: [ "0.0.0.0:8889" ]

exporters:
logzio:
logzio/traces:
account_token: "LOGZIOtraceTOKEN"
region: "us"

Expand All @@ -69,11 +145,17 @@ exporters:
headers:
Authorization: "Bearer LOGZIOprometheusTOKEN"

processors:
batch:
send_batch_size: 10000
timeout: 1s

service:
pipelines:
traces:
receivers: [jaeger]
exporters: [logzio]
processors: [batch]
exporters: [logzio/traces]

metrics:
receivers: [prometheus]
Expand All @@ -84,5 +166,7 @@ service:
level: debug #activate debug mode
```


[beta]:https://github.com/open-telemetry/opentelemetry-collector#beta
[stable]:https://github.com/open-telemetry/opentelemetry-collector#stable
dashpole marked this conversation as resolved.
Show resolved Hide resolved
[contrib]:https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[contrib]:https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
48 changes: 38 additions & 10 deletions exporter/logzioexporter/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,24 +17,52 @@ package logzioexporter // import "github.com/open-telemetry/opentelemetry-collec
import (
"errors"

"github.com/hashicorp/go-hclog"
"go.opentelemetry.io/collector/config"
"go.opentelemetry.io/collector/config/confighttp"
"go.opentelemetry.io/collector/exporter/exporterhelper"
)

// Config contains Logz.io specific configuration such as Account TracesToken, Region, etc.
type Config struct {
config.ExporterSettings `mapstructure:",squash"`
TracesToken string `mapstructure:"account_token"` // Your Logz.io Account Token, can be found at https://app.logz.io/#/dashboard/settings/general
MetricsToken string `mapstructure:"metrics_token"` // Your Logz.io Metrics Token, can be found at https://docs.logz.io/user-guide/accounts/finding-your-metrics-account-token/
Region string `mapstructure:"region"` // Your Logz.io 2-letter region code, can be found at https://docs.logz.io/user-guide/accounts/account-region.html#available-regions
CustomEndpoint string `mapstructure:"custom_endpoint"` // Custom endpoint to ship traces to. Use only for dev and tests.
DrainInterval int `mapstructure:"drain_interval"` // Queue drain interval in seconds. Defaults to `3`.
QueueCapacity int64 `mapstructure:"queue_capacity"` // Queue capacity in bytes. Defaults to `20 * 1024 * 1024` ~ 20mb.
QueueMaxLength int `mapstructure:"queue_max_length"` // Max number of items allowed in the queue. Defaults to `500000`.
config.ExporterSettings `mapstructure:",squash"` // squash ensures fields are correctly decoded in embedded struct
confighttp.HTTPClientSettings `mapstructure:",squash"` // confighttp client settings https://pkg.go.dev/go.opentelemetry.io/collector/config/confighttp#HTTPClientSettings
exporterhelper.QueueSettings `mapstructure:"sending_queue"` // exporter helper queue settings https://pkg.go.dev/go.opentelemetry.io/collector/exporter/exporterhelper#QueueSettings
exporterhelper.RetrySettings `mapstructure:"retry_on_failure"` // exporter helper retry settings https://pkg.go.dev/go.opentelemetry.io/collector/exporter/exporterhelper#RetrySettings
Token string `mapstructure:"account_token"` // Your Logz.io Account Token, can be found at https://app.logz.io/#/dashboard/settings/general
Region string `mapstructure:"region"` // Your Logz.io 2-letter region code, can be found at https://docs.logz.io/user-guide/accounts/account-region.html#available-regions
CustomEndpoint string `mapstructure:"custom_endpoint"` // **Deprecation** Custom endpoint to ship traces to. Use only for dev and tests.
DrainInterval int `mapstructure:"drain_interval"` // **Deprecation** Queue drain interval in seconds. Defaults to `3`.
QueueCapacity int64 `mapstructure:"queue_capacity"` // **Deprecation** Queue capacity in bytes. Defaults to `20 * 1024 * 1024` ~ 20mb.
QueueMaxLength int `mapstructure:"queue_max_length"` // **Deprecation** Max number of items allowed in the queue. Defaults to `500000`.
}

func (c *Config) validate() error {
if c.TracesToken == "" {
func (c *Config) Validate() error {
if c.Token == "" {
return errors.New("`account_token` not specified")
}
return nil
}

// CheckAndWarnDeprecatedOptions Is checking for soon deprecated configuration options (queue_max_length, queue_capacity, drain_interval, custom_endpoint) log a warning message and map to the relevant updated option
func (c *Config) checkAndWarnDeprecatedOptions(logger hclog.Logger) {
if c.QueueCapacity != 0 {
logger.Warn("You are using the deprecated`queue_capacity` option that will be removed in the next release; use exporter helper configuration instead: https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md")
}
// Warn and map queue_max_length -> QueueSettings.QueueSize
if c.QueueMaxLength != 0 {
logger.Warn("You are using the deprecated `queue_max_length` option that will be removed in the next release; use exporter helper `queue_size` configuration instead: https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md")
c.QueueSettings.Enabled = true
logger.Warn("Mapping `queue_max_length` -> `QueueSettings.QueueSize`")
c.QueueSettings.QueueSize = c.QueueMaxLength
}
if c.DrainInterval != 0 {
logger.Warn("You are using the deprecated `drain_interval` option that will be removed in the next release; use batch processor `timeout` configuration instead: https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor#batch-processor")
}
// Warn and map CustomEndpoint -> Endpoint
if c.CustomEndpoint != "" {
logger.Warn("You are using the deprecated `custom_endpoint` option that will be removed in the next release; please use `endpoint` configuration instead: https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/confighttp")
logger.Warn("Mapping `custom_endpoint` -> `Endpoint`")
c.HTTPClientSettings.Endpoint = c.CustomEndpoint
}
}
Loading