[exporter/Logzioexporter] logs support #10821

yotamloe · 2022-06-07T14:01:50Z

Description:
logzioexporter update:

Add support for logs pipeline
Remove logzio-go and jaeger-logzio dependencies
Add logzioSpan,logzioService,jsonLog in object directiry
Add support for exporter helper configuration options (queue, retry)
Generic export method for traces and logs using HTTP client
Warning logs for soon to be deprecated config options:
- custom_endpoint
- drain_interval
- queue_capacity
- queue_max_length

Testing:
Added tests for logs pipeline and for logzio objects in the following files:

exporter_test.go
config_test.go
factory_test.go
jsonlog_test.go
logziospan_test.go

Passing all unit tests:
coverage: 82% of statements
Documentation:

Added documentation for exporterhelper configuration option
Removed deprecated configuration options (custom_endpoint, drain_interval, queue_capacity, queue_max_length)
Added examples for traces and logs pipeline configuration

dashpole · 2022-06-07T15:46:40Z

It looks like the exporter was marked stable in #10324. Should we be making breaking changes to the configuration?

yotamloe · 2022-06-07T16:00:50Z

Hi, @dashpole thanks for replying.
The breaking changes are only for optional configuration options. We changed them to follow opentelemetry collector best practices (utilize exporterhelper). The struct that we used to represant logzio spans remained the same. (ref), and required configuration options as well.

dashpole · 2022-06-07T17:12:59Z

You might want to consider gating the configuration change with a feature-gate. I did this in the googlecloud exporter when we made changes to our config. See https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/googlecloudexporter/factory.go#L55 for an example of using different configurations based on a feature-gate.

dashpole · 2022-06-07T17:34:44Z

On a separate note, I don't see a CODEOWNERS entry for this component. It seems like you and potentially @jkowall or @Doron-Bargo (who were cc'd on the status header PR) would be reasonable owners?

yotamloe · 2022-06-07T19:26:09Z

Thanks for the advice @dashpole, I'll take a look at that :)
Regarding the code owners, I'll add the relevant people, should I do it in a different PR?

dashpole · 2022-06-07T19:37:12Z

yes, please add owners in a separate PR. If you could have one of the other codeowners review this PR, that would be helpful.

yotamloe · 2022-06-07T20:39:28Z

I checked and looks like this component does have an entry in CODEOWNERS: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/.github/CODEOWNERS#L45
The addition was made in this PR

…884b2dbdc

dashpole · 2022-06-08T13:24:42Z

Ah, sorry. I searched for logsio...

yotamloe · 2022-06-08T13:28:01Z

You might want to consider gating the configuration change with a feature-gate. I did this in the googlecloud exporter when we made changes to our config. See https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/googlecloudexporter/factory.go#L55 for an example of using different configurations based on a feature-gate.

Hi @dashpole :)
Is feature-gate a requirement for this addition? In case we decide to implement it, how long do usually components keep feature gates? (number of releases?)

dashpole · 2022-06-08T13:36:53Z

You wouldn't need to feature-gate the addition of logs support -- just the removal of config options from your stable configuration.

You just have to abide by this, since your component is stable:

Breaking changes, including configuration options and the component's output are not expected to happen without prior notice, unless under special circumstances.

A feature-gate, which is initially disabled by default, is one way to provide prior notice of removal of those options, since users can temporarily (during beta) re-enable them. But if you have a different way you'd like to provide prior notice (e.g. a warning log when using one of the options you plan to remove), that would also be acceptable. You should have at least one release where it is in alpha and one in beta, after which you can remove the feature gate.

yotamloe · 2022-06-08T13:58:40Z

Ok, that makes sense.
So if ill keep the legacy options for the next releases, map the values to the new config options when possible (custom_endpoint -> endpoint for example) and add a warning log for upcoming deprecation and the alternative config options (when the legacy option is used), would that be acceptable?

…d to warn for deprecation + unit test

yotamloe · 2022-06-13T11:24:58Z

Hi @dashpole, I hope you had a nice weekend.
I added the warning messages and option mapping in the latest commit

dashpole

I've reviewed the config + readme changes, which look good. I'll defer to one of the codeowners (who are hopefully more familiar with logzio) to review the log exporter implementation.

exporter/logzioexporter/config.go

dashpole · 2022-06-13T12:59:59Z

I can't, since they don't appear to be org members. Its fine though; they can still review.

…arnDeprecatedOptions

Doron-Bargo

lgtm

yotamloe · 2022-06-21T10:16:12Z

Hi @dashpole.
Anything else we need to move forward and approve this PR?

exporter/logzioexporter/README.md

…o logzioexporter-logs-support # Conflicts: # exporter/logzioexporter/go.mod # exporter/logzioexporter/go.sum

mx-psi · 2022-06-28T09:23:11Z

@yotamloe can you fix the merge conflicts?

mx-psi

This PR adds some new public API, is it intentional? In general, functions, methods and structs should be made private so that we can change the internals of the exporter without making breaking changes.

CHANGELOG.md

exporter/logzioexporter/go.mod

exporter/logzioexporter/logzioservice.go

exporter/logzioexporter/logziospan.go

exporter/logzioexporter/jsonlog.go

yotamloe · 2022-06-28T11:00:35Z

@mx-psi Done 😁

mx-psi

Thanks!! Just a couple more that I missed and we can merge this :)

exporter/logzioexporter/factory.go

* [logzioexporter] logs pipeline support * Update exporters_test.go (CustomEndpoint -> Endpoint) * go mod tidy * update CHANGELOG.md * go mod tidy * make goporto + go.opentelemetry.io/collector v0.49.0 -> v0.52.0 * go.opentelemetry.io/collector v0.52.0 -> v0.52.1-0.20220603175357-6fb884b2dbdc * make gotidy * Restore legacy config options + `CheckAndWarnDeprecatedOptions` method to warn for deprecation + unit test * Change warn log messages + CheckAndWarnDeprecatedOptions -> checkAndWarnDeprecatedOptions * Update changelog + Add opentelemetry links in README.md * Rely on http client timeout instead of exporter helper * go mod tidy + fix linting issues * make gotidy * Fix default value of timeout ( 5s -> 30s ) * Move changes to unreleased (CHANGELOG.md) * Restore `replace` statement in go.mod * Make unnecessary public variables and functions private * Make gotidy * make `CreateTracesExporter` and `CreateLogsExporter` private

yotamloe added 3 commits June 7, 2022 16:22

[logzioexporter] logs pipeline support

da0c5a6

Update exporters_test.go (CustomEndpoint -> Endpoint)

5262bf2

go mod tidy

9ab926f

yotamloe requested a review from a team as a code owner June 7, 2022 14:01

yotamloe requested a review from Aneurysm9 June 7, 2022 14:01

github-actions bot assigned dashpole Jun 7, 2022

yotamloe and others added 3 commits June 7, 2022 17:05

Merge branch 'main' into logzioexporter-logs-support

53242ef

Merge branch 'open-telemetry:main' into logzioexporter-logs-support

f269cd7

update CHANGELOG.md

1c7a4e3

yotamloe and others added 5 commits June 8, 2022 09:59

go mod tidy

9b142d0

Merge branch 'main' into logzioexporter-logs-support

1a59015

make goporto + go.opentelemetry.io/collector v0.49.0 -> v0.52.0

4455867

go.opentelemetry.io/collector v0.52.0 -> v0.52.1-0.20220603175357-6fb…

c615e17

…884b2dbdc

make gotidy

a0cfc1e

yotamloe changed the title ~~Logzioexporter logs support~~ [exporter/Logzioexporter] logs support Jun 8, 2022

yotamloe mentioned this pull request Jun 9, 2022

Implement otel best practices to pass tests logzio/otel-collector-distro#2

Merged

Restore legacy config options + CheckAndWarnDeprecatedOptions metho…

5258a90

…d to warn for deprecation + unit test

dashpole reviewed Jun 13, 2022

View reviewed changes

Change warn log messages + CheckAndWarnDeprecatedOptions -> checkAndW…

b4c3fe7

…arnDeprecatedOptions

yotamloe requested a review from dashpole June 13, 2022 14:17

Doron-Bargo approved these changes Jun 20, 2022

View reviewed changes

dashpole approved these changes Jun 21, 2022

View reviewed changes

exporter/logzioexporter/README.md Show resolved Hide resolved

yotamloe and others added 9 commits June 22, 2022 16:19

Update changelog + Add opentelemetry links in README.md

1f4cf60

Rely on http client timeout instead of exporter helper

5e9c426

Merge branch 'main' into logzioexporter-logs-support

2df3ee5

go mod tidy + fix linting issues

1db79f3

Merge remote-tracking branch 'origin/logzioexporter-logs-support' int…

86f6fea

…o logzioexporter-logs-support # Conflicts: # exporter/logzioexporter/go.mod # exporter/logzioexporter/go.sum

Merge branch 'main' into logzioexporter-logs-support

eebe6a6

Merge branch 'main' into logzioexporter-logs-support

6e0e48b

make gotidy

86c1777

Fix default value of timeout ( 5s -> 30s )

6828b1a

Merge branch 'main' into logzioexporter-logs-support

fbfef4c

mx-psi reviewed Jun 28, 2022

View reviewed changes

yotamloe added 4 commits June 28, 2022 13:02

Move changes to unreleased (CHANGELOG.md)

f7bf913

Restore replace statement in go.mod

f593e93

Make unnecessary public variables and functions private

d58df32

Make gotidy

2504cd8

mx-psi reviewed Jun 28, 2022

View reviewed changes

exporter/logzioexporter/factory.go Outdated Show resolved Hide resolved

make CreateTracesExporter and CreateLogsExporter private

16704ec

mx-psi approved these changes Jun 28, 2022

View reviewed changes

mx-psi merged commit 494f4fc into open-telemetry:main Jun 28, 2022

codeboten mentioned this pull request Jul 14, 2022

Define guideline for components to deprecate configuration open-telemetry/opentelemetry-collector#5686

Closed

yotamloe mentioned this pull request May 9, 2023

REQUEST: New membership for yotamloe open-telemetry/community#1471

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[exporter/Logzioexporter] logs support #10821

[exporter/Logzioexporter] logs support #10821

yotamloe commented Jun 7, 2022 •

edited

dashpole commented Jun 7, 2022

yotamloe commented Jun 7, 2022

dashpole commented Jun 7, 2022

dashpole commented Jun 7, 2022

yotamloe commented Jun 7, 2022

dashpole commented Jun 7, 2022

yotamloe commented Jun 7, 2022

dashpole commented Jun 8, 2022

yotamloe commented Jun 8, 2022

dashpole commented Jun 8, 2022

yotamloe commented Jun 8, 2022

yotamloe commented Jun 13, 2022

dashpole left a comment

dashpole commented Jun 13, 2022

Doron-Bargo left a comment

yotamloe commented Jun 21, 2022

mx-psi commented Jun 28, 2022

mx-psi left a comment

yotamloe commented Jun 28, 2022

mx-psi left a comment

[exporter/Logzioexporter] logs support #10821

[exporter/Logzioexporter] logs support #10821

Conversation

yotamloe commented Jun 7, 2022 • edited

dashpole commented Jun 7, 2022

yotamloe commented Jun 7, 2022

dashpole commented Jun 7, 2022

dashpole commented Jun 7, 2022

yotamloe commented Jun 7, 2022

dashpole commented Jun 7, 2022

yotamloe commented Jun 7, 2022

dashpole commented Jun 8, 2022

yotamloe commented Jun 8, 2022

dashpole commented Jun 8, 2022

yotamloe commented Jun 8, 2022

yotamloe commented Jun 13, 2022

dashpole left a comment

Choose a reason for hiding this comment

dashpole commented Jun 13, 2022

Doron-Bargo left a comment

Choose a reason for hiding this comment

yotamloe commented Jun 21, 2022

mx-psi commented Jun 28, 2022

mx-psi left a comment

Choose a reason for hiding this comment

yotamloe commented Jun 28, 2022

mx-psi left a comment

Choose a reason for hiding this comment

yotamloe commented Jun 7, 2022 •

edited