Add AlibabaCloud LogService exporter #259

shabicheng · 2020-05-26T14:39:08Z

Description:

This PR introduces an exporter for AlibabaCloud LogService. The exporter works by translating spans and metrics into the LogService protobuf format expected by LogService, and sending over HTTP.

The trace model is similar to Jaeger on Aliyun Log Service. The metrics model in LogService can be used as Prometheus's remote adapter.

Testing:

Unit tests added for translating resources, spans, and metrics to the LogService model. Now code coverage is 86.5%.

Manually tested, sending to an LogService's instance.

Documentation:

Added a README, which describes the exporter's config.

…try-collector-contrib

codecov · 2020-05-26T16:05:09Z

Codecov Report

Merging #259 into master will increase coverage by 1.25%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #259      +/-   ##
==========================================
+ Coverage   79.64%   80.90%   +1.25%     
==========================================
  Files         163      169       +6     
  Lines        8338     8887     +549     
==========================================
+ Hits         6641     7190     +549     
  Misses       1343     1343              
  Partials      354      354

Impacted Files	Coverage Δ
exporter/alibabacloudlogserviceexporter/factory.go	`100.00% <100.00%> (ø)`
...alibabacloudlogserviceexporter/metrics_exporter.go	`100.00% <100.00%> (ø)`
...oudlogserviceexporter/metricsdata_to_logservice.go	`100.00% <100.00%> (ø)`
...r/alibabacloudlogserviceexporter/trace_exporter.go	`100.00% <100.00%> (ø)`
...cloudlogserviceexporter/tracedata_to_logservice.go	`100.00% <100.00%> (ø)`
...xporter/alibabacloudlogserviceexporter/uploader.go	`100.00% <100.00%> (ø)`
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9af3381...8dcb469. Read the comment docs.

jrcamp

On shutdown exporters should flush any internal queues to ensure all metrics/traces are sent before returning. Does the producer you're using have those and is there a way of waiting for them to finish if so?

jrcamp · 2020-06-05T21:28:00Z

exporter/alibabacloudlogserviceexporter/README.md

+- `max_retry` (optional): max retry count when send fail.
+- `max_buffer_size` (optional): max buffer used in memory.


@shabicheng Apologies for the delayed review.

Retries and buffers are provided by processors. This ensures that it's handled consistently and that limits are applied globally. See these docs:

https://github.com/open-telemetry/opentelemetry-collector/blob/master/processor/queuedprocessor
https://github.com/open-telemetry/opentelemetry-collector/blob/master/processor/memorylimiter
https://github.com/open-telemetry/opentelemetry-collector/blob/master/processor/batchprocessor

@bogdandrutu please correct me if wrong.

Sending data to LogService with the producer is best practice, the producer will use different retry strategy according to the error code from the server. The max retry count and max buffer size are very important for the producer, so we expose these params.

@tigrannajaryan what are your thoughts? Is it a showstopper if retries and buffers are not being managed by the processors? I assume so given it defeats the purpose of having those things centrally managed but just verifying.

@jrcamp To avoid conceptual conflicts, I have removed the producer and use a normal client instead. And we will use some processors to do retry/batch/memory limit. If we have more requirements for processors, I will submit some processor PRs.

Great! One thing to keep in mind is that if you want to send things in parallel you'll still have to implement that part yourself. grep for NumWorkers to see current examples of it.

Actually you may can just use https://github.com/open-telemetry/opentelemetry-collector/tree/master/processor/queuedprocessor. Can you see if that will work for your use cases?

@jrcamp Thanks for reminding us, we can use queuedprocessor directly. Actually we will use the combination of memorylimiter&batchprocessor&queuedprocessor.

exporter/alibabacloudlogserviceexporter/metrics_exporter.go

exporter/alibabacloudlogserviceexporter/trace_exporter.go

jrcamp · 2020-06-05T22:10:26Z

Also please rebase since things have been delayed.

…try-collector-contrib

linux-foundation-easycla · 2020-06-06T09:47:47Z

The committers are authorized under a signed CLA.

✅ shabicheng (eaec232, 85d8a81, 009f434, 2f2f76b, ae53160)

shabicheng · 2020-06-06T10:54:07Z

On shutdown exporters should flush any internal queues to ensure all metrics/traces are sent before returning. Does the producer you're using have those and is there a way of waiting for them to finish if so?

Thanks for reminding, the producer can gracefully close and I've added shutdown function for the exporter.

shabicheng · 2020-06-06T12:06:02Z

The load test is failed, but it doesn't seem to be related to this PR

jrcamp · 2020-06-10T15:36:30Z

lgtm, I think last thing to address is comment above about if you need to handle parallelism here or not.

…try-collector-contrib

shabicheng · 2020-06-11T03:49:05Z

Now the target code coverage is 95.00%, we will add more tests to achieve the threshold.

jrcamp · 2020-06-11T17:52:02Z

@shabicheng is this ready for merge?

tigrannajaryan · 2020-06-11T18:01:18Z

@shabicheng please run make gotidy to clean go.sum files.

* Add alibabacloud_logservice exporter * refine code for ci * refine code for review commends * upgrade latest producer version * trgger ci * remove producer, use raw logservice client * refine code and add more unittest

* INITIAL * Properly implement name * reword comment * address comments and rename stop to shutdown * Use assert in tests, change to shutdown instead of shutdownFunc and add some missing comments.

* export noop meter * move mock meter to internal/metric package

#259)

shabicheng added 2 commits May 26, 2020 22:10

Add alibabacloud_logservice exporter

eaec232

Merge branch 'master' of https://github.com/open-telemetry/openteleme…

85d8a81

…try-collector-contrib

shabicheng requested a review from a team as a code owner May 26, 2020 14:39

refine code for ci

009f434

shabicheng force-pushed the master branch from 9b1fad5 to 009f434 Compare May 27, 2020 01:13

tigrannajaryan added this to In progress in Collector via automation May 28, 2020

tigrannajaryan assigned jrcamp May 28, 2020

jrcamp reviewed Jun 5, 2020

View reviewed changes

shabicheng added 2 commits June 6, 2020 17:23

refine code for review commends

2f2f76b

Merge branch 'master' of https://github.com/open-telemetry/openteleme…

ae53160

…try-collector-contrib

shabicheng requested a review from a team as a code owner June 6, 2020 09:47

upgrade latest producer version

4b8371e

shabicheng added 2 commits June 9, 2020 09:51

trgger ci

bfe39e3

remove producer, use raw logservice client

1061746

Merge branch 'master' of https://github.com/open-telemetry/openteleme…

e635fb4

…try-collector-contrib

Collector automation moved this from In progress to Reviewer approved Jun 11, 2020

jrcamp approved these changes Jun 11, 2020

View reviewed changes

refine code and add more unittest

8dcb469

bogdandrutu merged commit 5dabcfe into open-telemetry:master Jun 12, 2020

Collector automation moved this from Reviewer approved to Done Jun 12, 2020

ljmsc referenced this pull request in ljmsc/opentelemetry-collector-contrib Feb 21, 2022

move mock tracer to internal/metric (#259)

25e97e5

* export noop meter * move mock meter to internal/metric package

bogdandrutu pushed a commit that referenced this pull request May 12, 2022

fix go.sum entry for sigs.k8s.io/structured-merge-diff/v4 (#259)

ade7580

codeboten pushed a commit that referenced this pull request Nov 23, 2022

Fix requests and urllib instrumentations span name callback parameters (

65801c3

#259)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add AlibabaCloud LogService exporter #259

Add AlibabaCloud LogService exporter #259

shabicheng commented May 26, 2020

codecov bot commented May 26, 2020 •

edited

jrcamp left a comment

jrcamp Jun 5, 2020 •

edited

shabicheng Jun 6, 2020

jrcamp Jun 9, 2020

shabicheng Jun 10, 2020

jrcamp Jun 10, 2020

jrcamp Jun 10, 2020

shabicheng Jun 11, 2020

jrcamp commented Jun 5, 2020

linux-foundation-easycla bot commented Jun 6, 2020 •

edited

shabicheng commented Jun 6, 2020

shabicheng commented Jun 6, 2020

jrcamp commented Jun 10, 2020

shabicheng commented Jun 11, 2020

jrcamp commented Jun 11, 2020

tigrannajaryan commented Jun 11, 2020

		- `max_retry` (optional): max retry count when send fail.
		- `max_buffer_size` (optional): max buffer used in memory.

Add AlibabaCloud LogService exporter #259

Add AlibabaCloud LogService exporter #259

Conversation

shabicheng commented May 26, 2020

codecov bot commented May 26, 2020 • edited

Codecov Report

jrcamp left a comment

Choose a reason for hiding this comment

jrcamp Jun 5, 2020 • edited

Choose a reason for hiding this comment

shabicheng Jun 6, 2020

Choose a reason for hiding this comment

jrcamp Jun 9, 2020

Choose a reason for hiding this comment

shabicheng Jun 10, 2020

Choose a reason for hiding this comment

jrcamp Jun 10, 2020

Choose a reason for hiding this comment

jrcamp Jun 10, 2020

Choose a reason for hiding this comment

shabicheng Jun 11, 2020

Choose a reason for hiding this comment

jrcamp commented Jun 5, 2020

linux-foundation-easycla bot commented Jun 6, 2020 • edited

shabicheng commented Jun 6, 2020

shabicheng commented Jun 6, 2020

jrcamp commented Jun 10, 2020

shabicheng commented Jun 11, 2020

jrcamp commented Jun 11, 2020

tigrannajaryan commented Jun 11, 2020

codecov bot commented May 26, 2020 •

edited

jrcamp Jun 5, 2020 •

edited

linux-foundation-easycla bot commented Jun 6, 2020 •

edited