AWSEMFExporter - Add EMF Exporter to support exporting metrics to AWS CloudWatch #498

shaochengwang · 2020-07-23T18:37:03Z

Description:

This PR introduces an exporter for AWS CloudWatch Service. The exporter works by translating metrics into the Embedded Metric Format which enables you to ingest complex high-cardinality application data in the form of logs and to generate actionable metrics from them.

Testing:

Unit tests covered for emf_exporter, cloudwatchlogs client, pusher, publisher, request handler.
Manually tested the end to end functionality on AWS CloudWatch console.

Documentation:

A README is included. It explains this awsemfexporter's DataConversion, Configuration and AWS Credential.

Notes:
Split the original PR into several pieces. The current one only contains the very basic functionality to get metrics from pipeline, translate them into EMF format and publish to CloudWatch. It doesn't do any batching or queuing. The feature to correlate CloudWatch log group name and CloudWatch metrics namespace is also not included in this version. These features will be added in subsequent PRs.
Although there are 4783 lines added in this PR, most of them are dependencies change and unit tests. The source code change is around 1200 lines.

linux-foundation-easycla · 2020-07-23T18:37:06Z

The committers are authorized under a signed CLA.

✅ Shaocheng Wang (345fd2a)

bogdandrutu

It will be almost impossible for one of the maintainer to review 5000+ lines. Consider to split this PR into more readable PRs, as a rule of thumb 500 lines is considered a large PR

bogdandrutu · 2020-07-23T22:00:26Z

exporter/awsemfexporter/config.go

+	// CloudWatch metrics namespace
+	Namespace string `mapstructure:"namespace"`
+	// CWLogs service endpoint
+	Endpoint string `mapstructure:"endpoint"`


Can you use one of the confignet addresses?

anuraaga

Did a quick skim - agree it'd be good to split this up, perhaps something like

Initialize basic connection to cloudwatch
Add translation of data formats
Wire up translation to basic connection
Add advanced connection features (STS)

Also, this is probably going to happen naturally if going with the above structure but there seems to be an overcoupling of sending logic and translation logic. I think having translation logic separate makes the code easier to reason about, see our translator package for xray exporter https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/master/exporter/awsxrayexporter/translator/sql.go

anuraaga · 2020-07-24T06:42:30Z

exporter/awsemfexporter/README.md

+The following exporter configuration parameters are supported. They mirror and have the same affect as the
+comparable AWS X-Ray Daemon configuration values.


Suggested change

The following exporter configuration parameters are supported. They mirror and have the same affect as the

comparable AWS X-Ray Daemon configuration values.

The following exporter configuration parameters are supported.

Don't think the x-ray daemon is relevant to readers of this file.

anuraaga · 2020-07-24T06:44:53Z

exporter/awsemfexporter/config.go

+// Config defines configuration for AWS EMF exporter.
+type Config struct {
+	configmodels.ExporterSettings `mapstructure:",squash"` // squash ensures fields are correctly decoded in embedded struct.
+	// LogGroupName


Seem to be missing descriptions for these config fields. I think the receiver is a good example for what level of detail we should include here

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/master/receiver/awsxrayreceiver/config.go

anuraaga · 2020-07-24T06:49:24Z

exporter/awsemfexporter/mapWithExpiry/mapWIthExpiry.go

@@ -0,0 +1,52 @@
+package mapWithExpiry


Filename has typo

anuraaga · 2020-07-24T06:51:29Z

exporter/awsemfexporter/publisher/nonblockingfifoqueue.go

+}
+
+func (u *NonBlockingFifoQueue) Dequeue() (interface{}, bool) {
+	u.Lock()


It's not non-blocking if you lock

anuraaga · 2020-07-24T06:56:10Z

exporter/awsemfexporter/publisher/nonblockingfifoqueue.go

+)
+
+// It is a FIFO queue with the functionality that dropping the front if the queue size reaches to the maxSize
+type NonBlockingFifoQueue struct {


Just a thought, I didn't look in detail, but at first glance it seems strange to have a queue in the exporter. Shouldn't we be able to use the pipeline for queueing, batching, etc?

…t exporting metrics to CloudWatch

…t exporting metrics to CloudWatch, Fix unit tests timeout issue

codecov · 2020-08-04T20:52:47Z

Codecov Report

Merging #498 into master will increase coverage by 1.28%.
The diff coverage is 95.43%.

@@            Coverage Diff             @@
##           master     #498      +/-   ##
==========================================
+ Coverage   87.97%   89.25%   +1.28%     
==========================================
  Files         251      260       +9     
  Lines       12012    12700     +688     
==========================================
+ Hits        10568    11336     +768     
+ Misses       1109     1007     -102     
- Partials      335      357      +22

Flag	Coverage Δ
#integration	`75.42% <ø> (?)`
#unit	`88.34% <95.43%> (+0.36%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
exporter/awsemfexporter/metric_translator.go	`92.45% <92.45%> (ø)`
exporter/awsemfexporter/conn.go	`93.47% <93.47%> (ø)`
exporter/awsemfexporter/pusher.go	`96.49% <96.49%> (ø)`
exporter/awsemfexporter/emf_exporter.go	`97.53% <97.53%> (ø)`
exporter/awsemfexporter/cwlog_client.go	`100.00% <100.00%> (ø)`
exporter/awsemfexporter/factory.go	`100.00% <100.00%> (ø)`
...fexporter/handler/request_structuredlog_handler.go	`100.00% <100.00%> (ø)`
...er/awsemfexporter/mapwithexpiry/map_with_expiry.go	`100.00% <100.00%> (ø)`
... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f9082de...4cfefe8. Read the comment docs.

shaochengwang · 2020-08-05T20:47:50Z

Split the original PR into several pieces. The current one only contains the very basic functionality to get metrics from pipeline, translate them into EMF format and publish to CloudWatch. It doesn't include any batching or queuing logic. The feature to correlate CloudWatch log group name and CloudWatch metrics namespace is not included in this version as well. These features will be added in subsequent PRs.
Although there are 4783 lines added in this PR, most of them are dependencies change and unit tests. The source code change is around 1200 lines.

tag @alolita on AWS side for her visibility on this PR.

kbrockhoff · 2020-08-19T22:12:17Z

exporter/awsemfexporter/cwlog_client.go

+
+const (
+	// this is the retry count, the total attempts would be retry count + 1 at most.
+	defaultRetryCount          = 5


Default retries should be 1. This is usually too many retries for observability functionality not on the main execution path before dropping the data. In addition, this many retries will likely trigger throttling by CloudWatch API.

kbrockhoff · 2020-08-19T22:18:58Z

exporter/awsemfexporter/cwlog_client.go

+			default:
+				// ThrottlingException is handled here because the type cloudwatch.ThrottlingException is not yet available in public SDK
+				// Retry request if ThrottlingException happens
+				if awsErr.Code() == ErrCodeThrottlingException {


Retrying during throttling will never catch up. Just drop the data at this point. Possibly even the entire sequence.

kbrockhoff · 2020-08-19T22:22:20Z

exporter/awsemfexporter/cwlog_client.go

+}
+
+//sleep some back off time before retries.
+func backoffSleep(i int) {


Sleeping this long will never allow catchup

kbrockhoff · 2020-08-19T22:31:11Z

exporter/awsemfexporter/emf_exporter.go

+		retryCnt:           *awsConfig.MaxRetries,
+		logger:             logger,
+	}
+	if config.(*Config).ForceFlushInterval > 0 {


Upstream processors in the collector such as batch_retry should handle retries and queuing. Exporter should not need a timer. Just push any data received immediately. Only thing that needs to be handled is making sure maximum records is not exceeded per put. You will likely need a loop for that. See the X-ray exporter.

kbrockhoff · 2020-08-19T22:34:21Z

exporter/awsemfexporter/cwlog_client.go

+
+//Put log events. The method mainly handles different possible error could be returned from server side, and retries them
+//if necessary.
+func (client *CloudWatchLogClient) PutLogEvents(input *cloudwatchlogs.PutLogEventsInput, retryCnt int) *string {


Return error to collector pipeline instead of handling everything here. Requests that should not be retried (HTTP 4xx) must be wrapped in consumererror.Permanent

bogdandrutu · 2020-08-24T18:37:45Z

exporter/awsemfexporter/cwlog_client.go

+
+// Possible exceptions are combination of common errors (https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/CommonErrors.html)
+// and API specific erros (e.g. https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html#API_PutLogEvents_Errors)
+type CloudWatchLogClient struct {


Consider to make struct internal

bogdandrutu · 2020-08-24T18:38:24Z

exporter/awsemfexporter/factory.go

+)
+
+// Factory is the factory for AWS EMF exporter.
+type Factory struct {


Consider to use the helper factory

…Add the pusher to check the batched payload size.

tigrannajaryan · 2020-09-09T19:27:40Z

@shaochengwang can you split the PR as @bogdandrutu asked?

alolita · 2020-09-10T02:52:33Z

@tigrannajaryan @bogdandrutu - Ack will follow up w @shaochengwang

…try-collector-contrib into exporter_pr

shaochengwang · 2020-09-15T07:50:35Z

Thanks a lot for reviewing this PR. I have tried to make this feature as simple as possible in the second revision. Splitting it further may break its logic and functionality. Although it has 6000+ lines, I think most of them are the dependencies list and unit test files, the source code change should not be huge.

mxiamxia · 2020-08-06T21:14:55Z

exporter/awsemfexporter/README.md

+| `log_group_name`  | Customized log group name                                              |         |
+| `log_stream_name` | Customized log stream name                                             |         |


Can we fill the default values for these configurations?

mxiamxia · 2020-08-06T21:48:01Z

cmd/otelcontribcol/components.go

@@ -107,6 +108,7 @@ func components() (component.Factories, error) {
 		&splunkhecexporter.Factory{},
 		elasticexporter.NewFactory(),
 		&alibabacloudlogserviceexporter.Factory{},
+		&awsemfexporter.Factory{},


suggest update to NewFactory() interface

I've changed it in the latest version.

mxiamxia · 2020-09-16T03:01:05Z

exporter/awsemfexporter/cwlog_client.go

+					e.Message(),
+					e.Error(),
+					e))
+				backoffSleep(i)


should it be waited if the error type is InvalidSequenceTokenException? it seems we can retry it right away with the new token returned?

Good point.

mxiamxia · 2020-09-16T03:22:23Z

exporter/awsemfexporter/pusher.go

+	//The file name where this log event comes from
+	FileName string
+	//The offset for the input file
+	FilePosition int64


may not need?

mxiamxia · 2020-09-16T03:27:52Z

exporter/awsemfexporter/emf_exporter.go

+
+// Shutdown stops the exporter and is invoked during shutdown.
+func (emf *emfExporter) Shutdown(ctx context.Context) error {
+	return nil


can we force to flush the unpublished batched log events(in puhsers) here?

mxiamxia

We tested this implementation within AWS test environments. @kbrockhoff, pls help us review the PR. Thanks.

…try-collector-contrib into exporter_pr

bogdandrutu · 2020-09-23T19:12:45Z

Please follow the process described here:
https://github.com/open-telemetry/opentelemetry-collector/blob/master/CONTRIBUTING.md#when-adding-a-new-component

bogdandrutu · 2020-09-24T00:32:57Z

@anuraaga please review, we heard that this is urgent and we don't have time to review 5K lines PRs. But I trust that you as one of the future maintainer of this component can do this.

anuraaga · 2020-09-24T04:07:07Z

exporter/awsemfexporter/README.md

+This exporter converts OpenTelemetry metrics to 
+[AWS CloudWatch Embedded Metric Format(EMF)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html)
+and then sends them directly to CloudWatch Logs using the 
+[PutLogEvents](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html) API.


Not really for this PR but is it easy to support both API sending and writing to STDOUT? I guess STDOUT can help in many cases too.

anuraaga · 2020-09-24T04:09:19Z

exporter/awsemfexporter/README.md

+| `no_verify_ssl`   | Enable or disable TLS certificate verification.                        | false   |
+| `proxy_address`   | Upload Structured Logs to AWS CloudWatch through a proxy.              |         |
+| `region`          | Send Structured Logs to AWS CloudWatch in a specific region.           | determined by metadata |
+| `local_mode`      | Local mode to skip EC2 instance metadata check.                        | false   |


Can we remove local_mode? I know our other exporter has it but I want to consider removing it there too - I had trouble explaining this flag to a user before. We use it to control returning errors, but I don't think that's actually a good idea. I think we also use it to flag whether to fetch region info, but it seems ok to just try in any case instead of having this confusing flag.

yes. will remove this local_mode

anuraaga · 2020-09-24T04:11:08Z

exporter/awsemfexporter/README.md

+| `endpoint`        | Optionally override the default CloudWatch service endpoint.           |         |
+| `no_verify_ssl`   | Enable or disable TLS certificate verification.                        | false   |
+| `proxy_address`   | Upload Structured Logs to AWS CloudWatch through a proxy.              |         |
+| `region`          | Send Structured Logs to AWS CloudWatch in a specific region.           | determined by metadata |


Document the AWS_REGION flag here since it's other confusing that we have another mechanism for setting region.

anuraaga · 2020-09-24T04:12:00Z

exporter/awsemfexporter/conn.go

+)
+
+// GetAWSConfigSession returns AWS config and session instances.
+func GetAWSConfigSession(logger *zap.Logger, cn connAttr, cfg *Config) (*aws.Config, *session.Session, error) {


Whoever merges all the AWS conn.go files into one under internal/aws will get a prize

anuraaga · 2020-09-24T04:12:54Z

exporter/awsemfexporter/cwlog_client.go

+)
+
+const (
+	// this is the retry count, the total attempts would be retry count + 1 at most.


Suggested change

// this is the retry count, the total attempts would be retry count + 1 at most.

// this is the retry count, the total attempts will be at most retry count + 1.

anuraaga · 2020-09-24T04:48:41Z

exporter/awsemfexporter/metric_translator.go

+
+const (
+	CleanInterval = 5 * time.Minute
+	MinTimeDiff   = 50 // We assume 50 micro-seconds is the minimal gap between two collected data sample to be valid to calculate delta


MinTimeDiffMicros

Alternatively stick with time package, 50 * time.Microseconds

anuraaga · 2020-09-24T04:51:59Z

exporter/awsemfexporter/metric_translator.go

+		serviceNamespace, svcNsOk := attributes[conventions.AttributeServiceNamespace]
+
+		if svcNameOk {
+			svcAttrMode = ServiceNameOnly


This svcAttrMode seems unnecessarily complex, isn't it just

if svcNameOk && svcNsOk { namespace = fmt.... } else if svcNsOk { namespace = serviceNamespace } else if svcNameOk { namespace = serviceName }

anuraaga · 2020-09-24T04:53:48Z

exporter/awsemfexporter/metric_translator.go

+			totalDroppedMetrics += len(metric.GetTimeseries())
+			continue
+		}
+		//TODO: Handle OTLib as a dimension when it's supported


I think we're missing this since we converted to OC, but we should be sticking to the native OTel format

anuraaga · 2020-09-24T04:58:31Z

exporter/awsemfexporter/pusher.go

+
+	pusher := newPusher(logGroupName, logStreamName, svcStructuredLog)
+
+	// For blocking queue, assuming the log batch payload size is 1MB. Set queue size to 2


Is this comment in the wrong place? No queue size here

anuraaga · 2020-09-24T04:59:57Z

exporter/awsemfexporter/pusher.go

+	//* None of the log events in the batch can be older than 14 days or the
+	//retention period of the log group.
+	currentTime := time.Now().UTC()
+	utcTime := time.Unix(0, *logEvent.InputLogEvent.Timestamp*1e6).UTC()


Think you can use something in time. instead of magic number 1e6

…ove logging with zap logger, remove unnecessary metrics transformation to OCMetrics

anuraaga

Found a couple more small but important points but should be straightforward. Just a bit more, thanks!

anuraaga · 2020-09-25T07:30:47Z

exporter/awsemfexporter/cwlog_client.go

+
+		if response != nil {
+			if response.RejectedLogEventsInfo != nil {
+				rejectedLogEventsInfo := response.RejectedLogEventsInfo


Reminder that later we'll need metrics for these sort of failures

anuraaga · 2020-09-25T07:31:44Z

exporter/awsemfexporter/emf_exporter.go

+					err = wrapErrorIfBadRequest(&returnError)
+				}
+				if err != nil {
+					emf.logger.Error("Experiences some errors when gracefully shutting down emf_exporter. Skipping to next pusher.", zap.Error(err))


Suggested change

emf.logger.Error("Experiences some errors when gracefully shutting down emf_exporter. Skipping to next pusher.", zap.Error(err))

emf.logger.Error("Error when gracefully shutting down emf_exporter. Skipping to next pusher.", zap.Error(err))

anuraaga · 2020-09-25T07:32:59Z

exporter/awsemfexporter/pusher.go

+
+func (logEvent *LogEvent) truncateIfNeeded() bool {
+	if logEvent.eventPayloadBytes() > MaxEventPayloadBytes {
+		log.Printf("W! logpusher: the single log event size is %v, which is larger than the max event payload allowed %v. Truncate the log event.", logEvent.eventPayloadBytes(), MaxEventPayloadBytes)


Oh can you make sure to use the zap logger throughout the PR? We don't use standard library logging in the collector

will do. Thanks

anuraaga · 2020-09-25T07:34:35Z

exporter/awsemfexporter/pusher.go

+	//retention period of the log group.
+	currentTime := time.Now().UTC()
+	utcTime := time.Unix(0, *logEvent.InputLogEvent.Timestamp*int64(time.Millisecond)).UTC()
+	duration := currentTime.Sub(utcTime).Hours()


Is it ok to round the hours? How about converting 24*14 to duration instead?

Should probably have constants for the durations for the staleness check

Agree it's better to use constants here.

…limit

anuraaga

Thanks!

This adds a processor that drops data according to configured memory limits. The processor is important for high load situations when receiving rate exceeds exporting rate (and an extreme case of this is when the target of exporting is unavailable). Typical production run will need to have this processor included in every pipeline immediately after the batch processor.

Add EMF Exporter to support exporting metrics to AWS CloudWatch

345fd2a

shaochengwang requested a review from a team as a code owner July 23, 2020 18:37

project-bot bot added this to In progress in Collector Jul 23, 2020

bogdandrutu reviewed Jul 23, 2020

View reviewed changes

anuraaga reviewed Jul 24, 2020

View reviewed changes

shaochengwang added 4 commits July 29, 2020 15:47

AWSEMFExporter - Add basic functionality for awsemfexporter to suppor…

3198b6d

…t exporting metrics to CloudWatch

AWSEMFExporter - Add basic functionality for awsemfexporter to suppor…

7f7135c

…t exporting metrics to CloudWatch, Fix unit tests timeout issue

AWSEMFExporter - Add License

886226e

AWSEMFExporter - fixed lint impi check

f11c747

shaochengwang added 2 commits August 4, 2020 15:45

Merge remote upstream branch 'master' into exporter_pr

5332c79

remove unnecessary test and correct license date

c112fba

shaochengwang requested a review from bogdandrutu August 5, 2020 18:48

shaochengwang changed the title ~~Add EMF Exporter to support exporting metrics to AWS CloudWatch~~ AWSEMFExporter - Add EMF Exporter to support exporting metrics to AWS CloudWatch Aug 5, 2020

jrcamp assigned kbrockhoff Aug 19, 2020

kbrockhoff suggested changes Aug 19, 2020

View reviewed changes

Collector automation moved this from In progress to Review in progress Aug 19, 2020

bogdandrutu reviewed Aug 24, 2020

View reviewed changes

Throw the error back to pipeline instead of handling it in exporter. …

61807e8

…Add the pusher to check the batched payload size.

shaochengwang added 4 commits September 10, 2020 14:04

Merge branch 'master' of https://github.com/open-telemetry/openteleme…

4e32082

…try-collector-contrib into exporter_pr

Merge branch 'master' of https://github.com/open-telemetry/openteleme…

6771341

…try-collector-contrib into exporter_pr

Merge branch 'master' of https://github.com/open-telemetry/openteleme…

83d50f9

…try-collector-contrib into exporter_pr

Upgrade sdk to v0.9.1 and made some adjustment

50b3d0e

shaochengwang requested a review from kbrockhoff September 11, 2020 16:59

mxiamxia reviewed Sep 16, 2020

View reviewed changes

update collector to v0.10.1

9aa7de3

mxiamxia approved these changes Sep 21, 2020

View reviewed changes

shaochengwang added 9 commits September 21, 2020 16:58

Add some unit tests

fcab947

Improve unit test coverage

11c79f8

Merge branch 'master' of https://github.com/open-telemetry/openteleme…

3c5b935

…try-collector-contrib into exporter_pr

minor fix for lint test

e962306

correct the local mode for one unit test

cd7b8b4

simplify one exporter test to decrease runtime

9ab9275

add unit test coverage

bedc659

improve unit test coverage

cf82cbc

sync go.sum from upstream

0cd4f6d

kbrockhoff approved these changes Sep 22, 2020

View reviewed changes

shaochengwang added 2 commits September 23, 2020 00:40

Merge branch 'master' of https://github.com/open-telemetry/openteleme…

617c8eb

…try-collector-contrib into exporter_pr

Modify conn test to be working on sandbox

9e682c8

anuraaga reviewed Sep 24, 2020

View reviewed changes

shaochengwang added 2 commits September 24, 2020 23:33

Make README more clear, remove sleep&retry logic in CreatStream, impr…

82b73cc

…ove logging with zap logger, remove unnecessary metrics transformation to OCMetrics

Add unit test for emf_exporter_test

d58124e

shaochengwang requested a review from anuraaga September 25, 2020 07:27

anuraaga reviewed Sep 25, 2020

View reviewed changes

Some minor changes on the logger and constants for LogEventTimestamp …

f56e52a

…limit

anuraaga approved these changes Sep 25, 2020

View reviewed changes

Collector automation moved this from Review in progress to Reviewer approved Sep 25, 2020

shaochengwang added 2 commits September 25, 2020 01:50

make minor change on unit tests

db97dd5

add unit test in cwlog_client

4cfefe8

bogdandrutu merged commit 434bfda into open-telemetry:master Sep 26, 2020

Collector automation moved this from Reviewer approved to Done Sep 26, 2020

shaochengwang mentioned this pull request Jul 26, 2023

REQUEST: New membership for shaochengwang open-telemetry/community#1615

Closed

6 tasks

		The following exporter configuration parameters are supported. They mirror and have the same affect as the
		comparable AWS X-Ray Daemon configuration values.

		\| `log_group_name` \| Customized log group name \| \|
		\| `log_stream_name` \| Customized log stream name \| \|

	// this is the retry count, the total attempts would be retry count + 1 at most.
	// this is the retry count, the total attempts will be at most retry count + 1.


		pusher := newPusher(logGroupName, logStreamName, svcStructuredLog)

		// For blocking queue, assuming the log batch payload size is 1MB. Set queue size to 2

	emf.logger.Error("Experiences some errors when gracefully shutting down emf_exporter. Skipping to next pusher.", zap.Error(err))
	emf.logger.Error("Error when gracefully shutting down emf_exporter. Skipping to next pusher.", zap.Error(err))

AWSEMFExporter - Add EMF Exporter to support exporting metrics to AWS CloudWatch #498

AWSEMFExporter - Add EMF Exporter to support exporting metrics to AWS CloudWatch #498

Conversation

shaochengwang commented Jul 23, 2020 • edited

linux-foundation-easycla bot commented Jul 23, 2020 • edited

bogdandrutu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anuraaga left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Aug 4, 2020 • edited

Codecov Report

shaochengwang commented Aug 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tigrannajaryan commented Sep 9, 2020

alolita commented Sep 10, 2020

shaochengwang commented Sep 15, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shaochengwang Sep 16, 2020 • edited

Choose a reason for hiding this comment

mxiamxia Sep 16, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mxiamxia left a comment

Choose a reason for hiding this comment

bogdandrutu commented Sep 23, 2020

bogdandrutu commented Sep 24, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anuraaga left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anuraaga left a comment

Choose a reason for hiding this comment

shaochengwang commented Jul 23, 2020 •

edited

linux-foundation-easycla bot commented Jul 23, 2020 •

edited

codecov bot commented Aug 4, 2020 •

edited

shaochengwang Sep 16, 2020 •

edited

mxiamxia Sep 16, 2020 •

edited