Fix the race condition issue for xray receiver

[JohnWu20]

Description:
Aws xrayreceiver throws fatal errors when the input data load is intensive. Solving the race condition issue.

Link to tracking Issue:
aws-observability/aws-otel-collector#92

Testing:
Did the local test when the input data load to the receiver is intensive (1000 segments/s in my test), the receiver works well.

[anuraaga]

Thanks - didn't realize we have concurrent pollers. Instead of making the buffer every single time, can you move buffer creation to line 121 where the polling routines are initialized and pass in as a parameter to poll()?

[anuraaga]

Actually - nevermind, it shouldn't matter. Can you point me at the race condition? This loop should not be called concurrently so there's something else going on.

[anuraaga]

I guess it's because we pass the buffer to here

opentelemetry-collector-contrib/receiver/awsxrayreceiver/receiver.go

Line 129 in 65f181d

traces, totalSpansCount, err := translator.ToTraces(seg.Payload)

How about doing the translation inline without the jump? There doesn't seem to be that much point in having multiple pollers anyways if we end up just translating in a single loop.

[JohnWu20]

Yes, you are right, we actually pass the body of the buffer to translator. And because they are different threads which caused the race condition. I didn't design the receiver, so I don't know why we have multiple pollers but just one translator. There may have some special purpose to design it in that way. However we just released it, and Min @mxiamxia believes if we don't fix it as soon as possible, we should revert the release, so I think we can fix it in this way for now.

[anuraaga]

The performance penalty for reallocating every iteration seems extremely high though. I guess it's better than broken but I wouldn't release the version with this code either. Can you commit to optimizing this right away?

[JohnWu20]

Sure, I will optimize this.

[codecov]

Codecov Report

Merging #1490 into master will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1490      +/-   ##
==========================================
+ Coverage   88.72%   88.73%   +0.01%     
==========================================
  Files         344      344              
  Lines       16844    16844              
==========================================
+ Hits        14944    14946       +2     
+ Misses       1434     1433       -1     
+ Partials      466      465       -1     

Flag	Coverage Δ
integration	70.87% <ø> (+0.07%)	⬆️
unit	87.37% <100.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...eiver/awsxrayreceiver/internal/udppoller/poller.go	100.00% <100.00%> (ø)
receiver/k8sclusterreceiver/watcher.go	97.64% <0.00%> (+2.35%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db507b3...7061478. Read the comment docs.

[anuraaga]

Approving since it goes from broken to inefficient, which is an improvement, but we'll need to optimize this better

[anuraaga]

The performance penalty for reallocating every iteration seems extremely high though. I guess it's better than broken but I wouldn't release the version with this code either. Can you commit to optimizing this right away?