awsxrayreceiver throws fatal errors when the input data load is intensiv

[mxiamxia]

I've just tested awsxrayreceiver if the input data load to the receiver is intensive (300 segments/s in my test), 'awsxrayreceiver' will give the fatal error and crash the collector. I've done the same test on OTLP receiver with even higher data load(1000 spans/s) but everything works fine. We need to resolve this issue, otherwise we'll consider to revert awsxrayreceiver in the collector. @JohnWu20

2020-11-03T12:17:52.990-0800    WARN    awsxrayreceiver@v0.13.1-0.20201022225542-7c196e77a175/receiver.go:131   X-Ray segment to OT traces conversion failed    {"component_kind": "receiver", "component_type": "awsxray", "component_name": "awsxray", "error": "unexpected end of JSON input"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awsxrayreceiver.(*xrayReceiver).start
        /Users/xiami/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awsxrayreceiver@v0.13.1-0.20201022225542-7c196e77a175/receiver.go:131
2020-11-03 12:17:52.990843 -0800 PST m=+541.715129136 write error: can't rename log file: rename /opt/aws/aws-otel-collector/logs/aws-otel-collector.log /opt/aws/aws-otel-collector/logs/aws-otel-collector-2020-11-03T20-17-52.990.log: permission denied
panic: JSON decoder out of sync - data changing underfoot?

goroutine 49 [running]:
encoding/json.(*decodeState).value(0xc001427810, 0x0, 0x0, 0x0, 0x5, 0xf92e)
        /usr/local/go/src/encoding/json/decode.go:373 +0x234
encoding/json.(*decodeState).object(0xc001427810, 0x6851060, 0xc00084dce8, 0x199, 0xc001427838, 0x7b)
        /usr/local/go/src/encoding/json/decode.go:782 +0x12e5
encoding/json.(*decodeState).value(0xc001427810, 0x6851060, 0xc00084dce8, 0x199, 0x6851060, 0xc00084dce8)
        /usr/local/go/src/encoding/json/decode.go:387 +0x6d
encoding/json.(*decodeState).array(0xc001427810, 0x6181280, 0xc0004b0810, 0x197, 0xc001427838, 0x5b)
        /usr/local/go/src/encoding/json/decode.go:575 +0x1a7
encoding/json.(*decodeState).value(0xc001427810, 0x6181280, 0xc0004b0810, 0x197, 0x6181280, 0xc0004b0810)
        /usr/local/go/src/encoding/json/decode.go:377 +0xfd
encoding/json.(*decodeState).object(0xc001427810, 0x6344620, 0xc0004b0780, 0x16, 0xc001427838, 0xc00044dd7b)
        /usr/local/go/src/encoding/json/decode.go:782 +0x12e5
encoding/json.(*decodeState).value(0xc001427810, 0x6344620, 0xc0004b0780, 0x16, 0xc00067d9a0, 0x43892f6)
        /usr/local/go/src/encoding/json/decode.go:387 +0x6d
encoding/json.(*decodeState).unmarshal(0xc001427810, 0x6344620, 0xc0004b0780, 0xc001427838, 0x0)
        /usr/local/go/src/encoding/json/decode.go:180 +0x1f0
encoding/json.Unmarshal(0xc00085e021, 0xc49, 0xffdf, 0x6344620, 0xc0004b0780, 0x4859e77, 0x851aa80)
        /usr/local/go/src/encoding/json/decode.go:107 +0x112
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awsxrayreceiver/internal/translator.ToTraces(0xc00085e021, 0xc49, 0xffdf, 0x7, 0x3, 0x0, 0x0)
        /Users/xiami/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awsxrayreceiver@v0.13.1-0.20201022225542-7c196e77a175/internal/translator/translator.go:39 +0x8b
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awsxrayreceiver.(*xrayReceiver).start(0xc0002c0230)
        /Users/xiami/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awsxrayreceiver@v0.13.1-0.20201022225542-7c196e77a175/receiver.go:129 +0x111
created by github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awsxrayreceiver.(*xrayReceiver).Start.func1
        /Users/xiami/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/awsxrayreceiver@v0.13.1-0.20201022225542-7c196e77a175/receiver.go:96 +0xd9
Process finished with exit code 2

[JohnWu20]

Thanks @mxiamxia I am working on this bug, it could be caused by race condition. I will solve it as soon as possible to avoid reverting awsxrayreceiver in the collector.

[JohnWu20]

Fixed the issue:
open-telemetry/opentelemetry-collector-contrib#1490

[JohnWu20]

Hi Min @mxiamxia, thanks for pointing out the problem. I think I have fixed the issue and the code has been merged, please check if we can close this issue or not.