Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestQueuedRetryPersistentEnabled_shutdown_dataIsRequeued is flaky #8124

Closed
atoulme opened this issue Jul 24, 2023 · 1 comment · Fixed by #8986
Closed

TestQueuedRetryPersistentEnabled_shutdown_dataIsRequeued is flaky #8124

atoulme opened this issue Jul 24, 2023 · 1 comment · Fixed by #8986
Assignees
@atoulme
Labels
bug Something isn't working

Comments

@atoulme
Copy link
Contributor

atoulme commented Jul 24, 2023

See https://github.com/open-telemetry/opentelemetry-collector/actions/runs/5641800307/job/15280483750

WARNING: DATA RACE
Write at 0x00c00008ac70 by goroutine 184:
  runtime.closechan()
      /opt/hostedtoolcache/go/1.20.6/x64/src/runtime/chan.go:357 +0x0
  go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Stop()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/internal/bounded_memory_queue.go:82 +0x6b
  go.opentelemetry.io/collector/exporter/exporterhelper.(*producerConsumerQueueWithCounter).Stop()
      <autogenerated>:1 +0x49
  go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).shutdown()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queued_retry.go:233 +0x11b
  go.opentelemetry.io/collector/exporter/exporterhelper.newBaseExporter.func2()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/common.go:170 +0x5e
  go.opentelemetry.io/collector/component.ShutdownFunc.Shutdown()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/component/component.go:84 +0xa92
  go.opentelemetry.io/collector/exporter/exporterhelper.TestQueuedRetryPersistentEnabled_shutdown_dataIsRequeued()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queued_retry_test.go:614 +0xa45
  testing.tRunner()
      /opt/hostedtoolcache/go/1.20.6/x64/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.20.6/x64/src/testing/testing.go:1629 +0x47

Previous read at 0x00c00008ac70 by goroutine 185:
  runtime.chansend()
      /opt/hostedtoolcache/go/1.20.6/x64/src/runtime/chan.go:160 +0x0
  go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Produce()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/internal/bounded_memory_queue.go:69 +0x104
  go.opentelemetry.io/collector/exporter/exporterhelper.(*producerConsumerQueueWithCounter).Produce()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queued_retry_test.go:822 +0x83
  go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queued_retry.go:173 +0x38d
  go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure-fm()
      <autogenerated>:1 +0x7d
  go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queued_retry.go:435 +0x1187
  go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queued_retry.go:195 +0x64
  go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/internal/bounded_memory_queue.go:47 +0x151

Goroutine 184 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.20.6/x64/src/testing/testing.go:1629 +0x805
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.20.6/x64/src/testing/testing.go:2036 +0x8d
  testing.tRunner()
      /opt/hostedtoolcache/go/1.20.6/x64/src/testing/testing.go:1576 +0x216
  testing.runTests()
      /opt/hostedtoolcache/go/1.20.6/x64/src/testing/testing.go:2034 +0x87c
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.20.6/x64/src/testing/testing.go:1906 +0xb44
  main.main()
      _testmain.go:175 +0x2e9

Goroutine 185 (running) created at:
  go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/internal/bounded_memory_queue.go:42 +0x67
  go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queued_retry.go:194 +0x152
  go.opentelemetry.io/collector/exporter/exporterhelper.newBaseExporter.func1()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/common.go:166 +0xe9
  go.opentelemetry.io/collector/component.StartFunc.Start()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/component/component.go:73 +0x5c1
  go.opentelemetry.io/collector/exporter/exporterhelper.TestQueuedRetryPersistentEnabled_shutdown_dataIsRequeued()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queued_retry_test.go:589 +0x520
  testing.tRunner()
      /opt/hostedtoolcache/go/1.20.6/x64/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.20.6/x64/src/testing/testing.go:1629 +0x47
==================
--- FAIL: TestQueuedRetryPersistentEnabled_shutdown_dataIsRequeued (1.01s)
    queued_retry_test.go:615: 
        	Error Trace:	/home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queued_retry_test.go:615
        	Error:      	Condition never satisfied
        	Test:       	TestQueuedRetryPersistentEnabled_shutdown_dataIsRequeued
    testing.go:[144](https://github.com/open-telemetry/opentelemetry-collector/actions/runs/5641800307/job/15280483750#step:6:145)6: race detected during execution of test
@atoulme atoulme added the bug Something isn't working label Jul 24, 2023
@atoulme atoulme self-assigned this Sep 28, 2023
@atoulme atoulme mentioned this issue Sep 28, 2023
Closed
@atoulme
Copy link
Contributor Author

atoulme commented Sep 28, 2023

Still happening, slight change of line numbers:

==================
WARNING: DATA RACE
Write at 0x00c00016c850 by goroutine 277:
  runtime.closechan()
      /opt/hostedtoolcache/go/1.21.1/x64/src/runtime/chan.go:357 +0x0
  go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Stop()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/internal/bounded_memory_queue.go:88 +0x64
  go.opentelemetry.io/collector/exporter/exporterhelper.(*producerConsumerQueueWithCounter).Stop()
      <autogenerated>:1 +0x43
  go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).shutdown()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queue_sender.go:159 +0xe4
  go.opentelemetry.io/collector/exporter/exporterhelper.(*baseExporter).Shutdown()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/common.go:227 +0x8a
  go.opentelemetry.io/collector/exporter/exporterhelper.TestQueuedRetryPersistentEnabled_shutdown_dataIsRequeued()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queue_sender_test.go:336 +0x9b0
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.1/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.1/x64/src/testing/testing.go:1648 +0x44

Previous read at 0x00c00016c850 by goroutine 278:
  runtime.chansend()
      /opt/hostedtoolcache/go/1.21.1/x64/src/runtime/chan.go:160 +0x0
  go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Produce()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/internal/bounded_memory_queue.go:75 +0xf3
  go.opentelemetry.io/collector/exporter/exporterhelper.(*producerConsumerQueueWithCounter).Produce()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/retry_sender_test.go:405 +0x79
  go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).onTemporaryFailure()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queue_sender.go:99 +0x357
  go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).onTemporaryFailure-fm()
      <autogenerated>:1 +0x73
  go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/retry_sender.go:189 +0x1182
  go.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/common.go:40 +0x51
  go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).start.func1()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queue_sender.go:[124](https://github.com/open-telemetry/opentelemetry-collector/actions/runs/6255168264/job/16984055692#step:6:125) +0x55
  go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Start.func1()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/internal/bounded_memory_queue.go:52 +0x156

Goroutine 277 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.21.1/x64/src/testing/testing.go:1648 +0x82a
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.21.1/x64/src/testing/testing.go:2054 +0x84
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.1/x64/src/testing/testing.go:1595 +0x238
  testing.runTests()
      /opt/hostedtoolcache/go/1.21.1/x64/src/testing/testing.go:2052 +0x896
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.21.1/x64/src/testing/testing.go:1925 +0xb57
  main.main()
      _testmain.go:261 +0x2bd

Goroutine 278 (running) created at:
  go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Start()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/internal/bounded_memory_queue.go:47 +0xcd
  go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).start()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queue_sender.go:120 +0x238
  go.opentelemetry.io/collector/exporter/exporterhelper.(*baseExporter).Start()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/common.go:219 +0x1d7
  go.opentelemetry.io/collector/exporter/exporterhelper.TestQueuedRetryPersistentEnabled_shutdown_dataIsRequeued()
      /home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queue_sender_test.go:313 +0x524
  testing.tRunner()
      /opt/hostedtoolcache/go/1.21.1/x64/src/testing/testing.go:1595 +0x238
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.21.1/x64/src/testing/testing.go:1648 +0x44
==================
--- FAIL: TestQueuedRetryPersistentEnabled_shutdown_dataIsRequeued (1.01s)
    queue_sender_test.go:337: 
        	Error Trace:	/home/runner/work/opentelemetry-collector/opentelemetry-collector/exporter/exporterhelper/queue_sender_test.go:337
        	Error:      	Condition never satisfied
        	Test:       	TestQueuedRetryPersistentEnabled_shutdown_dataIsRequeued
    testing.go:[146](https://github.com/open-telemetry/opentelemetry-collector/actions/runs/6255168264/job/16984055692#step:6:147)5: race detected during execution of test

@atoulme atoulme mentioned this issue Sep 28, 2023
Closed
This was referenced Nov 23, 2023
Merged
Closed
bogdandrutu pushed a commit that referenced this issue Nov 23, 2023
@dmitryax
[chore] Fix TestQueuedRetryPersistentEnabled_shutdown_dataIsRequeued (#…
3b56bd0 
…8986)

Fix flaky TestQueuedRetryPersistentEnabled_shutdown_dataIsRequeued by
moving it to persistent queue.
- It makes the test easy to validate given that the size of the
persistent queue is always available even if it's closed.
- It brings behavior closer to the name of the test
- It removes the flakiness associated with data race specific to
re-enqueuing to the bounded memory queue by shutdown which should be
resolved separately once the re-enqueue option is available for the
memory queue

Fixes
#8124
pantuza pushed a commit to pantuza/opentelemetry-collector that referenced this issue Dec 8, 2023
@dmitryax @pantuza
[chore] Fix TestQueuedRetryPersistentEnabled_shutdown_dataIsRequeued (o…
44474a2 
…pen-telemetry#8986)

Fix flaky TestQueuedRetryPersistentEnabled_shutdown_dataIsRequeued by
moving it to persistent queue.
- It makes the test easy to validate given that the size of the
persistent queue is always available even if it's closed.
- It brings behavior closer to the name of the test
- It removes the flakiness associated with data race specific to
re-enqueuing to the bounded memory queue by shutdown which should be
resolved separately once the re-enqueue option is available for the
memory queue

Fixes
open-telemetry#8124
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@atoulme atoulme

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
1 participant
@atoulme