Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky TestDataForward_WithContextClose #412

Closed
whynowy opened this issue Dec 7, 2022 · 2 comments · Fixed by #413
Closed

Flaky TestDataForward_WithContextClose #412

whynowy opened this issue Dec 7, 2022 · 2 comments · Fixed by #413
Assignees
@yhl25
Labels
area/reduce Reduce operations like GroupByKey bug Something isn't working

Comments

@whynowy
Copy link
Member

whynowy commented Dec 7, 2022 •
edited by vigith
Loading 

{"level":"info","ts":1670371887.9807837,"logger":"numaflow","caller":"readloop/readloop.go:98","msg":"Partitions to be replayed ","count":0,"partitions":[]}
{"level":"info","ts":1670371887.993508,"logger":"numaflow","caller":"pbq/pbqmanager.go:233","msg":"Finished replaying records from store","took":0.0000025,"partitions":[]}
==================
WARNING: DATA RACE
Read at 0x00c004439b30 by goroutine 3773:
  runtime.mapiterinit()
      /opt/hostedtoolcache/go/1.19.3/x64/src/runtime/map.go:815 +0x0
  github.com/numaproj/numaflow/pkg/pbq/store/memory.(*memoryStores).DiscoverPartitions()
      /home/runner/work/numaflow/numaflow/pkg/pbq/store/memory/stores.go:71 +0xe6
  github.com/numaproj/numaflow/pkg/reduce.TestDataForward_WithContextClose()
      /home/runner/work/numaflow/numaflow/pkg/reduce/reduce_test.go:718 +0x904
  testing.tRunner()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1493 +0x47

Previous write at 0x00c004439b30 by goroutine 3783:
  runtime.mapassign()
      /opt/hostedtoolcache/go/1.19.3/x64/src/runtime/map.go:578 +0x0
  github.com/numaproj/numaflow/pkg/pbq/store/memory.(*memoryStores).CreateStore()
      /home/runner/work/numaflow/numaflow/pkg/pbq/store/memory/stores.go:64 +0x555
  github.com/numaproj/numaflow/pkg/pbq.(*Manager).CreateNewPBQ()
      /home/runner/work/numaflow/numaflow/pkg/pbq/pbqmanager.go:71 +0xf4
  github.com/numaproj/numaflow/pkg/reduce/readloop.(*ReadLoop).associatePBQAndPnF.func1()
      /home/runner/work/numaflow/numaflow/pkg/reduce/readloop/readloop.go:296 +0x10c
  k8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:220 +0x30
  k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:233 +0x7b
  k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:226 +0x5c
  k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:738 +0xba
  github.com/numaproj/numaflow/pkg/reduce/readloop.(*ReadLoop).associatePBQAndPnF()
      /home/runner/work/numaflow/numaflow/pkg/reduce/readloop/readloop.go:294 +0x494
  github.com/numaproj/numaflow/pkg/reduce/readloop.(*ReadLoop).writeToPBQ()
      /home/runner/work/numaflow/numaflow/pkg/reduce/readloop/readloop.go:207 +0xf0
  github.com/numaproj/numaflow/pkg/reduce/readloop.(*ReadLoop).writeMessagesToWindows()
      /home/runner/work/numaflow/numaflow/pkg/reduce/readloop/readloop.go:182 +0x9d4
  github.com/numaproj/numaflow/pkg/reduce/readloop.(*ReadLoop).Process()
      /home/runner/work/numaflow/numaflow/pkg/reduce/readloop/readloop.go:126 +0x9a
  github.com/numaproj/numaflow/pkg/reduce.(*DataForward).forwardAChunk()
      /home/runner/work/numaflow/numaflow/pkg/reduce/reduce.go:163 +0x57e
  github.com/numaproj/numaflow/pkg/reduce.(*DataForward).Start()
      /home/runner/work/numaflow/numaflow/pkg/reduce/reduce.go:135 +0x24e
  github.com/numaproj/numaflow/pkg/reduce.TestDataForward_WithContextClose.func5()
      /home/runner/work/numaflow/numaflow/pkg/reduce/reduce_test.go:697 +0x58

Goroutine 3773 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1493 +0x75d
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1846 +0x99
  testing.tRunner()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1446 +0x216
  testing.runTests()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1844 +0x7ec
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1726 +0xa84
  main.main()
      _testmain.go:59 +0x2e9

Goroutine 3783 (running) created at:
  github.com/numaproj/numaflow/pkg/reduce.TestDataForward_WithContextClose()
      /home/runner/work/numaflow/numaflow/pkg/reduce/reduce_test.go:697 +0x793
  testing.tRunner()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1493 +0x47
==================
    testing.go:1319: race detected during execution of test
--- FAIL: TestDataForward_WithContextClose (0.07s)
@whynowy whynowy added the bug Something isn't working label Dec 7, 2022
@vigith vigith added the area/reduce Reduce operations like GroupByKey label Dec 7, 2022
@vigith
Copy link
Contributor

vigith commented Dec 7, 2022

I have also seen this recently


{"level":"info","ts":1669748576.0926795,"logger":"numaflow","caller":"pbq/pbqmanager.go:233","msg":"Finished replaying records from store","took":0.0000025,"partitions":[]}
    reduce_test.go:630: 
        	Error Trace:	reduce_test.go:630
        	Error:      	"[0-300-even]" should have 2 item(s), but has 1
        	Test:       	TestDataForward_WithContextClose
--- FAIL: TestDataForward_WithContextClose (0.04s)
==================
WARNING: DATA RACE
Write at 0x00c000c92a50 by goroutine 2563:
  runtime.mapassign()
      /opt/hostedtoolcache/go/1.19.3/x64/src/runtime/map.go:578 +0x0
  github.com/numaproj/numaflow/pkg/pbq/store/memory.(*memoryStores).CreateStore()
      /home/runner/work/numaflow/numaflow/pkg/pbq/store/memory/stores.go:64 +0x555
  github.com/numaproj/numaflow/pkg/pbq.(*Manager).CreateNewPBQ()
      /home/runner/work/numaflow/numaflow/pkg/pbq/pbqmanager.go:71 +0xf4
  github.com/numaproj/numaflow/pkg/reduce/readloop.(*ReadLoop).associatePBQAndPnF.func1()
      /home/runner/work/numaflow/numaflow/pkg/reduce/readloop/readloop.go:296 +0x10c
  k8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:220 +0x30
  k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:233 +0x7b
  k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:226 +0x5c
  k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext()
      /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.23.3/pkg/util/wait/wait.go:738 +0xba
  github.com/numaproj/numaflow/pkg/reduce/readloop.(*ReadLoop).associatePBQAndPnF()
      /home/runner/work/numaflow/numaflow/pkg/reduce/readloop/readloop.go:294 +0x494
  github.com/numaproj/numaflow/pkg/reduce/readloop.(*ReadLoop).writeToPBQ()
      /home/runner/work/numaflow/numaflow/pkg/reduce/readloop/readloop.go:207 +0xf0
  github.com/numaproj/numaflow/pkg/reduce/readloop.(*ReadLoop).writeMessagesToWindows()
      /home/runner/work/numaflow/numaflow/pkg/reduce/readloop/readloop.go:182 +0x974
  github.com/numaproj/numaflow/pkg/reduce/readloop.(*ReadLoop).Process()
      /home/runner/work/numaflow/numaflow/pkg/reduce/readloop/readloop.go:126 +0x9a
  github.com/numaproj/numaflow/pkg/reduce.(*DataForward).forwardAChunk()
      /home/runner/work/numaflow/numaflow/pkg/reduce/reduce.go:155 +0x424
  github.com/numaproj/numaflow/pkg/reduce.(*DataForward).Start()
      /home/runner/work/numaflow/numaflow/pkg/reduce/reduce.go:131 +0x24e
  github.com/numaproj/numaflow/pkg/reduce.TestDataForward_WithContextClose.func5()
      /home/runner/work/numaflow/numaflow/pkg/reduce/reduce_test.go:594 +0x58

Previous read at 0x00c000c92a50 by goroutine 2553:
  runtime.mapiterinit()
      /opt/hostedtoolcache/go/1.19.3/x64/src/runtime/map.go:815 +0x0
  github.com/numaproj/numaflow/pkg/pbq/store/memory.(*memoryStores).DiscoverPartitions()
      /home/runner/work/numaflow/numaflow/pkg/pbq/store/memory/stores.go:71 +0xe6
  github.com/numaproj/numaflow/pkg/reduce.TestDataForward_WithContextClose()
      /home/runner/work/numaflow/numaflow/pkg/reduce/reduce_test.go:615 +0x8e4
  testing.tRunner()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1493 +0x47

Goroutine 2563 (running) created at:
  github.com/numaproj/numaflow/pkg/reduce.TestDataForward_WithContextClose()
      /home/runner/work/numaflow/numaflow/pkg/reduce/reduce_test.go:594 +0x779
  testing.tRunner()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1493 +0x47

Goroutine 2553 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1493 +0x75d
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1846 +0x99
  testing.tRunner()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1446 +0x216
  testing.runTests()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1844 +0x7ec
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.19.3/x64/src/testing/testing.go:1726 +0xa84
  main.main()
      _testmain.go:57 +0x2e9
==================

@yhl25
Copy link
Contributor

yhl25 commented Dec 7, 2022

I will check

@yhl25 yhl25 mentioned this issue Dec 7, 2022
Merged
whynowy pushed a commit that referenced this issue Dec 7, 2022
@yhl25
fix: adding lock while discovering partitions, Closes #412 (#413)
f1e7c73 
Signed-off-by: Yashash H L <yashashhl25@gmail.com>
whynowy pushed a commit that referenced this issue Dec 7, 2022
@yhl25 @whynowy
fix: adding lock while discovering partitions, Closes #412 (#413)
676ea1c 
Signed-off-by: Yashash H L <yashashhl25@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yhl25 yhl25

Labels
area/reduce Reduce operations like GroupByKey bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: adding lock while discovering partitions, Closes #412 yhl25/numaflow
3 participants
@vigith @whynowy @yhl25