Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Route Global Resync test is still flaky #2443

Closed
mattmoor opened this issue Nov 9, 2018 · 2 comments · Fixed by #2848
Closed

Route Global Resync test is still flaky #2443

mattmoor opened this issue Nov 9, 2018 · 2 comments · Fixed by #2848
Labels
area/test-and-release It flags unit/e2e/conformance/perf test issues for product features kind/bug Categorizes issue or PR as related to a bug.

Comments

@mattmoor
Copy link
Member

mattmoor commented Nov 9, 2018

e.g. https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_serving/2397/pull-knative-serving-unit-tests/1060728363024388097/

cc @imikushin
@knative-prow-robot knative-prow-robot added area/test-and-release It flags unit/e2e/conformance/perf test issues for product features kind/bug Categorizes issue or PR as related to a bug. labels Nov 9, 2018
@mattmoor mattmoor added this to the Needs Triage milestone Nov 27, 2018
@markusthoemmes markusthoemmes mentioned this issue Nov 30, 2018
Merged
@mattmoor
Copy link
Member Author

mattmoor commented Dec 9, 2018

That links to the wrong test flake, let's try this: https://gubernator.knative.dev/build/knative-prow/logs/ci-knative-serving-continuous/1071162866708516864

mattmoor added a commit to mattmoor/serving that referenced this issue Dec 10, 2018
@mattmoor
Refactor global resync tests to use our "hooks" package.
e7977df 
I spent a little time looking at knative#2443 and it seemed like a good use of the hook stuff that @grantr put together in the early days of knative/serving.

Running this with high iteration counts locally, I still see occasional errors, so I'm not going to close out the linked issue, but I feel like this is a step in the right direction as this library has been fairly well worn at this point.

Related: knative#2443
@mattmoor mattmoor mentioned this issue Dec 10, 2018
Merged
mattmoor added a commit to mattmoor/serving that referenced this issue Dec 17, 2018
@mattmoor
Refactor global resync tests to use our "hooks" package.
3033736 
I spent a little time looking at knative#2443 and it seemed like a good use of the hook stuff that @grantr put together in the early days of knative/serving.

Running this with high iteration counts locally, I still see occasional errors, so I'm not going to close out the linked issue, but I feel like this is a step in the right direction as this library has been fairly well worn at this point.

Related: knative#2443
knative-prow-robot pushed a commit that referenced this issue Dec 18, 2018
@mattmoor @knative-prow-robot
Refactor global resync tests to use our "hooks" package. (#2685)
d872d6a 
I spent a little time looking at #2443 and it seemed like a good use of the hook stuff that @grantr put together in the early days of knative/serving.

Running this with high iteration counts locally, I still see occasional errors, so I'm not going to close out the linked issue, but I feel like this is a step in the right direction as this library has been fairly well worn at this point.

Related: #2443
@dgerd dgerd mentioned this issue Dec 26, 2018
Closed
@markusthoemmes
Copy link
Contributor

TestGlobalResyncOnConfigMapUpdate/Update_Fluentd_Image has a data race:

I0103 22:06:31.436] ==================
I0103 22:06:31.436] WARNING: DATA RACE
I0103 22:06:31.436] Read at 0x00c4207d0138 by goroutine 150:
I0103 22:06:31.436]   github.com/knative/serving/pkg/reconciler/v1alpha1/revision.TestGlobalResyncOnConfigMapUpdate.func6.1()
I0103 22:06:31.436]       /go/src/github.com/knative/serving/pkg/reconciler/v1alpha1/revision/revision_test.go:748 +0x2b2
I0103 22:06:31.437]   github.com/knative/serving/pkg/reconciler/testing.(*Hooks).OnUpdate.func1()
I0103 22:06:31.437]       /go/src/github.com/knative/serving/pkg/reconciler/testing/hooks.go:114 +0xd2
I0103 22:06:31.437]   github.com/knative/serving/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()
I0103 22:06:31.437]       /go/src/github.com/knative/serving/vendor/k8s.io/client-go/testing/fixture.go:487 +0x64
I0103 22:06:31.437]   github.com/knative/serving/vendor/k8s.io/client-go/testing.(*Fake).Invokes()
I0103 22:06:31.437]       /go/src/github.com/knative/serving/vendor/k8s.io/client-go/testing/fake.go:140 +0x275
I0103 22:06:31.437]   github.com/knative/serving/pkg/client/clientset/versioned/typed/serving/v1alpha1/fake.(*FakeRevisions).Update()
I0103 22:06:31.438]       /go/src/github.com/knative/serving/pkg/client/clientset/versioned/typed/serving/v1alpha1/fake/fake_revision.go:92 +0x29b
I0103 22:06:31.438]   github.com/knative/serving/pkg/reconciler/v1alpha1/revision.(*Reconciler).updateStatus()
I0103 22:06:31.438]       /go/src/github.com/knative/serving/pkg/reconciler/v1alpha1/revision/revision.go:433 +0x2c9
I0103 22:06:31.438]   github.com/knative/serving/pkg/reconciler/v1alpha1/revision.(*Reconciler).Reconcile()
I0103 22:06:31.438]       /go/src/github.com/knative/serving/pkg/reconciler/v1alpha1/revision/revision.go:269 +0x3c6
I0103 22:06:31.438]   github.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).processNextWorkItem()
I0103 22:06:31.439]       /go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:266 +0x4bf
I0103 22:06:31.439]   github.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).Run.func1()
I0103 22:06:31.439]       /go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:218 +0x38
I0103 22:06:31.439] 
I0103 22:06:31.439] Previous write at 0x00c4207d0138 by goroutine 56:
I0103 22:06:31.439]   github.com/knative/serving/pkg/reconciler/v1alpha1/revision.TestGlobalResyncOnConfigMapUpdate()
I0103 22:06:31.439]       /go/src/github.com/knative/serving/pkg/reconciler/v1alpha1/revision/revision_test.go:727 +0x86a
I0103 22:06:31.439]   testing.tRunner()
I0103 22:06:31.440]       /usr/local/go/src/testing/testing.go:777 +0x16d
I0103 22:06:31.440] 
I0103 22:06:31.440] Goroutine 150 (running) created at:
I0103 22:06:31.440]   github.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).Run()
I0103 22:06:31.440]       /go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:217 +0x14d
I0103 22:06:31.440] 
I0103 22:06:31.440] Goroutine 56 (running) created at:
I0103 22:06:31.440]   testing.(*T).Run()
I0103 22:06:31.441]       /usr/local/go/src/testing/testing.go:824 +0x564
I0103 22:06:31.441]   testing.runTests.func1()
I0103 22:06:31.441]       /usr/local/go/src/testing/testing.go:1063 +0xa4
I0103 22:06:31.441]   testing.tRunner()
I0103 22:06:31.441]       /usr/local/go/src/testing/testing.go:777 +0x16d
I0103 22:06:31.441]   testing.runTests()
I0103 22:06:31.441]       /usr/local/go/src/testing/testing.go:1061 +0x4e1
I0103 22:06:31.441]   testing.(*M).Run()
I0103 22:06:31.442]       /usr/local/go/src/testing/testing.go:978 +0x2cd
I0103 22:06:31.442]   main.main()
I0103 22:06:31.442]       _testmain.go:82 +0x22a
I0103 22:06:31.442] ==================

TestGlobalResyncOnUpdateDomainConfigMap/newprod.net has one as well:

I0104 02:06:04.645] ==================
I0104 02:06:04.645] WARNING: DATA RACE
I0104 02:06:04.645] Read at 0x00c420410068 by goroutine 109:
I0104 02:06:04.646]   runtime.convT2Estring()
I0104 02:06:04.646]       /usr/local/go/src/runtime/iface.go:354 +0x0
I0104 02:06:04.646]   github.com/knative/serving/pkg/reconciler/v1alpha1/route.TestGlobalResyncOnUpdateDomainConfigMap.func5.1()
I0104 02:06:04.646]       /go/src/github.com/knative/serving/pkg/reconciler/v1alpha1/route/route_test.go:944 +0x20c
I0104 02:06:04.646]   github.com/knative/serving/pkg/reconciler/testing.(*Hooks).OnUpdate.func1()
I0104 02:06:04.646]       /go/src/github.com/knative/serving/pkg/reconciler/testing/hooks.go:114 +0xd2
I0104 02:06:04.646]   github.com/knative/serving/vendor/k8s.io/client-go/testing.(*SimpleReactor).React()
I0104 02:06:04.646]       /go/src/github.com/knative/serving/vendor/k8s.io/client-go/testing/fixture.go:487 +0x64
I0104 02:06:04.646]   github.com/knative/serving/vendor/k8s.io/client-go/testing.(*Fake).Invokes()
I0104 02:06:04.646]       /go/src/github.com/knative/serving/vendor/k8s.io/client-go/testing/fake.go:140 +0x275
I0104 02:06:04.646]   github.com/knative/serving/pkg/client/clientset/versioned/typed/serving/v1alpha1/fake.(*FakeRoutes).Update()
I0104 02:06:04.647]       /go/src/github.com/knative/serving/pkg/client/clientset/versioned/typed/serving/v1alpha1/fake/fake_route.go:92 +0x297
I0104 02:06:04.647]   github.com/knative/serving/pkg/reconciler/v1alpha1/route.(*Reconciler).updateStatus()
I0104 02:06:04.647]       /go/src/github.com/knative/serving/pkg/reconciler/v1alpha1/route/reconcile_resources.go:157 +0x2c9
I0104 02:06:04.647]   github.com/knative/serving/pkg/reconciler/v1alpha1/route.(*Reconciler).Reconcile()
I0104 02:06:04.647]       /go/src/github.com/knative/serving/pkg/reconciler/v1alpha1/route/route.go:205 +0x353
I0104 02:06:04.647]   github.com/knative/serving/pkg/reconciler/v1alpha1/route.(*Reconciler).Reconcile()
I0104 02:06:04.647]       /go/src/github.com/knative/serving/pkg/reconciler/v1alpha1/route/route.go:199 +0x233
I0104 02:06:04.647]   github.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).processNextWorkItem()
I0104 02:06:04.647]       /go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:266 +0x4bf
I0104 02:06:04.647]   github.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).Run.func1()
I0104 02:06:04.648]       /go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:218 +0x38
I0104 02:06:04.648]   github.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).Run.func1()
I0104 02:06:04.648]       /go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:218 +0x38
I0104 02:06:04.648] 
I0104 02:06:04.648] Previous write at 0x00c420410068 by goroutine 28:
I0104 02:06:04.648]   github.com/knative/serving/pkg/reconciler/v1alpha1/route.TestGlobalResyncOnUpdateDomainConfigMap()
I0104 02:06:04.648]       /go/src/github.com/knative/serving/pkg/reconciler/v1alpha1/route/route_test.go:930 +0x24d
I0104 02:06:04.648]   testing.tRunner()
I0104 02:06:04.648]       /usr/local/go/src/testing/testing.go:777 +0x16d
I0104 02:06:04.648] 
I0104 02:06:04.648] Goroutine 109 (running) created at:
I0104 02:06:04.648]   github.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).Run()
I0104 02:06:04.649]       /go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:217 +0x14d
I0104 02:06:04.649] 
I0104 02:06:04.649] Goroutine 28 (running) created at:
I0104 02:06:04.649]   testing.(*T).Run()
I0104 02:06:04.649]       /usr/local/go/src/testing/testing.go:824 +0x564
I0104 02:06:04.649]   testing.runTests.func1()
I0104 02:06:04.649]       /usr/local/go/src/testing/testing.go:1063 +0xa4
I0104 02:06:04.649]   testing.tRunner()
I0104 02:06:04.649]       /usr/local/go/src/testing/testing.go:777 +0x16d
I0104 02:06:04.649]   testing.runTests()
I0104 02:06:04.649]       /usr/local/go/src/testing/testing.go:1061 +0x4e1
I0104 02:06:04.649]   testing.(*M).Run()
I0104 02:06:04.649]       /usr/local/go/src/testing/testing.go:978 +0x2cd
I0104 02:06:04.649]   main.main()
I0104 02:06:04.650]       _testmain.go:64 +0x22a
I0104 02:06:04.650] ==================

I'm going to continue to dig.

@markusthoemmes markusthoemmes mentioned this issue Jan 4, 2019
Merged
@eallred-google eallred-google modified the milestones: Needs Triage, Ice Box Oct 23, 2019
@dprotaso dprotaso removed this from the Ice Box milestone Oct 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/test-and-release It flags unit/e2e/conformance/perf test issues for product features kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Wait for informers to start to fix test flakes. markusthoemmes/knative-serving
5 participants
@dprotaso @markusthoemmes @mattmoor @knative-prow-robot @eallred-google