Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

server: TestInitHandshake failed #114158

Closed
cockroach-teamcity opened this issue Nov 9, 2023 · 5 comments · Fixed by #115355
Closed

server: TestInitHandshake failed #114158

cockroach-teamcity opened this issue Nov 9, 2023 · 5 comments · Fixed by #115355
Assignees
@andrewbaptist
Labels
branch-release-22.2.17-rc C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team
Projects
KV
Milestone
22.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Nov 9, 2023 •
edited by rickystewart

server.TestInitHandshake failed with artifacts on release-22.2.17-rc @ 272d49bece0b5debdde6e5b537c50a1314b389d2:

        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:667
        	            	  |   | github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:91
        	            	  |   | github.com/cockroachdb/cockroach/pkg/server.InitHandshake
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:665
        	            	  |   | github.com/cockroachdb/cockroach/pkg/server.TestInitHandshake.func7
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake_test.go:107
        	            	  |   | runtime.goexit
        	            	  |   | 	GOROOT/src/runtime/asm_amd64.s:1594
        	            	  | Wraps: (3) context canceled before init bundle received from leader
        	            	  | Error types: (1) *contextutil.TimeoutError (2) *withstack.withStack (3) *errutil.leafError
        	            	Wraps: (2) secondary error attachment
        	            	  | operation "init handshake" timed out after 10m0.001s (given timeout 10m0s): context canceled before init bundle received from leader
        	            	  | (1) operation "init handshake" timed out after 10m0.001s (given timeout 10m0s)
        	            	  | Wraps: (2) attached stack trace
        	            	  |   -- stack trace:
        	            	  |   | github.com/cockroachdb/cockroach/pkg/server.initHandshakeHelper
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:646
        	            	  |   | github.com/cockroachdb/cockroach/pkg/server.InitHandshake.func1
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:667
        	            	  |   | github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:91
        	            	  |   | github.com/cockroachdb/cockroach/pkg/server.InitHandshake
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:665
        	            	  |   | github.com/cockroachdb/cockroach/pkg/server.TestInitHandshake.func6
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake_test.go:104
        	            	  |   | runtime.goexit
        	            	  |   | 	GOROOT/src/runtime/asm_amd64.s:1594
        	            	  | Wraps: (3) context canceled before init bundle received from leader
        	            	  | Error types: (1) *contextutil.TimeoutError (2) *withstack.withStack (3) *errutil.leafError
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/server.initHandshakeHelper
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:554
        	            	  | github.com/cockroachdb/cockroach/pkg/server.InitHandshake.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:667
        	            	  | github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:91
        	            	  | github.com/cockroachdb/cockroach/pkg/server.InitHandshake
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:665
        	            	  | github.com/cockroachdb/cockroach/pkg/server.TestInitHandshake.func5
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake_test.go:101
        	            	  | runtime.goexit
        	            	  | 	GOROOT/src/runtime/asm_amd64.s:1594
        	            	Wraps: (4) invalid signature in messages from peers; likely due to token mismatch
        	            	Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *errutil.leafError
        	Test:       	TestInitHandshake
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/5b2c9b3a394428c7572d34050aad8975/logTestInitHandshake285367254
--- FAIL: TestInitHandshake (600.03s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/kv @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-33372
@cockroach-teamcity cockroach-teamcity added branch-release-22.2.17-rc C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Nov 9, 2023
@cockroach-teamcity cockroach-teamcity added this to the 22.2 milestone Nov 9, 2023
@cockroach-teamcity cockroach-teamcity added this to roachtest/unit test backlog in KV Nov 9, 2023
@miraradeva
Copy link
Contributor

Looks like we looked at this recently (#105606) and decided it's not a CY issue or a release blocker.

@nvanbenschoten nvanbenschoten removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Nov 11, 2023
@kvoli
Copy link
Collaborator

kvoli commented Nov 13, 2023

Adding c-bug based on prior issue #105606.

@kvoli kvoli added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Nov 13, 2023
@andrewbaptist
Copy link
Collaborator

I'm going to delete all these tests in 22.2, 23.1 and 23.2 as soon as we make a final decision and merge #113893 to master. This functionality is already marked as deprecated in 23.2 since #114153 is merged, so its not worth spending time to fix this test.

The only remaining concern was if someone is using this functionality on a "community" version, but there is no easy way to know this, and hopefuly with the deprecation in 23.2 they will stop soon.

@andrewbaptist andrewbaptist self-assigned this Nov 13, 2023
@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 20, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

server.TestInitHandshake failed with artifacts on release-22.2.17-rc @ 0ad77cfcc5a81b96dd2f27ed608536388b680537:

    init_handshake_test.go:96: n3: generating cert bundle for cluster
    init_handshake_test.go:96: n3: sending cert bundle to peers
    init_handshake_test.go:96: n1: received cert bundle
    init_handshake_test.go:122: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/server/init_handshake_test.go:122
        	Error:      	Received unexpected error:
        	            	invalid signature in messages from peers; likely due to token mismatch
        	            	(1) secondary error attachment
        	            	  | operation "init handshake" timed out after 10m0.001s (given timeout 10m0s): error when sending bundle to peers as leader: Post "https://127.0.0.1:44793/deliverBundle/": dial tcp 127.0.0.1:44793: connect: connection refused
        	            	  | (1) operation "init handshake" timed out after 10m0.001s (given timeout 10m0s)
        	            	  | Wraps: (2) attached stack trace
        	            	  |   -- stack trace:
        	            	  |   | github.com/cockroachdb/cockroach/pkg/server.initHandshakeHelper
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:621
        	            	  |   | github.com/cockroachdb/cockroach/pkg/server.InitHandshake.func1
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:667
        	            	  |   | github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:91
        	            	  |   | github.com/cockroachdb/cockroach/pkg/server.InitHandshake
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:665
        	            	  |   | github.com/cockroachdb/cockroach/pkg/server.TestInitHandshake.func7
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake_test.go:107
        	            	  |   | runtime.goexit
        	            	  |   | 	GOROOT/src/runtime/asm_amd64.s:1594
        	            	  | Wraps: (3) error when sending bundle to peers as leader
        	            	  | Wraps: (4) Post "https://127.0.0.1:44793/deliverBundle/"
        	            	  | Wraps: (5) dial tcp 127.0.0.1:44793
        	            	  | Wraps: (6) connect
        	            	  | Wraps: (7) connection refused
        	            	  | Error types: (1) *contextutil.TimeoutError (2) *withstack.withStack (3) *errutil.withPrefix (4) *url.Error (5) *net.OpError (6) *os.SyscallError (7) syscall.Errno
        	            	Wraps: (2) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/server.initHandshakeHelper
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:554
        	            	  | github.com/cockroachdb/cockroach/pkg/server.InitHandshake.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:667
        	            	  | github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout
        	            	  | 	github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:91
        	            	  | github.com/cockroachdb/cockroach/pkg/server.InitHandshake
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake.go:665
        	            	  | github.com/cockroachdb/cockroach/pkg/server.TestInitHandshake.func6
        	            	  | 	github.com/cockroachdb/cockroach/pkg/server/init_handshake_test.go:104
        	            	  | runtime.goexit
        	            	  | 	GOROOT/src/runtime/asm_amd64.s:1594
        	            	Wraps: (3) invalid signature in messages from peers; likely due to token mismatch
        	            	Error types: (1) *secondary.withSecondaryError (2) *withstack.withStack (3) *errutil.leafError
        	Test:       	TestInitHandshake
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/5b2c9b3a394428c7572d34050aad8975/logTestInitHandshake3712361803
--- FAIL: TestInitHandshake (600.03s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@andrewbaptist andrewbaptist mentioned this issue Nov 30, 2023
Merged
@kvoli
Copy link
Collaborator

kvoli commented Dec 1, 2023

Assigning P3, as we are removing this test.

@kvoli kvoli added the P-3 Issues/test failures with no fix SLA label Dec 1, 2023
craig bot pushed a commit that referenced this issue Dec 5, 2023
@aadityasondhi @andrewbaptist
Merge #114446 #115355
84dcf70 
114446: roachtest: deflake admission_control_intent_resolution r=sumeerbhola a=aadityasondhi

Previously when sampling the state of the progress made by intent
resolution, we were asserting on L0 sublevel count to be under 20. We
noticed that a periodic slowness in compactions may pass this threshold
and made the test flaky.

This patch uses the mean over the last two minutes to determine if the
assertion fails. It also increases the sampling rate to every 10s to get
more sample data.

Fixes: #114011

Release note: None

115355: server: remove TestInitHandshake r=kvoli a=andrewbaptist

The TestInitHandshake test is flakey and the functionality is deprecated and will be removed soon. Rather than trying to track down and fix the test, it is easier to remove the test.

Epic: none
Fixes: #114726
Fixes: #114158
Fixes: #113046

Release note: None

Co-authored-by: Aaditya Sondhi <20070511+aadityasondhi@users.noreply.github.com>
Co-authored-by: Andrew Baptist <baptist@cockroachlabs.com>
@craig craig bot closed this as completed in fca265d Dec 5, 2023
KV automation moved this from roachtest/unit test backlog to Closed Dec 5, 2023
blathers-crl bot pushed a commit that referenced this issue Dec 5, 2023
@andrewbaptist
server: remove TestInitHandshake
2916156 
The TestInitHandshake test is flakey and the functionality is deprecated
and will be removed soon. Rather than trying to track down and fix the
test, it is easier to remove the test.

Epic: none
Fixes: #114726
Fixes: #114158
Fixes: #113046

Release note: None
This was referenced Dec 5, 2023
Merged
Merged
blathers-crl bot pushed a commit that referenced this issue Dec 5, 2023
@andrewbaptist
server: remove TestInitHandshake
0766952 
The TestInitHandshake test is flakey and the functionality is deprecated
and will be removed soon. Rather than trying to track down and fix the
test, it is easier to remove the test.

Epic: none
Fixes: #114726
Fixes: #114158
Fixes: #113046

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Dec 5, 2023
Merged
blathers-crl bot pushed a commit that referenced this issue Dec 5, 2023
@andrewbaptist
server: remove TestInitHandshake
5ac9a22 
The TestInitHandshake test is flakey and the functionality is deprecated
and will be removed soon. Rather than trying to track down and fix the
test, it is easier to remove the test.

Epic: none
Fixes: #114726
Fixes: #114158
Fixes: #113046

Release note: None
blathers-crl bot pushed a commit that referenced this issue Dec 16, 2023
@andrewbaptist
server: remove TestInitHandshake
12a5fad 
The TestInitHandshake test is flakey and the functionality is deprecated
and will be removed soon. Rather than trying to track down and fix the
test, it is easier to remove the test.

Epic: none
Fixes: #114726
Fixes: #114158
Fixes: #113046

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Dec 16, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andrewbaptist andrewbaptist

Labels
branch-release-22.2.17-rc C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team
Projects
KV
Closed
Milestone
22.2
Development

Successfully merging a pull request may close this issue.

server: remove TestInitHandshake andrewbaptist/cockroach
5 participants
@nvanbenschoten @cockroach-teamcity @kvoli @andrewbaptist @miraradeva