Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestSSHLoadAllCAs/TLS_routing_enabled flakiness #24185

Closed
tigrato opened this issue Apr 6, 2023 · 3 comments · Fixed by #24400
Closed

TestSSHLoadAllCAs/TLS_routing_enabled flakiness #24185

tigrato opened this issue Apr 6, 2023 · 3 comments · Fixed by #24400
Assignees
@rosstimothy
Labels
flaky tests

Comments

@tigrato
Copy link
Contributor

tigrato commented Apr 6, 2023 •
edited

Failure

Link(s) to logs

Relevant snippet

WARNING: DATA RACE
Write at 0x00c001601aa0 by goroutine 89741:
  github.com/gravitational/teleport/lib/client.(*proxyClusterGuesser).hostKeyCallback()
      /__w/teleport/teleport/lib/client/api.go:3082 +0xec
  github.com/gravitational/teleport/lib/client.(*proxyClusterGuesser).hostKeyCallback-fm()
      <autogenerated>:1 +0x92
  golang.org/x/crypto/ssh.(*handshakeTransport).client()
      /go/pkg/mod/github.com/gravitational/crypto@v0.6.0-1/ssh/handshake.go:729 +0x27d
  golang.org/x/crypto/ssh.(*handshakeTransport).enterKeyExchange()
      /go/pkg/mod/github.com/gravitational/crypto@v0.6.0-1/ssh/handshake.go:623 +0x7d7
  golang.org/x/crypto/ssh.(*handshakeTransport).kexLoop()
      /go/pkg/mod/github.com/gravitational/crypto@v0.6.0-1/ssh/handshake.go:303 +0x124
  golang.org/x/crypto/ssh.newClientTransport.func2()
      /go/pkg/mod/github.com/gravitational/crypto@v0.6.0-1/ssh/handshake.go:137 +0x39

Previous write at 0x00c001601aa0 by goroutine 89736:
  github.com/gravitational/teleport/lib/client.(*proxyClusterGuesser).hostKeyCallback()
      /__w/teleport/teleport/lib/client/api.go:3082 +0xec
  github.com/gravitational/teleport/lib/client.(*proxyClusterGuesser).hostKeyCallback-fm()
      <autogenerated>:1 +0x92
  golang.org/x/crypto/ssh.(*handshakeTransport).client()
      /go/pkg/mod/github.com/gravitational/crypto@v0.6.0-1/ssh/handshake.go:729 +0x27d
  golang.org/x/crypto/ssh.(*handshakeTransport).enterKeyExchange()
      /go/pkg/mod/github.com/gravitational/crypto@v0.6.0-1/ssh/handshake.go:623 +0x7d7
  golang.org/x/crypto/ssh.(*handshakeTransport).kexLoop()
      /go/pkg/mod/github.com/gravitational/crypto@v0.6.0-1/ssh/handshake.go:303 +0x124
  golang.org/x/crypto/ssh.newClientTransport.func2()
      /go/pkg/mod/github.com/gravitational/crypto@v0.6.0-1/ssh/handshake.go:137 +0x39

Goroutine 89741 (running) created at:
  golang.org/x/crypto/ssh.newClientTransport()
      /go/pkg/mod/github.com/gravitational/crypto@v0.6.0-1/ssh/handshake.go:137 +0x4ae
  golang.org/x/crypto/ssh.(*connection).clientHandshake()
      /go/pkg/mod/github.com/gravitational/crypto@v0.6.0-1/ssh/client.go:105 +0x48c
  golang.org/x/crypto/ssh.NewClientConn()
      /go/pkg/mod/github.com/gravitational/crypto@v0.6.0-1/ssh/client.go:83 +0x28a
  github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConn()
      /__w/teleport/teleport/api/observability/tracing/ssh/ssh.go:160 +0xba8
  github.com/gravitational/teleport/lib/client.newClientConn.func1()
      /__w/teleport/teleport/lib/client/client.go:1762 +0x130

Goroutine 89736 (running) created at:
  golang.org/x/crypto/ssh.newClientTransport()
      /go/pkg/mod/github.com/gravitational/crypto@v0.6.0-1/ssh/handshake.go:137 +0x4ae
  golang.org/x/crypto/ssh.(*connection).clientHandshake()
      /go/pkg/mod/github.com/gravitational/crypto@v0.6.0-1/ssh/client.go:105 +0x48c
  golang.org/x/crypto/ssh.NewClientConn()
      /go/pkg/mod/github.com/gravitational/crypto@v0.6.0-1/ssh/client.go:83 +0x28a
  github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConn()
      /__w/teleport/teleport/api/observability/tracing/ssh/ssh.go:160 +0xba8
  github.com/gravitational/teleport/lib/client.newClientConn.func1()
      /__w/teleport/teleport/lib/client/client.go:1762 +0x130
==================
...
=== RUN   TestSSHLoadAllCAs/TLS_routing_enabled
=== PAUSE TestSSHLoadAllCAs/TLS_routing_enabled
=== CONT  TestSSHLoadAllCAs/TLS_routing_enabled
    testing.go:1446: race detected during execution of test
--- FAIL: TestSSHLoadAllCAs/TLS_routing_enabled (12.43s)
@Tener
Copy link
Contributor

Tener commented Apr 6, 2023 •
edited

https://github.com/gravitational/teleport/actions/runs/4628282381/jobs/8187171142?pr=23305

Bit weird as there is little value in the logs, but it may be related.

@rosstimothy
Copy link
Contributor

I think this should be fixed by #24195

@mdwn
Copy link
Contributor

mdwn commented Apr 11, 2023

It looks like I just hit this against master: https://github.com/gravitational/teleport/actions/runs/4668676892/jobs/8266070260?pr=24377

@mdwn mdwn reopened this Apr 11, 2023
@rosstimothy rosstimothy assigned rosstimothy and unassigned Joerger Apr 11, 2023
rosstimothy added a commit that referenced this issue Apr 11, 2023
@rosstimothy
Prevent races when using WithoutJumpHosts
2cfdd3a 
Instead of modifying the current `TeleportClient` to have no jump
hosts, a new one is cloned from the existing one without the jump
hosts set.

Closes #24185
@rosstimothy rosstimothy mentioned this issue Apr 11, 2023
Merged
@GavinFrazar GavinFrazar mentioned this issue Aug 29, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rosstimothy rosstimothy

Labels
flaky tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Prevent races when using WithoutJumpHosts gravitational/teleport
5 participants
@Tener @mdwn @tigrato @Joerger @rosstimothy