tsh 14.0 from macOS Sonoma cannot connect with certificate issues #32531

SMillerDev · 2023-09-26T02:19:08Z

Expected behavior:
tsh ssh --cluster cluster-x server-y will start an SSH session on a server

Current behavior:
After login in and authenticating with touchID

$ tsh ssh -d --cluster cluster-x server-y
2023-09-26T04:11:10+02:00 INFO [CLIENT]    ALPN connection upgrade required for "teleport.domain.com:443": true. client/api.go:723
2023-09-26T04:11:10+02:00 DEBU [CLIENT]    Skipping connection to the local ssh-agent. client/keyagent.go:137
2023-09-26T04:11:10+02:00 DEBU [KEYSTORE]  Reading certificates from path "/Users/sean.molenaar/.tsh/keys/teleport.domain.com/sean-ssh/cluster-x-cert.pub". client/keystore.go:354
2023-09-26T04:11:10+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-09-26 14:07:05 +0000 UTC". client/client_store.go:106
2023-09-26T04:11:10+02:00 INFO [KEYAGENT]  Loading SSH key for user "sean" and cluster "cluster-x". client/keyagent.go:196
2023-09-26T04:11:10+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-09-26 14:07:05 +0000 UTC". client/client_store.go:106
2023-09-26T04:11:10+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-09-26 14:07:05 +0000 UTC". client/client_store.go:106
2023-09-26T04:11:10+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-09-26 14:07:05 +0000 UTC". client/client_store.go:106
2023-09-26T04:11:10+02:00 DEBU             ALPN connection upgrade for teleport.domain.com:3023. client/alpn_conn_upgrade.go:187
2023-09-26T04:11:10+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-09-26 14:07:05 +0000 UTC". client/client_store.go:106
2023-09-26T04:11:10+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-09-26 14:07:05 +0000 UTC". client/client_store.go:106
2023-09-26T04:11:10+02:00 DEBU [CLIENT]    Attempting to issue a single-use user certificate with an MFA check. client/cluster_client.go:105

ERROR REPORT:
Original Error: transport.ConnectionError connection error: desc = &#34;transport: Error while dialing: failed to dial: unable to establish proxy stream\n\trpc error: code = Unavailable desc = connection error: desc = \&#34;transport: Error while dialing: tls: failed to verify certificate: x509: “7b5563ef-4099-494e-b21f-29476f71046f.cluster-x” certificate is not standards compliant\&#34;&#34;
Stack Trace:
        github.com/gravitational/teleport/api@v0.0.0/client/client.go:1617 github.com/gravitational/teleport/api/client.(*Client).GenerateUserSingleUseCerts
        github.com/gravitational/teleport/lib/client/cluster_client.go:241 github.com/gravitational/teleport/lib/client.(*ClusterClient).performMFACeremony
        github.com/gravitational/teleport/lib/client/cluster_client.go:106 github.com/gravitational/teleport/lib/client.(*ClusterClient).SessionSSHConfig
        github.com/gravitational/teleport/lib/client/api.go:1742 github.com/gravitational/teleport/lib/client.(*TeleportClient).connectToNodeWithMFA
        github.com/gravitational/teleport/lib/client/api.go:1623 github.com/gravitational/teleport/lib/client.(*TeleportClient).ConnectToNode.func2
        runtime/asm_arm64.s:1197 runtime.goexit
User Message: connection error: desc = &#34;transport: Error while dialing: failed to dial: unable to establish proxy stream\n\trpc error: code = Unavailable desc = connection error: desc = \&#34;transport: Error while dialing: tls: failed to verify certificate: x509: “7b5563ef-4099-494e-b21f-29476f71046f.cluster-x” certificate is not standards compliant\&#34;&#34;

Bug details:

Teleport version: 14.0.0 on client, cluster, trusted cluster and my local machine
Recreation steps: Not sure, it seems to just be: run tsh ssh from the latest macOS version.
Debug logs: See above

The text was updated successfully, but these errors were encountered:

zmb3 · 2023-09-26T18:34:55Z

Which version of tsh are you on exactly? I upgraded to Sonoma and don't see this behavior. Additionally, the line numbers in the stack trace shown above don't appear to match up with v14.0.0.

coltonparsons-vgs · 2023-09-26T18:47:17Z

I'm experiencing the same issue:

Version Information

tsh version

Teleport v14.0.0 git:v14.0.0-0-g599429b9 go1.21.1
Proxy version: 14.0.0
Proxy: teleport.vgs.io:443

edit: this is the tsh version downloaded from https://cdn.teleport.dev/tsh-14.0.0.pkg, installed by Homebrew.

Debug Output

tsh -d ls

2023-09-26T11:45:45-07:00 INFO [CLIENT]    ALPN connection upgrade required for "teleport.vgs.io:443": true. client/api.go:723
2023-09-26T11:45:45-07:00 INFO [CLIENT]    no host login given. defaulting to colton client/api.go:1060
2023-09-26T11:45:45-07:00 INFO [CLIENT]    [KEY AGENT] Connected to the system agent: "/private/tmp/com.apple.launchd.MwA3IdjUNY/Listeners" client/api.go:4562
2023-09-26T11:45:45-07:00 DEBU [KEYSTORE]  Reading certificates from path "/Users/colton/.tsh/keys/teleport.vgs.io/colton.parsons@verygoodsecurity.com-ssh/sso-okta-cert.pub". client/keystore.go:354
2023-09-26T11:45:45-07:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-09-27 06:44:27 +0000 UTC". client/client_store.go:106
2023-09-26T11:45:45-07:00 INFO [KEYAGENT]  Loading SSH key for user "colton.parsons@verygoodsecurity.com" and cluster "sso-okta". client/keyagent.go:196
2023-09-26T11:45:45-07:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-09-27 06:44:27 +0000 UTC". client/client_store.go:106
2023-09-26T11:45:45-07:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-09-27 06:44:27 +0000 UTC". client/client_store.go:106
2023-09-26T11:45:45-07:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-09-27 06:44:27 +0000 UTC". client/client_store.go:106
2023-09-26T11:45:45-07:00 DEBU             ALPN connection upgrade for teleport-trunk.vgs.io:3023. client/alpn_conn_upgrade.go:187
2023-09-26T11:45:45-07:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-09-27 06:44:27 +0000 UTC". client/client_store.go:106

ERROR REPORT:
Original Error: *trace.ConnectionProblemError connection error: desc = &#34;transport: Error while dialing: failed to dial: unable to establish proxy stream\n\trpc error: code = Unavailable desc = connection error: desc = \&#34;transport: Error while dialing: tls: failed to verify certificate: x509: “7fd50b84-120e-4cc0-82c5-b94a6469c74f.sso-okta” certificate is not standards compliant\&#34;&#34;
Stack Trace:
	github.com/gravitational/teleport/api@v0.0.0/client/client.go:3106 github.com/gravitational/teleport/api/client.(*Client).GetResources
	github.com/gravitational/teleport/api@v0.0.0/client/client.go:3238 github.com/gravitational/teleport/api/client.GetResourcePage[...]
	github.com/gravitational/teleport/api@v0.0.0/client/client.go:3305 github.com/gravitational/teleport/api/client.GetAllResources[...]
	github.com/gravitational/teleport/lib/client/api.go:2282 github.com/gravitational/teleport/lib/client.(*TeleportClient).ListNodesWithFilters
	github.com/gravitational/teleport/tool/tsh/common/tsh.go:2155 github.com/gravitational/teleport/tool/tsh/common.onListNodes.func1
	github.com/gravitational/teleport/lib/client/api.go:570 github.com/gravitational/teleport/lib/client.RetryWithRelogin
	github.com/gravitational/teleport/tool/tsh/common/tsh.go:2154 github.com/gravitational/teleport/tool/tsh/common.onListNodes
	github.com/gravitational/teleport/tool/tsh/common/tsh.go:1313 github.com/gravitational/teleport/tool/tsh/common.Run
	github.com/gravitational/teleport/tool/tsh/common/tsh.go:540 github.com/gravitational/teleport/tool/tsh/common.Main
	github.com/gravitational/teleport/tool/tsh/main.go:24 main.main
	runtime/proc.go:267 runtime.main
	runtime/asm_arm64.s:1197 runtime.goexit
User Message: connection error: desc = &#34;transport: Error while dialing: failed to dial: unable to establish proxy stream\n\trpc error: code = Unavailable desc = connection error: desc = \&#34;transport: Error while dialing: tls: failed to verify certificate: x509: “7fd50b84-120e-4cc0-82c5-b94a6469c74f.sso-okta” certificate is not standards compliant\&#34;&#34;

Downgrading to tsh v13 works around the issue for now.

coltonparsons-vgs · 2023-09-26T18:53:52Z

Side note: the key error message seems to be

certificate is not standards compliant

but I don't know which standards this refers to. (Is tsh v14 FIPS by default or something?)

zmb3 · 2023-09-26T19:15:26Z

Is tsh v14 FIPS by default or something?

No, this message comes from macOS if it doesn't like a certificate. Still looking in to what certificate it doesn't like and why.

SMillerDev · 2023-09-27T00:21:55Z

I had another think of my certificates but it might be good to also note that the clusters are all running CentOS 7 with a letsencrypt certificate in front that is not issued through teleport but separately and served by nginx.

vietvudanh · 2023-09-27T05:47:45Z

I had problem with tsh 14.0.0, MacOS Monterey (12.), self host proxy with let's encrypt.

Login was ok ,but when I tried tsh ls the log was:

ERROR: connection error: desc = "transport: Error while dialing: failed to dial: unable to establish proxy stream\n\trpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: tls: failed to verify certificate: x509: "uuid.<proxy host>" certificate is not standards compliant\""

When I downgraded tsh to 13.4.1 it connected just fine.

zmb3 · 2023-09-27T18:08:07Z

So tsh v13 would first attempt to connect with the new gRPC API that's supported in teleport 14, and would fall back to the old SSH connection method if the gRPC one failed. In tsh v14 the fallback was eliminated.

I suspect this certificate issue has also been present on tsh v13 for you and it went unnoticed due to the fallback behavior. (We could confirm this with a trace)

I'm not sure that the Sonoma upgrade has anything to do with this, and it might just be coincidence that the macOS update happened around the same time teleport was upgraded to v14. @vietvudanh's experience on Monterey seems to back this up.

coltonparsons-vgs · 2023-09-27T20:13:07Z

I may have narrowed down the issue a little. My company's setup uses a Load Balancer (w/TLS termination) for the initial connection, but Teleport also generates a self-signed cert internally. It looks like the self-signed cert is what's failing the MacOS CT Policy (at least according to this tool).

Excerpt from /etc/teleport.yaml:

...

proxy_service:
  enabled: yes

  listen_addr: 0.0.0.0:3023
  ssh_public_addr: tp-alpha-trunk.verygoodsecurity.io:3023

  tunnel_listen_addr: 0.0.0.0:3080
  tunnel_public_addr: tp-alpha-trunk.verygoodsecurity.io:3080

  web_listen_addr: 0.0.0.0:3080
  public_addr: tp-alpha.verygoodsecurity.io:443

Where tp-alpha.verygoodsecurity.io is the Load Balancer and tp-alpha-trunk.verygoodsecurity.io is the node itself.

Interestingly, setting tunnel_public_addr: tp-alpha.verygoodsecurity.io:443 (and restarting Teleport) did not appear to change the result.

The cert in question is the auto-generated /var/lib/teleport/webproxy_cert.pem, which I regenerated after upgrading the server to Teleport 14.0.0. ~~I'll see if I can find a way to test this file directly, right now I'm just assuming this is what's being used on port 3080.~~ edit: I've verified this is the cert being hosted on port 3080.

zmb3 · 2023-09-27T20:54:18Z

Thanks @coltonparsons-vgs - that was just enough to help me identify the issue. Fix is on the way.

As per https://support.apple.com/en-in/HT210176: > TLS server certificates must contain an ExtendedKeyUsage (EKU) extension containing the id-kp-serverAuth OID. We were not specifying this EKU. Validated by checking with the old self-signed certs: $ security verify-cert -c webproxy_cert.pem -p ssl -r webproxy_cert.pem Cert Verify Result: Invalid Extended Key Usage for policy And then repeating the process after this change: $ security verify-cert -c webproxy_cert.pem -p ssl -r webproxy_cert.pem ...certificate verification successful. Closes #32531

keenan-v1 · 2023-10-10T17:27:38Z

Curious, any word on when this will be released?

apiening · 2023-10-11T10:18:18Z

I have the same issue. Only downgrading to teleport 13.x solved it for me.
I will be happy to try the fixed version as soon as it is released. Please let us know when it is available.

webvictim · 2023-10-11T17:09:39Z

This should be in 14.0.3 which will be released this week.

keenan-v1 · 2023-10-11T21:48:30Z

I just downloaded tsh 14.0.3 and tested it, I'm still getting this error on macOS Ventura 13.6:

ERROR: connection error: desc = "transport: Error while dialing: failed to dial: unable to establish proxy stream\n\trpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: tls: failed to verify certificate: x509: “<uuid>.<proxy-host>” certificate is not standards compliant\""

Like the person earlier in this thread, we use a load balancer with TLS termination.

coltonparsons-vgs · 2023-10-12T01:47:31Z

I can confirm I'm having the same issue as keenan-v1 (tsh 14.0.3 on MacOS Ventura 13.6).

I'll see if I can get a Sonoma box going to test it there.

edit: issue persists after updating the Teleport servers to 14.0.3 and regenerating webproxy_key.pem and webproxy_cert.pem.

SMillerDev · 2023-10-12T06:47:15Z

Can't reopen it, but after updating the proxy host and my local tsh install to 14.0.3 and restarting teleport on the proxy host I now get it on tsh login --proxy <host> as well, so... not much improved for me.

webvictim · 2023-10-12T11:58:23Z

Could someone having this issue share the full output of their tsh command when using tsh --debug please?

SMillerDev · 2023-10-12T12:01:24Z

Replaced the target host with host1 and the proxy with bastion, but otherwise this is the full output.

~ $ tsh ssh --debug --cluster m2mobi host1.m2mobi.com
2023-10-12T13:59:02+02:00 INFO [CLIENT]    ALPN connection upgrade required for "bastion.m2mobi.com:443": true. client/api.go:723
2023-10-12T13:59:02+02:00 DEBU [CLIENT]    Skipping connection to the local ssh-agent. client/keyagent.go:137
2023-10-12T13:59:02+02:00 DEBU [KEYSTORE]  Reading certificates from path "/Users/sean.molenaar/.tsh/keys/bastion.m2mobi.com/sean-ssh/m2mobi-cert.pub". client/keystore.go:355
2023-10-12T13:59:02+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-10-12 18:43:42 +0000 UTC". client/client_store.go:106
2023-10-12T13:59:02+02:00 INFO [KEYAGENT]  Loading SSH key for user "sean" and cluster "m2mobi". client/keyagent.go:196
2023-10-12T13:59:02+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-10-12 18:43:42 +0000 UTC". client/client_store.go:106
2023-10-12T13:59:02+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-10-12 18:43:42 +0000 UTC". client/client_store.go:106
2023-10-12T13:59:02+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-10-12 18:43:42 +0000 UTC". client/client_store.go:106
2023-10-12T13:59:02+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-10-12 18:43:42 +0000 UTC". client/client_store.go:106
2023-10-12T13:59:02+02:00 DEBU             ALPN connection upgrade for bastion.m2mobi.com:3023. client/alpn_conn_upgrade.go:187
2023-10-12T13:59:02+02:00 DEBU [KEYSTORE]  Reading certificates from path "/Users/sean.molenaar/.tsh/keys/bastion.m2mobi.com/sean-ssh/m2mobi-cert.pub". client/keystore.go:355
2023-10-12T13:59:02+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-10-12 18:43:42 +0000 UTC". client/client_store.go:106
2023-10-12T13:59:02+02:00 DEBU [CLIENT]    Attempting to issue a single-use user certificate with an MFA check. client/cluster_client.go:135

ERROR REPORT:
Original Error: transport.ConnectionError connection error: desc = &#34;transport: Error while dialing: failed to dial: unable to establish proxy stream\n\trpc error: code = Unavailable desc = connection error: desc = \&#34;transport: Error while dialing: tls: failed to verify certificate: x509: “7b5563ef-4099-494e-b21f-29476f71046f.m2mobi” certificate is not standards compliant\&#34;&#34;
Stack Trace:
        github.com/gravitational/teleport/api@v0.0.0/client/client.go:1630 github.com/gravitational/teleport/api/client.(*Client).GenerateUserSingleUseCerts
        github.com/gravitational/teleport/lib/client/cluster_client.go:271 github.com/gravitational/teleport/lib/client.(*ClusterClient).performMFACeremony
        github.com/gravitational/teleport/lib/client/cluster_client.go:136 github.com/gravitational/teleport/lib/client.(*ClusterClient).SessionSSHConfig
        github.com/gravitational/teleport/lib/client/api.go:1738 github.com/gravitational/teleport/lib/client.(*TeleportClient).connectToNodeWithMFA
        github.com/gravitational/teleport/lib/client/api.go:1619 github.com/gravitational/teleport/lib/client.(*TeleportClient).ConnectToNode.func2
        runtime/asm_arm64.s:1197 runtime.goexit
User Message: connection error: desc = &#34;transport: Error while dialing: failed to dial: unable to establish proxy stream\n\trpc error: code = Unavailable desc = connection error: desc = \&#34;transport: Error while dialing: tls: failed to verify certificate: x509: “7b5563ef-4099-494e-b21f-29476f71046f.m2mobi” certificate is not standards compliant\&#34;&#34;

webvictim · 2023-10-12T12:07:23Z

@greedy52 Do you think this could be related to connection upgrades somehow?

greedy52 · 2023-10-12T14:15:54Z

2023-10-12T13:59:02+02:00 INFO [CLIENT] ALPN connection upgrade required for "bastion.m2mobi.com:443": true. client/api.go:723
2023-10-12T13:59:02+02:00 DEBU ALPN connection upgrade for bastion.m2mobi.com:3023. client/alpn_conn_upgrade.go:187

Notice the port is different. Same thing here.

2023-09-26T11:45:45-07:00 INFO [CLIENT] ALPN connection upgrade required for "teleport.vgs.io:443": true. client/api.go:723
2023-09-26T04:11:10+02:00 DEBU ALPN connection upgrade for teleport.domain.com:3023. client/alpn_conn_upgrade.go:187

So I am assuming TLS routing is not enabled. Connection upgrade should NOT be applied to 3023.

I am setting up this (separate port env with webport behind ALB), will update this comment with my findings.

-- update
Yes. I am able to repro with a separate port setup:

ERROR REPORT: Original Error: trace.aggregate connection error: desc = "transport: Error while dialing: failed to dial: unable to establish proxy stream\n\trpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: tls: failed to verify certificate: x509: “5dbb380c-4a16-4062-bf9f-40090b6bace6.teleport.root.dev.aws.stevexin.me” certificate is not standards compliant\""

I can confirm that api/client/proxy.newDialerForGRPCClient is getting <>:3023 as ProxyAddress and connection upgrade is enabled.

keenan-v1 · 2023-10-20T16:05:34Z

Just popping back to say everything is working now. Thank you!

SMillerDev · 2023-10-21T07:47:35Z

I just tried, same for me. Thanks everyone!

coltonparsons-vgs · 2023-10-23T15:47:29Z

I can confirm as well, tsh 14.1.0 works perfectly for us. Thank you!

vietvudanh · 2023-11-08T06:54:33Z

Thank team, working well with tsh 14.1.1.

SMillerDev added the bug label Sep 26, 2023

zmb3 added tsh tsh - Teleport's command line tool for logging into nodes running Teleport. mac labels Sep 26, 2023

zmb3 mentioned this issue Sep 27, 2023

Fix self-signed cert validity on macOS systems #32698

Merged

zmb3 closed this as completed in #32698 Oct 9, 2023

webvictim reopened this Oct 12, 2023

greedy52 mentioned this issue Oct 12, 2023

Fix an issue tsh fails when Proxy is in separate mode and web port is TLS-terminated by a load balancer #33374

Merged

greedy52 closed this as completed in #33374 Oct 12, 2023

camscale mentioned this issue Oct 16, 2023

Release 14.1.0 #33507

Merged

BrewTestBot mentioned this issue Oct 19, 2023

teleport 14.1.0 Homebrew/homebrew-core#151786

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tsh 14.0 from macOS Sonoma cannot connect with certificate issues #32531

tsh 14.0 from macOS Sonoma cannot connect with certificate issues #32531

SMillerDev commented Sep 26, 2023

zmb3 commented Sep 26, 2023

coltonparsons-vgs commented Sep 26, 2023 •

edited

coltonparsons-vgs commented Sep 26, 2023

zmb3 commented Sep 26, 2023

SMillerDev commented Sep 27, 2023

vietvudanh commented Sep 27, 2023 •

edited

zmb3 commented Sep 27, 2023 •

edited

coltonparsons-vgs commented Sep 27, 2023 •

edited

zmb3 commented Sep 27, 2023

keenan-v1 commented Oct 10, 2023

apiening commented Oct 11, 2023

webvictim commented Oct 11, 2023

keenan-v1 commented Oct 11, 2023

coltonparsons-vgs commented Oct 12, 2023 •

edited

SMillerDev commented Oct 12, 2023

webvictim commented Oct 12, 2023

SMillerDev commented Oct 12, 2023

webvictim commented Oct 12, 2023

greedy52 commented Oct 12, 2023 •

edited

keenan-v1 commented Oct 20, 2023

SMillerDev commented Oct 21, 2023

coltonparsons-vgs commented Oct 23, 2023

vietvudanh commented Nov 8, 2023

tsh 14.0 from macOS Sonoma cannot connect with certificate issues #32531

tsh 14.0 from macOS Sonoma cannot connect with certificate issues #32531

Comments

SMillerDev commented Sep 26, 2023

zmb3 commented Sep 26, 2023

coltonparsons-vgs commented Sep 26, 2023 • edited

Version Information

Debug Output

coltonparsons-vgs commented Sep 26, 2023

zmb3 commented Sep 26, 2023

SMillerDev commented Sep 27, 2023

vietvudanh commented Sep 27, 2023 • edited

zmb3 commented Sep 27, 2023 • edited

coltonparsons-vgs commented Sep 27, 2023 • edited

zmb3 commented Sep 27, 2023

keenan-v1 commented Oct 10, 2023

apiening commented Oct 11, 2023

webvictim commented Oct 11, 2023

keenan-v1 commented Oct 11, 2023

coltonparsons-vgs commented Oct 12, 2023 • edited

SMillerDev commented Oct 12, 2023

webvictim commented Oct 12, 2023

SMillerDev commented Oct 12, 2023

webvictim commented Oct 12, 2023

greedy52 commented Oct 12, 2023 • edited

keenan-v1 commented Oct 20, 2023

SMillerDev commented Oct 21, 2023

coltonparsons-vgs commented Oct 23, 2023

vietvudanh commented Nov 8, 2023

coltonparsons-vgs commented Sep 26, 2023 •

edited

vietvudanh commented Sep 27, 2023 •

edited

zmb3 commented Sep 27, 2023 •

edited

coltonparsons-vgs commented Sep 27, 2023 •

edited

coltonparsons-vgs commented Oct 12, 2023 •

edited

greedy52 commented Oct 12, 2023 •

edited