Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mTLS connection fails with @grpc/grpc-js and succeeds with grpc #1784

Closed
bergundy opened this issue May 11, 2021 · 6 comments
Closed

mTLS connection fails with @grpc/grpc-js and succeeds with grpc #1784

bergundy opened this issue May 11, 2021 · 6 comments
Labels
package: @grpc/grpc-js

Comments

@bergundy
Copy link

Problem description

I've set up a golang server with mTLS using the sample from this repo: https://github.com/temporalio/customization-samples/tree/master/tls/tls-full

I can connect to the server with the golang client and the native grpc node client but not with @grpc/grpc-js.
I'm getting the following error with no further details:

Error: 14 UNAVAILABLE: No connection established

Reproduction steps

  1. Clone the linked repo above
  2. cd tls/tls-full
  3. bash generate-certs.sh
  4. bash start-temporal.sh
  5. Run this script in the tls-full directory

Environment

  • OS name, version and architecture: MacOS arm64 (although I've seen this reproduced with x86 too)
  • Node version: 15.14.0 installed with nvm
  • Package name and version @grpc/grpc-js@1.3.0
@murgatroid99
Copy link
Member

Can you run your failing client with the environment variables GRPC_TRACE=all and GRPC_VERBOSITY=DEBUG and share the output?

@bergundy
Copy link
Author

Sure, here it is, thanks.

2021-05-12T07:22:29.961Z | connectivity_state | dns:127.0.0.1:7233 IDLE -> IDLE
2021-05-12T07:22:29.961Z | dns_resolver | Resolver constructed for target dns:127.0.0.1:7233
2021-05-12T07:22:29.968Z | dns_resolver | Resolution update requested for target dns:127.0.0.1:7233
2021-05-12T07:22:29.968Z | dns_resolver | Returning IP address for target dns:127.0.0.1:7233
2021-05-12T07:22:29.968Z | resolving_load_balancer | dns:127.0.0.1:7233 IDLE -> CONNECTING
2021-05-12T07:22:29.968Z | connectivity_state | dns:127.0.0.1:7233 IDLE -> CONNECTING
2021-05-12T07:22:29.968Z | resolving_load_balancer | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:29.968Z | connectivity_state | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:29.968Z | pick_first | Connect to address list 127.0.0.1:7233
2021-05-12T07:22:29.968Z | subchannel_refcount | 127.0.0.1:7233 refcount 0 -> 1
2021-05-12T07:22:29.968Z | subchannel_refcount | 127.0.0.1:7233 refcount 1 -> 2
2021-05-12T07:22:29.968Z | pick_first | Start connecting to subchannel with address 127.0.0.1:7233
2021-05-12T07:22:29.968Z | pick_first | IDLE -> CONNECTING
2021-05-12T07:22:29.969Z | resolving_load_balancer | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:29.969Z | connectivity_state | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:29.969Z | subchannel | 127.0.0.1:7233 IDLE -> CONNECTING
2021-05-12T07:22:29.969Z | pick_first | CONNECTING -> CONNECTING
2021-05-12T07:22:29.969Z | resolving_load_balancer | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:29.969Z | connectivity_state | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:30.064Z | subchannel | 127.0.0.1:7233 connection closed with error unable to get issuer certificate
2021-05-12T07:22:30.064Z | subchannel | 127.0.0.1:7233 CONNECTING -> TRANSIENT_FAILURE
2021-05-12T07:22:30.064Z | pick_first | CONNECTING -> TRANSIENT_FAILURE
2021-05-12T07:22:30.064Z | resolving_load_balancer | dns:127.0.0.1:7233 CONNECTING -> TRANSIENT_FAILURE
2021-05-12T07:22:30.064Z | connectivity_state | dns:127.0.0.1:7233 CONNECTING -> TRANSIENT_FAILURE
2021-05-12T07:22:30.974Z | subchannel | 127.0.0.1:7233 TRANSIENT_FAILURE -> IDLE
2021-05-12T07:22:30.975Z | subchannel_refcount | 127.0.0.1:7233 refcount 2 -> 1
2021-05-12T07:22:30.975Z | pick_first | TRANSIENT_FAILURE -> IDLE
2021-05-12T07:22:30.975Z | resolving_load_balancer | dns:127.0.0.1:7233 TRANSIENT_FAILURE -> IDLE
2021-05-12T07:22:30.975Z | connectivity_state | dns:127.0.0.1:7233 TRANSIENT_FAILURE -> IDLE
2021-05-12T07:22:30.975Z | dns_resolver | Resolution update requested for target dns:127.0.0.1:7233
2021-05-12T07:22:30.975Z | dns_resolver | Returning IP address for target dns:127.0.0.1:7233
2021-05-12T07:22:30.975Z | resolving_load_balancer | dns:127.0.0.1:7233 IDLE -> CONNECTING
2021-05-12T07:22:30.975Z | connectivity_state | dns:127.0.0.1:7233 IDLE -> CONNECTING
2021-05-12T07:22:30.975Z | resolving_load_balancer | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:30.976Z | connectivity_state | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:30.977Z | pick_first | Connect to address list 127.0.0.1:7233
2021-05-12T07:22:30.977Z | subchannel_refcount | 127.0.0.1:7233 refcount 1 -> 2
2021-05-12T07:22:30.977Z | pick_first | Start connecting to subchannel with address 127.0.0.1:7233
2021-05-12T07:22:30.977Z | pick_first | IDLE -> CONNECTING
2021-05-12T07:22:30.977Z | resolving_load_balancer | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:30.977Z | connectivity_state | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:30.977Z | subchannel | 127.0.0.1:7233 IDLE -> CONNECTING
2021-05-12T07:22:30.978Z | pick_first | CONNECTING -> CONNECTING
2021-05-12T07:22:30.978Z | resolving_load_balancer | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:30.978Z | connectivity_state | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:31.087Z | subchannel | 127.0.0.1:7233 connection closed with error unable to get issuer certificate
2021-05-12T07:22:31.087Z | subchannel | 127.0.0.1:7233 CONNECTING -> TRANSIENT_FAILURE
2021-05-12T07:22:31.087Z | pick_first | CONNECTING -> TRANSIENT_FAILURE
2021-05-12T07:22:31.087Z | resolving_load_balancer | dns:127.0.0.1:7233 CONNECTING -> TRANSIENT_FAILURE
2021-05-12T07:22:31.087Z | connectivity_state | dns:127.0.0.1:7233 CONNECTING -> TRANSIENT_FAILURE
2021-05-12T07:22:32.510Z | subchannel | 127.0.0.1:7233 TRANSIENT_FAILURE -> IDLE
2021-05-12T07:22:32.511Z | subchannel_refcount | 127.0.0.1:7233 refcount 2 -> 1
2021-05-12T07:22:32.511Z | pick_first | TRANSIENT_FAILURE -> IDLE
2021-05-12T07:22:32.511Z | resolving_load_balancer | dns:127.0.0.1:7233 TRANSIENT_FAILURE -> IDLE
2021-05-12T07:22:32.511Z | connectivity_state | dns:127.0.0.1:7233 TRANSIENT_FAILURE -> IDLE
2021-05-12T07:22:32.511Z | dns_resolver | Resolution update requested for target dns:127.0.0.1:7233
2021-05-12T07:22:32.511Z | dns_resolver | Returning IP address for target dns:127.0.0.1:7233
2021-05-12T07:22:32.511Z | resolving_load_balancer | dns:127.0.0.1:7233 IDLE -> CONNECTING
2021-05-12T07:22:32.511Z | connectivity_state | dns:127.0.0.1:7233 IDLE -> CONNECTING
2021-05-12T07:22:32.511Z | resolving_load_balancer | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:32.511Z | connectivity_state | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:32.512Z | pick_first | Connect to address list 127.0.0.1:7233
2021-05-12T07:22:32.512Z | subchannel_refcount | 127.0.0.1:7233 refcount 1 -> 2
2021-05-12T07:22:32.512Z | pick_first | Start connecting to subchannel with address 127.0.0.1:7233
2021-05-12T07:22:32.512Z | pick_first | IDLE -> CONNECTING
2021-05-12T07:22:32.512Z | resolving_load_balancer | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:32.512Z | connectivity_state | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:32.513Z | subchannel | 127.0.0.1:7233 IDLE -> CONNECTING
2021-05-12T07:22:32.513Z | pick_first | CONNECTING -> CONNECTING
2021-05-12T07:22:32.513Z | resolving_load_balancer | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:32.513Z | connectivity_state | dns:127.0.0.1:7233 CONNECTING -> CONNECTING
2021-05-12T07:22:32.621Z | subchannel | 127.0.0.1:7233 connection closed with error unable to get issuer certificate
2021-05-12T07:22:32.621Z | subchannel | 127.0.0.1:7233 CONNECTING -> TRANSIENT_FAILURE
2021-05-12T07:22:32.621Z | pick_first | CONNECTING -> TRANSIENT_FAILURE
2021-05-12T07:22:32.621Z | resolving_load_balancer | dns:127.0.0.1:7233 CONNECTING -> TRANSIENT_FAILURE
2021-05-12T07:22:32.621Z | connectivity_state | dns:127.0.0.1:7233 CONNECTING -> TRANSIENT_FAILURE

@murgatroid99
Copy link
Member

OK, the relevant error there is "unable to get issuer certificate". From what I can find, that means that there's some issue with the certificate chain or the CA certificates list. If you run your code with the environment variable NODE_DEBUG=tls, you might get some more useful details.

@bergundy
Copy link
Author

Yes, so that was obvious.
I didn't even have to use NODE_DEBUG=tls but thanks for the tip.
I managed to connect using the root certificate instead of the intermediate certificate.
I still find it strange that grpc and the golang client can connect using the intermediate certificate.
Is this expected behavior?

@murgatroid99
Copy link
Member

grpc-js just delegates the TLS functionality to Node's built in tls module. And the grpc package uses the C++-based gRPC core library, which lives in https://github.com/grpc/grpc, so that's probably the best place to learn about why it functions that way.

@bergundy
Copy link
Author

I'll look into it when I have more time, thanks for your help.

@liammurray liammurray mentioned this issue Dec 21, 2021
Closed
@Jeff-Tian Jeff-Tian mentioned this issue Jul 6, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
package: @grpc/grpc-js
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bergundy @murgatroid99