Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TLS errors unclear response by neofs-cli #2561

Closed
carpawell opened this issue Sep 8, 2023 · 1 comment
Closed

TLS errors unclear response by neofs-cli #2561

carpawell opened this issue Sep 8, 2023 · 1 comment
Assignees
@cthulhu-rider
Labels
bug Something isn't working I4 No visible changes neofs-cli NeoFS CLI application issues S4 Routine U3 Regular
Milestone
v0.41.0

Comments

@carpawell
Copy link
Member

carpawell commented Sep 8, 2023 •
edited
Loading

Expected Behavior

v0.32.0:

▶ ./neofs-cli-amd64 netmap nodeinfo --generate-key -r grpcs://st1.t5.fs.neo.org:8080                             
rpc error: write request: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

Immediate response, understandable error.

Current Behavior

v0.33.0+:

▶ ./neofs-cli-amd64 netmap nodeinfo --generate-key -r grpcs://st1.t5.fs.neo.org:8080
can't create API client: can't init SDK client: open gRPC connection: gRPC dial: context deadline exceeded

15 seconds timeout. Error... well, could be better.

Possible Solution

  • SDK is broken?
  • go is broken?
  • neofs-cli is broken?
  • gRPC lib is broken?
  • ???

Steps to Reproduce (for bugs)

Get the provided CLI versions from the GH release pages, use testnet endpoints, pretend to be an idiot and use a secure (grpcs "protocol") client for an insecure endpoint.

Context

Extremely annoying bug while you try to find out what is wrong with your certs: nspcc-dev/neofs-dev-env#287.

Regression

v0.33.0 CLI release.

Your Environment

Does not work locally and on GH workers (ubuntu if it means something).
@carpawell carpawell added bug Something isn't working neofs-cli NeoFS CLI application issues UX labels Sep 8, 2023
@roman-khimov roman-khimov added this to the v0.40.0 milestone Sep 8, 2023
@roman-khimov roman-khimov added U3 Regular S4 Routine I4 No visible changes and removed UX labels Dec 21, 2023
@roman-khimov roman-khimov modified the milestones: v0.40.0, v0.41.0 Feb 9, 2024
@cthulhu-rider
Copy link
Contributor

@carpawell were you able to verify/refute any of the hypotheses?

@cthulhu-rider cthulhu-rider self-assigned this Feb 27, 2024
cthulhu-rider added a commit to nspcc-dev/neofs-api-go that referenced this issue Feb 28, 2024
@cthulhu-rider
rpc/client: Add unit test for TLS handshake failure
a4b7d73 
Inspired by nspcc-dev/neofs-node#2561.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit to nspcc-dev/neofs-api-go that referenced this issue Feb 28, 2024
@cthulhu-rider
rpc/client: Dial with grpc.WithReturnConnectionError option
cc4cae6 
Since 2b89b7e, RPC client used
`grpc.WithBlock` option to dial the server. This option make dialer to
return either `nil` or `context.DeadlineExceeded` errors, with any
connection error resulting in the latter. In particular, TLS handshake
failures were shadowed by deadline error.

Now `WithReturnConnectionError` option is used instead:
 * it still blocks similar to `WithBlock`;
 * it adds connection failure to the deadline error.

As a result, TLS unit test passes now. This should fix the problem
originally posted in nspcc-dev/neofs-node#2561.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit to nspcc-dev/neofs-sdk-go that referenced this issue Feb 28, 2024
@cthulhu-rider
client: Isolate dial errors
b253e13 
Fix came from nspcc-dev/neofs-api-go#445 for the
problem described in nspcc-dev/neofs-node#2561.
Фlso now any irreparable errors like invalid address will be returned
immediately, not upon reaching the deadline.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit to nspcc-dev/neofs-sdk-go that referenced this issue Feb 28, 2024
@cthulhu-rider
client: Isolate dial errors
8b8b7e3 
Fix came from nspcc-dev/neofs-api-go#445 for the
problem described in nspcc-dev/neofs-node#2561.
Фlso now any irreparable errors (like invalid net address or TLS
handshake) will be returned immediately, not upon reaching the deadline.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Feb 28, 2024
@cthulhu-rider
*: Receive explicit reasons for NeoFS API dial failures
2b4cf2e 
This pulls NeoFS SDK version containing fix of the NeoFS API client dial
failures nspcc-dev/neofs-sdk-go#561. Previously,
any connection errors, including TLS, resulted in waiting for a
deadline and returning `context.DeadlineExceeded`. This did not make it
possible to distinguish them from timeouts and identify the root cause.

In addition, irreparable errors such as an incorrect network address or
connection refusal were also reduced to `context.DeadlineExceeded`, even
if they were received quickly. This behavior has also been fixed.

Fixes #2561.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit to nspcc-dev/neofs-api-go that referenced this issue Feb 28, 2024
@cthulhu-rider
rpc/client: Add unit test for TLS handshake failure
2d1f747 
Inspired by nspcc-dev/neofs-node#2561.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit to nspcc-dev/neofs-api-go that referenced this issue Feb 28, 2024
@cthulhu-rider
rpc/client: Dial with grpc.WithReturnConnectionError option
1d860f2 
Since 2b89b7e, RPC client used
`grpc.WithBlock` option to dial the server. This option make dialer to
return either `nil` or `context.DeadlineExceeded` errors, with any
connection error resulting in the latter. In particular, TLS handshake
failures were shadowed by deadline error.

Now `WithReturnConnectionError` option is used instead:
 * it still blocks similar to `WithBlock`;
 * it adds connection failure to the deadline error.

As a result, TLS unit test passes now. This should fix the problem
originally posted in nspcc-dev/neofs-node#2561.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit to nspcc-dev/neofs-api-go that referenced this issue Feb 28, 2024
@cthulhu-rider
rpc/client: Add unit test for TLS handshake failure
6cca5d1 
Inspired by nspcc-dev/neofs-node#2561.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit to nspcc-dev/neofs-api-go that referenced this issue Feb 28, 2024
@cthulhu-rider
rpc/client: Dial with grpc.WithReturnConnectionError option
05ac62e 
Since 2b89b7e, RPC client used
`grpc.WithBlock` option to dial the server. This option make dialer to
return either `nil` or `context.DeadlineExceeded` errors, with any
connection error resulting in the latter. In particular, TLS handshake
failures were shadowed by deadline error.

Now `WithReturnConnectionError` option is used instead:
 * it still blocks similar to `WithBlock`;
 * it adds connection failure to the deadline error.

As a result, TLS unit test passes now. This should fix the problem
originally posted in nspcc-dev/neofs-node#2561.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit to nspcc-dev/neofs-api-go that referenced this issue Feb 28, 2024
@cthulhu-rider
rpc/client: Add unit test for TLS handshake failure
a134213 
Inspired by nspcc-dev/neofs-node#2561.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit to nspcc-dev/neofs-api-go that referenced this issue Feb 28, 2024
@cthulhu-rider
rpc/client: Dial with grpc.WithReturnConnectionError option
c1aa37a 
Since 2b89b7e, RPC client used
`grpc.WithBlock` option to dial the server. This option make dialer to
return either `nil` or `context.DeadlineExceeded` errors, with any
connection error resulting in the latter. In particular, TLS handshake
failures were shadowed by deadline error.

Now `WithReturnConnectionError` option is used instead:
 * it still blocks similar to `WithBlock`;
 * it adds connection failure to the deadline error.

As a result, TLS unit test passes now. This should fix the problem
originally posted in nspcc-dev/neofs-node#2561.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit to nspcc-dev/neofs-sdk-go that referenced this issue Feb 28, 2024
@cthulhu-rider
client: Isolate dial errors
2ea075a 
Fix came from nspcc-dev/neofs-api-go#445 for the
problem described in nspcc-dev/neofs-node#2561.
Фlso now any irreparable errors (like invalid net address or TLS
handshake) will be returned immediately, not upon reaching the deadline.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit to nspcc-dev/neofs-sdk-go that referenced this issue Feb 28, 2024
@cthulhu-rider
client: Isolate dial errors
bb7a1eb 
Fix came from nspcc-dev/neofs-api-go#445 for the
problem described in nspcc-dev/neofs-node#2561.
Also now any irreparable errors (like invalid net address or TLS
handshake) will be returned immediately, not upon reaching the deadline.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Feb 29, 2024
@cthulhu-rider
*: Receive explicit reasons for NeoFS API dial failures
4c73b08 
This pulls NeoFS SDK version containing fix of the NeoFS API client dial
failures nspcc-dev/neofs-sdk-go#561. Previously,
any connection errors, including TLS, resulted in waiting for a
deadline and returning `context.DeadlineExceeded`. This did not make it
possible to distinguish them from timeouts and identify the root cause.

In addition, irreparable errors such as an incorrect network address or
connection refusal were also reduced to `context.DeadlineExceeded`, even
if they were received quickly. This behavior has also been fixed.

Fixes #2561.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
cthulhu-rider added a commit that referenced this issue Mar 4, 2024
@cthulhu-rider
*: Receive explicit reasons for NeoFS API dial failures
b52644e 
This pulls NeoFS API Go version containing fix of the NeoFS API client dial
failures nspcc-dev/neofs-sdk-go#561. Previously,
any connection errors, including TLS, resulted in waiting for a
deadline and returning `context.DeadlineExceeded`. This did not make it
possible to distinguish them from timeouts and identify the root cause.

In addition, irreparable errors such as an incorrect network address or
connection refusal were also reduced to `context.DeadlineExceeded`, even
if they were received quickly. This behavior has also been fixed.

Fixes #2561.

Signed-off-by: Leonard Lyubich <leonard@morphbits.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cthulhu-rider cthulhu-rider

Labels
bug Something isn't working I4 No visible changes neofs-cli NeoFS CLI application issues S4 Routine U3 Regular
Projects
None yet
Milestone
v0.41.0
Development

No branches or pull requests
3 participants
@roman-khimov @carpawell @cthulhu-rider