Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use available port for gRPC server tests #895

Merged
merged 4 commits into from
Jan 6, 2020

Conversation

marcotc
Copy link
Member

@marcotc marcotc commented Dec 20, 2019

gRPC tests sometimes fail with this error:

E1219 23:49:07.801805162    2827 server_chttp2.cc:40]        {
  "description": "No address added out of total 1 resolved",
  "file": "src/core/ext/transport/chttp2/server/chttp2_server.cc:394",
  "referenced_errors": [
    {
      "description": "Address already in use",
      "errno": 98,
      "file": "src/core/lib/iomgr/tcp_server_utils_posix_common.cc:181",
      "os_error": "Address already in use",
      "syscall": "bind"
    }
  ]
}

Failures:

  1) gRPC integration test multiple client configurations uses the correct configuration information
     Failure/Error: server.add_http2_port(address, :this_port_is_insecure)
     
     RuntimeError:
       could not add port 0.0.0.0:50052 to server, not sure why
     # ./spec/ddtrace/contrib/grpc/support/grpc_helper.rb:28:in `runner'
     # ./spec/ddtrace/contrib/grpc/support/grpc_helper.rb:5:in `run_request_reply'
     # ./spec/ddtrace/contrib/grpc/integration_spec.rb:35:in `block (3 levels) in <top (required)>'

This happens when we try to bind a gRPC server to new local endpoint that is already in use.

Given we hard-coded the ports, it is possible that another process will randomly choose to bind to these high ports by chance.

The best way to bind to an available local port is to bind to port 0, letting the OS find a port for us, but GRPC::RpcServer does not support that, raising an error instead when provided with 0.

So we do the next best thing which is to find an available port ourselves and quickly hand it to gRPC. There's a very small chance that another process will bind to the port we just found available between the time we retrieve its number and hand it to gRPC, but there's no perfect solution with the APIs we are provided by the gem.

@marcotc marcotc added the dev/testing Involves testing processes (e.g. RSpec) label Dec 20, 2019
@marcotc marcotc requested a review from a team December 20, 2019 00:46
@marcotc marcotc self-assigned this Dec 20, 2019
Copy link
Contributor

@delner delner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems pretty straightforward! I think if we can add a comment this is good to merge.

spec/support/network_helpers.rb Show resolved Hide resolved
@marcotc marcotc force-pushed the test/fix-flaky-grpc-server-test branch from bbc6ff4 to 3821095 Compare January 6, 2020 19:49
@marcotc marcotc requested a review from delner January 6, 2020 19:50
Copy link
Contributor

@delner delner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@marcotc marcotc merged commit aee17c9 into master Jan 6, 2020
@marcotc marcotc deleted the test/fix-flaky-grpc-server-test branch January 6, 2020 21:40
@delner delner added this to the 0.31.0 milestone Jan 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dev/testing Involves testing processes (e.g. RSpec)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants