Support listening on UNIX domain sockets. #2701

jmillikin-stripe · 2018-03-02T19:59:37Z

Description:
Since the Address protobuf type is already used for listener addresses, there is no API change. Both file-like and Linux's abstract socket addresses are supported.

Risk Level: Medium

Testing:
Unit test for the proto->address conversion, integration tests for listeners and hot restart, and I've done some local testing on a MacOS machine to verify basic functionality.

Docs Changes:
N/A

Release Notes:
Added support for listening on UNIX domain sockets.

Fixes #1605
Fixes #2697

Signed-off-by: John Millikin jmillikin@stripe.com

Since the `Address` protobuf type is already used for listener addresses, there is no API change. Both file-like and Linux's abstract socket addresses are supported. There's a comment in `ProdListenerComponentFactory` that implies UNIX sockets aren't supported by hot restart. I didn't try to get that working, since I'm planning to use this feature in a container where Envoy never hot restarts. Signed-off-by: John Millikin <jmillikin@stripe.com>

jmillikin-stripe · 2018-03-02T20:01:58Z

I'm having some trouble writing an integration test for this, since the existing test class hierarchy (HttpIntegrationTest, etc) all assume a TCP listener. Will keep working on it unless reviewers feel the existing tests are sufficient.

mattklein123 · 2018-03-02T20:45:37Z

source/server/listener_manager_impl.cc

+  ASSERT(address->type() == Network::Address::Type::Ip ||
+         address->type() == Network::Address::Type::Pipe);
+  if (address->type() == Network::Address::Type::Pipe) {
+    return std::make_shared<Network::UdsListenSocket>(address->asString());


Quick non-full review comment: This is not going to support hot restart. I think it should be really easy to support hot restart by folding this logic into the hot restart duplication logic below. We already pass a string for the address name to the parent process, where we can then look up the socket by name. I'm pretty sure that it will mostly just work if you add code below to handle creating a "unix://" socket name and passing it.

Also, we should definitely have an integration test for this, and optimally add a unix socket to the hot restart test also. @ggreenway or @alyssawilk can likely guide you.

OK, updated this code to use the same hot-restart recovery logic for UDS as is used for TCP.

ggreenway

The code looks good.

I'd see if you can override the connect code in the test tcp client of the integration tests; I don't think you need to do any UDS testing on the upstream side.

Also, please add some coverage in the hot-restart integration test.

ggreenway · 2018-03-02T23:57:46Z

source/common/network/listen_socket_impl.cc

-  fd_ = local_address_->socket(Address::SocketType::Stream);
+UdsListenSocket::UdsListenSocket(const Address::InstanceConstSharedPtr& address)
+    : ListenSocketImpl(-1, address) {
+  fd_ = address->socket(Address::SocketType::Stream);


Can this be directly passed to ListenSocketImpl on the previous line? Then the code would mirror the TcpListenSocket constructor a few lines up.

There's an interaction here with the filesystem (see call to remove() in the old code) that I haven't fully understood yet. If I can verify in the tests that it's safe to remove within the address impl, I'll clean up this line.

Sounds good. If it can't be cleaned up, please add a comment explaining why it is that way.

Signed-off-by: John Millikin <jmillikin@stripe.com>

jmillikin-stripe · 2018-03-06T04:26:12Z

Integration tests added for UNIX listeners and hot restart.

Signed-off-by: John Millikin <jmillikin@stripe.com>

ggreenway · 2018-03-06T16:49:12Z

test/integration/uds_integration_test.h

+
+  std::string getAdminSocketName() {
+    return abstract_namespace_
+               ? "@/envoy-tests/uds-integration-test/admin"


I think we need some variation in the name to avoid collisions between concurrent test runs.

Signed-off-by: John Millikin <jmillikin@stripe.com>

ggreenway · 2018-03-06T18:12:48Z

test/integration/uds_integration_test.h

    return abstract_namespace_
-               ? "@/envoy-tests/uds-integration-test/admin"
-               : TestEnvironment::unixDomainSocketPath("uds-integration-test.admin.sock");
+               ? fmt::format("@/envoy-tests/{}/{}/admin", testInfo->test_case_name(),


I'm worried about the case where multiple top-level builds are running concurrently on a box, like coverage and ASAN. These abstract-namespace UDS paths are global to the box, which is great for their intended use case, and difficult for making these tests work. @htuch is there any precedent anywhere else in the tests for a collision type like this? Should we put a random number in the path? Or maybe call mktmp and use the pathname it returns?

Sorry to be so picky about this; just trying to avoid flaky tests.

Note that the existing UDS integration test is using abstract sockets for its admin address.

Yes, this is an issue, see what was done recently in

envoy/test/exe/main_common_test.cc

Line 47 in b6b1564

/**

. Can we add a unixDomainSocketAbstract() util that does the same as TestEnvironment::unixDomainSocketPath for abstract sockets? We could have a per-instance global for the entropy.

OK, I changed this to take the output of TestEnvironment::unixDomainSocketPath and prepend "@" for abstract sockets. This should be sufficiently random, since the file paths are already sharing a system-wide temp dir.

htuch · 2018-03-06T19:42:31Z

source/common/network/address_impl.cc

@@ -280,6 +280,7 @@ int PipeInstance::bind(int fd) const {
    return ::bind(fd, reinterpret_cast<const sockaddr*>(&address_),
                  offsetof(struct sockaddr_un, sun_path) + address_length_);
  }
+  unlink(address_.sun_path);


Should we RELEASE_ASSERT that this is successful? Is this behavior tested?

Added a test. If this fails, the bind will return an appropriate errno depending on whether the existing file is a pre-existing socket, or some other file type.

Maybe just add a comment here. I generally find that when reading code, if an error is deliberately ignored it's helpful to know this vs. suspecting some missing error handling.

htuch · 2018-03-06T19:45:31Z

source/common/network/utility.h

@@ -201,6 +201,9 @@ class Utility {
   */
  static absl::uint128 Ip6htonl(const absl::uint128& address);

+  static Address::InstanceConstSharedPtr
+  parseProtobufAddress(const envoy::api::v2::core::Address& proto_address);


Maybe rename to protobufAddressToAddress for symmetry?

This lets `--max-obj-name-len 500` be used in validation mode, and fixes assertion errors from the name length being calculated during the construction of `HotRestartImpl`. Signed-off-by: John Millikin <jmillikin@stripe.com>

Signed-off-by: John Millikin <jmillikin@stripe.com>

ggreenway

I think this looks good.

mattklein123

Awesome work. This is great. Just 2 minor comments if you feel like it.

mattklein123 · 2018-03-06T23:20:23Z

source/common/network/address_impl.cc

@@ -261,6 +261,11 @@ PipeInstance::PipeInstance(const sockaddr_un* address, socklen_t ss_len)
 }

 PipeInstance::PipeInstance(const std::string& pipe_path) : InstanceBase(Type::Pipe) {
+  if (pipe_path.size() >= sizeof(address_.sun_path)) {
+    throw EnvoyException(


Can we get an EXPECT_THROW_WITH_MESSAGE test for this?

Done. I used WITH_REGEX to avoid making too many assumptions about sizeof(sun_path).

mattklein123 · 2018-03-06T23:20:41Z

source/common/network/utility.cc

+  case envoy::api::v2::core::Address::kPipe:
+    return std::make_shared<Address::PipeInstance>(proto_address.pipe().path());
+  default:
+    throw EnvoyException("Address must be a socket or pipe: " + proto_address.DebugString());


I think I would do NOT_REACHED here. This should never happen w/ schema checks.

Signed-off-by: John Millikin <jmillikin@stripe.com>

htuch

LGTM modulo comments.

htuch · 2018-03-07T15:45:09Z

source/common/network/address_impl.cc

@@ -280,6 +280,7 @@ int PipeInstance::bind(int fd) const {
    return ::bind(fd, reinterpret_cast<const sockaddr*>(&address_),
                  offsetof(struct sockaddr_un, sun_path) + address_length_);
  }
+  unlink(address_.sun_path);


Maybe just add a comment here. I generally find that when reading code, if an error is deliberately ignored it's helpful to know this vs. suspecting some missing error handling.

htuch · 2018-03-07T15:47:12Z

test/config/integration/server_unix_listener.json

@@ -0,0 +1,38 @@
+{


Can you make this a v2 YAML? We're trying to avoid adding any new v1 configs to integration tests. Thanks.

I started out with v2 yaml, then went to v1 json because the test helpers aren't compatible with the v2 configs at all.

htuch · 2018-03-07T15:48:04Z

test/integration/hotrestart_test.sh

+  sed -e "s#{{ socket_dir }}#${SOCKET_DIR}#" | \
+  sed -e "s#{{ ip_loopback_address }}#127.0.0.1#" | \
+  cat > "${HOT_RESTART_JSON_UDS}"
+JSON_TEST_ARRAY+=("${HOT_RESTART_JSON_UDS}")


Oh, I see, hotrestart_test is also v1 JSON. This makes me sad, but no need to fix in this PR.

htuch · 2018-03-07T15:51:32Z

test/integration/uds_integration_test.h

+    if (!abstract_namespace_) {
+      return path;
+    }
+    return "@" + path;


This and the above method could be slightly refactored to reduce common code.

Signed-off-by: John Millikin <jmillikin@stripe.com>

htuch

Thanks!

…ogle import. envoyproxy#2701 hardcoded IPv4 for the integration test. When parameterizing this, envoyproxy#2829 was encountered, so the corresponding test was disabled. Signed-off-by: Harvey Tuch <htuch@google.com>

…ogle import. (#2830) #2701 hardcoded IPv4 for the integration test. When parameterizing this, #2829 was encountered, so the corresponding test was disabled. Signed-off-by: Harvey Tuch <htuch@google.com>

* fix duration unit conversion * format * wasm gen * add test

Rename release_validator to validation_cleanup. The release_validator function does not actually release the validator. On iOS is does nothing and on Android it merely detaches the JVM from the current thread. So "cleanup validation" is a better description of it's function than "release validator" and matches the existing description "to clean up after validation completion." Signed-off-by: Ryan Hamilton rch@google.com Signed-off-by: JP Simard <jp@jpsim.com>

jmillikin-stripe force-pushed the listen-on-unix branch from afa5246 to 0eba9e9 Compare March 2, 2018 20:00

mattklein123 reviewed Mar 2, 2018

View reviewed changes

mattklein123 mentioned this pull request Mar 2, 2018

listeners should unlink() unix socket before binding to it #2697

Closed

ggreenway self-assigned this Mar 2, 2018

ggreenway requested changes Mar 3, 2018

View reviewed changes

jmillikin-stripe added 3 commits March 5, 2018 19:36

Allow UdsListenSocket to be constructed from a hot-restarted FD.

fee1f0f

Signed-off-by: John Millikin <jmillikin@stripe.com>

Support sending HTTP requests to UNIX sockets in integration tests.

26448e7

Signed-off-by: John Millikin <jmillikin@stripe.com>

Add an integration test for UNIX sockets.

3144b52

Signed-off-by: John Millikin <jmillikin@stripe.com>

jmillikin-stripe force-pushed the listen-on-unix branch from c5c1eb8 to 3144b52 Compare March 6, 2018 03:52

Add UNIX listener coverage to the hot restart test.

51d1a89

Signed-off-by: John Millikin <jmillikin@stripe.com>

fix up comment in test/integration/utility.h

647e76e

Signed-off-by: John Millikin <jmillikin@stripe.com>

htuch assigned ggreenway and htuch and unassigned ggreenway Mar 6, 2018

ggreenway requested changes Mar 6, 2018

View reviewed changes

jmillikin-stripe added 2 commits March 6, 2018 09:17

generate per-test socket paths

4be00d2

Signed-off-by: John Millikin <jmillikin@stripe.com>

add unix listener config to the integration test data files

3faa599

Signed-off-by: John Millikin <jmillikin@stripe.com>

jmillikin-stripe force-pushed the listen-on-unix branch from 5f5bede to 3faa599 Compare March 6, 2018 17:42

ggreenway reviewed Mar 6, 2018

View reviewed changes

htuch reviewed Mar 6, 2018

View reviewed changes

jmillikin-stripe added 7 commits March 6, 2018 12:38

use v1 config for hot restart integration test

39d8f91

Signed-off-by: John Millikin <jmillikin@stripe.com>

Print a nicer error if a UNIX domain socket path is too long.

68f9a99

Signed-off-by: John Millikin <jmillikin@stripe.com>

In hotrestart test, save sockets to /tmp to avoid Bazel's long paths.

c5f80da

Signed-off-by: John Millikin <jmillikin@stripe.com>

Rename parseProtoAddress -> protobufAddressToAddress

e7eb6dc

Signed-off-by: John Millikin <jmillikin@stripe.com>

Use more random abstract unix socket paths in UDS integration test.

9615cfd

Signed-off-by: John Millikin <jmillikin@stripe.com>

Add a unit test for unlink() during UDS bind.

6cdf4af

Signed-off-by: John Millikin <jmillikin@stripe.com>

ggreenway previously approved these changes Mar 6, 2018

View reviewed changes

mattklein123 reviewed Mar 6, 2018

View reviewed changes

Use NOT_REACHED and EXPECT_THROW_WITH_REGEX.

87e178c

Signed-off-by: John Millikin <jmillikin@stripe.com>

jmillikin-stripe dismissed ggreenway’s stale review via 87e178c March 6, 2018 23:37

htuch suggested changes Mar 7, 2018

View reviewed changes

add unlink comment; reuse admin socket name calculation.

bbdc4cf

Signed-off-by: John Millikin <jmillikin@stripe.com>

htuch approved these changes Mar 7, 2018

View reviewed changes

htuch merged commit 1db0c7a into envoyproxy:master Mar 7, 2018

jmillikin-stripe deleted the listen-on-unix branch March 7, 2018 23:14

ggreenway mentioned this pull request Mar 12, 2018

stats: Configure default max_obj_name_length at compile time #2768

Closed

htuch mentioned this pull request Mar 15, 2018

test: fixes for uds_integration_test for issues encountered during Go… #2830

Merged

Shikugawa pushed a commit to Shikugawa/envoy that referenced this pull request Mar 28, 2020

Fix duration unit conversion (envoyproxy#2701)

c38b04c

* fix duration unit conversion * format * wasm gen * add test

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support listening on UNIX domain sockets. #2701

Support listening on UNIX domain sockets. #2701

jmillikin-stripe commented Mar 2, 2018 •

edited

Loading

jmillikin-stripe commented Mar 2, 2018

mattklein123 Mar 2, 2018

jmillikin-stripe Mar 2, 2018

ggreenway left a comment

ggreenway Mar 2, 2018

jmillikin-stripe Mar 3, 2018

ggreenway Mar 3, 2018

jmillikin-stripe commented Mar 6, 2018

ggreenway Mar 6, 2018

jmillikin-stripe Mar 6, 2018

ggreenway Mar 6, 2018 •

edited

Loading

jmillikin-stripe Mar 6, 2018

htuch Mar 6, 2018

jmillikin-stripe Mar 6, 2018

htuch Mar 6, 2018

jmillikin-stripe Mar 6, 2018

htuch Mar 7, 2018

jmillikin-stripe Mar 7, 2018

htuch Mar 6, 2018

jmillikin-stripe Mar 6, 2018

ggreenway left a comment

mattklein123 left a comment

mattklein123 Mar 6, 2018

jmillikin-stripe Mar 6, 2018

mattklein123 Mar 6, 2018

jmillikin-stripe Mar 6, 2018

htuch left a comment

htuch Mar 7, 2018

htuch Mar 7, 2018

jmillikin-stripe Mar 7, 2018

htuch Mar 7, 2018

htuch Mar 7, 2018

jmillikin-stripe Mar 7, 2018

htuch left a comment

Support listening on UNIX domain sockets. #2701

Support listening on UNIX domain sockets. #2701

Conversation

jmillikin-stripe commented Mar 2, 2018 • edited Loading

jmillikin-stripe commented Mar 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggreenway left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmillikin-stripe commented Mar 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggreenway Mar 6, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ggreenway left a comment

Choose a reason for hiding this comment

mattklein123 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

htuch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

htuch left a comment

Choose a reason for hiding this comment

jmillikin-stripe commented Mar 2, 2018 •

edited

Loading

ggreenway Mar 6, 2018 •

edited

Loading