Use libuv for I/O #69

djs55 · 2016-06-30T11:26:33Z

The current Lwt_unix implementation has problems on both Mac and Windows, specifically:

on Mac, both the built-in default Lwt_engine and the libev one will use select. Unfortunately even with DARWIN_UNLIMITIED_SELECT we still see EINVAL errors (Lots of connections causes Unix.select to fail with EINVAL #66). Even if this worked it would not perform well with lots of connections.
on Windows, the select FD_SETSIZE is 64 and the OCaml Unix.select falls back to a much more complicated emulated code path. The API we should be using for scalable networking on Windows is IO Completion Ports

This PR changes the socket side of the stack to use libuv, the abstraction layer used by node.js which uses IOCP on Windows and kqueue on Mac.

This PR changes all the direct uses of Lwt_unix.file_descr into abstract types Socket.Stream.Tcp.t,Socket.Stream.Udp.tetc. This higher-level interface can then be mapped ontolibuvvia [fdopen/uwt] as well as traditionalLwt_unix.

A side-effect is that duplicated code was coalesced-- in particular there were several places where read and write functions had slightly different error handling.

Another side-effect is adding a Channel on top of the FLOW to avoid reading packet-by-packet. This has increased performance -- some benchmarks to follow.

Note that we must avoid using libraries which use Lwt_unix directly, as these calls will block. Happily most Mirage libraries are already pure in this sense.

The unit tests have been extended to run both the Lwt_unix and Uwt versions. I was able to run nmap tests in containers and observe a stable number of fds (~2000) during the test runs with lsof.

Fixes #66

Signed-off-by: David Scott <dave.scott@docker.com>

Instead of calling `FLOW.read` and `FLOW.write` directly which map onto `read(2)` and `write(2)`, layer a buffered Channel on top. This will allow us to reduce the number of times we make syscalls, for example we could read several packets in one go if the underlying `FLOW` has a big enough read buffer. Signed-off-by: David Scott <dave.scott@docker.com>

Now that we are using a buffered Channel, we can read more bytes at once from Unix file descriptors. This should reduce the rate at which we make expensive syscalls. With a simple iperf test sending from a container and receiving on the host, this boosts throughput from 1.01 Gbits/sec to 1.33 Gbits/sec on the Mac. Signed-off-by: David Scott <dave.scott@docker.com>

Both the Mac and Windows frontends use the same protocol to request a connection to a given container IP and port, so move this code to the shared library. Signed-off-by: David Scott <dave.scott@docker.com>

Signed-off-by: David Scott <dave@recoil.org>

Rather than use bare `Lwt_unix.send`, `Lwt_unix.recvfrom` this patch shares the existing code in `Socket.Datagram`. This will make a future patch to use libuv smaller. The main difference between the two `recvfrom` uses is that the previous DNS one would always timeout after 5s, whereas the general UDP NAT (activated for example if the VM sends DNS to 8.8.8.8) will be listening for 60s. So if a response is delayed > 5s we might receive it now. Presumably the same thing happens on the Internet anyway as there is no `Lwt.cancel` for transmitted UDP packets. Signed-off-by: David Scott <dave.scott@docker.com>

The generic Unix `SOCK_STREAM` as-a `FLOW` was duplicated: this patch merges them back together. Note the error handling in `Socket.Stream` was slightly better: it cached whethe the fd is closed and returned `Eof` if so. Signed-off-by: David Scott <dave.scott@docker.com>

Since libuv requires us to use a typed API where Unix domain sockets, TCP sockets etc are all treated differently, this patch creates such an API on top of the existing Lwt_unix implementation. Signed-off-by: David Scott <dave.scott@docker.com>

The current Lwt_unix implementation is the same, but we should be able to make a libuv compatible version too. Note this moves the function from vmnet-client to hostnet in order to hide it behind the abstract interface in hostnet. Signed-off-by: David Scott <dave.scott@docker.com>

Signed-off-by: David Scott <dave.scott@docker.com>

Since we can't use Lwt_unix directly, this patch hides it behind a simple `FILES.read_file` interface. Signed-off-by: David Scott <dave.scott@docker.com>

This removes the remaining hardcoded use of Lwt_unix.sleep. Signed-off-by: David Scott <dave.scott@docker.com>

This means we should be able to swap out the Lwt_unix implementation completely for another one. Signed-off-by: David Scott <dave.scott@docker.com>

This uses - Uwt.Timer - Uwt.Pipe (Unix domain sockets and Windows named pipes) - Uwt.Tcp - Uwt.Udp Signed-off-by: David Scott <dave.scott@docker.com>

When using `Lwt_unix` we should use `Lwt_main.run`, when using `Uwt` we should use `Uwt.Main.run` Signed-off-by: David Scott <dave.scott@docker.com>

We still default to a select-based event loop. If the --libuv argument is provided then we switch to libuv instead. Signed-off-by: David Scott <dave.scott@docker.com>

This is already the case for the Windows frontend. This makes testing easier. Signed-off-by: David Scott <dave.scott@docker.com>

We're not using the mirage-unix "platform" when in Uwt mode. Therefore we shouldn't use the OS.Time module. Signed-off-by: David Scott <dave.scott@docker.com>

Previously the module was hardcoded to use Lwt_unix via Flow_lwt_unix. This API change allows the client to choose the FLOW and TIME implementations. Signed-off-by: David Scott <dave.scott@docker.com>

Also known as 879b0fc68d2d44f5c6317f033ac50ffcc3d4277f The backport is here: djs55/mirage-tcpip@0da78c9 In particular this avoids `Console.log_s` which won't work if we're using `libuv` (or anything other than `Lwt_main.run` and `Lwt_unix`) Signed-off-by: David Scott <dave.scott@docker.com>

Note that Host_uwt already supports these via libuv. Signed-off-by: David Scott <dave.scott@docker.com>

Signed-off-by: David Scott <dave.scott@docker.com>

This is mostly straightforward, but it did require extending the `Host.Main` interface with `run_in_main` so the hyper-V sockets implementation can use the correct function. Signed-off-by: David Scott <dave.scott@docker.com>

We run the unit tests with both Lwt_unix and Uwt. Signed-off-by: David Scott <dave.scott@docker.com>

This is needed because libuv is not in cygwin. Signed-off-by: David Scott <dave.scott@docker.com>

Signed-off-by: David Scott <dave.scott@docker.com>

mirage-console.1.1.0 has a half-close fix, and uwt.999 has a modified dependency: rather than depending on conf-libuv, it depends on libuv which actually builds libuv. Signed-off-by: David Scott <dave.scott@docker.com>

Signed-off-by: David Scott <dave.scott@docker.com>

Although these are referenced in the hostnet library they aren't used and so probably aren't really included, but they are needed. Signed-off-by: David Scott <dave.scott@docker.com>

Since libuv seems to work, let's use it to work around the problems with select (notably: EINVAL with large fd sets) Signed-off-by: David Scott <dave.scott@docker.com>

Signed-off-by: David Scott <dave.scott@docker.com>

avsm · 2016-06-30T11:52:59Z

src/com.docker.slirp.exe/src/main.ml

@@ -21,149 +21,76 @@ let default d = function None -> d | Some x -> x
 let ethernet_serviceid = "30D48B34-7D27-4B0B-AAAF-BBBED334DD59"
 let ports_serviceid = "0B95756A-9985-48AD-9470-78E060895BE7"

-(* These could be shared with datakit. Perhaps they should live in mirage/conduit? *)
+let hvsock_addr_of_uri ~default_serviceid uri =


Yes this can indeed live in Conduit as a custom registration

Signed-off-by: David Scott <dave.scott@docker.com>

Signed-off-by: David Scott <dave@recoil.org>

We were already in an Lwt.t context, so Lwt.fail is more appropriate. Signed-off-by: David Scott <dave.scott@docker.com>

This reverts commit 6d36ff1. Signed-off-by: David Scott <dave.scott@docker.com>

This reverts commit f19ad8d. Signed-off-by: David Scott <dave.scott@docker.com>

Now that we depend on libuv, we must include it in our package. Signed-off-by: David Scott <dave.scott@docker.com>

Signed-off-by: David Scott <dave.scott@docker.com>

This is needed to respect the MACOS_DEPLOYMENT_TARGET Signed-off-by: David Scott <dave.scott@docker.com>

Signed-off-by: David Scott <dave.scott@docker.com>

djs55 · 2016-07-01T10:15:50Z

Regression tests are looking good -- let's try to include this in Docker for Mac/Windows soon.

yomimono · 2016-07-01T10:22:25Z

Belated LGTM :)

djs55 · 2016-07-01T10:26:00Z

Thanks :) (sorry for the rush)

avsm · 2016-07-01T10:34:33Z

lgtm

Metadata support (executable files and symlinks)

djs55 added 30 commits June 21, 2016 21:08

Remove dead code ("Socket_stack")

0e06d6c

Signed-off-by: David Scott <dave.scott@docker.com>

De-duplicate the connection forwarding header writing

6c044d0

Both the Mac and Windows frontends use the same protocol to request a connection to a given container IP and port, so move this code to the shared library. Signed-off-by: David Scott <dave.scott@docker.com>

proto-vmnet: Regenerate OASIS

cfea5c7

Signed-off-by: David Scott <dave@recoil.org>

proto-vmnet: Regenerate OASIS

c04c43f

Signed-off-by: David Scott <dave.scott@docker.com>

Add an abstract interface for file reading, use for /etc/resolv.conf

70bb00c

Since we can't use Lwt_unix directly, this patch hides it behind a simple `FILES.read_file` interface. Signed-off-by: David Scott <dave.scott@docker.com>

Functorise over V1_LWT.TIME

add258f

This removes the remaining hardcoded use of Lwt_unix.sleep. Signed-off-by: David Scott <dave.scott@docker.com>

Create a HOST signature and gather Sockets, Files and Time inside it

b1c0d44

This means we should be able to swap out the Lwt_unix implementation completely for another one. Signed-off-by: David Scott <dave.scott@docker.com>

Add a Uwt implementation of the HOST signature

b69aac2

This uses - Uwt.Timer - Uwt.Pipe (Unix domain sockets and Windows named pipes) - Uwt.Tcp - Uwt.Udp Signed-off-by: David Scott <dave.scott@docker.com>

Add a Main.run function to run the event loop

96835eb

When using `Lwt_unix` we should use `Lwt_main.run`, when using `Uwt` we should use `Uwt.Main.run` Signed-off-by: David Scott <dave.scott@docker.com>

com.docker.slirp: add argument --libuv

d40ebac

We still default to a select-based event loop. If the --libuv argument is provided then we switch to libuv instead. Signed-off-by: David Scott <dave.scott@docker.com>

com.docker.slirp: make the database optional

30902b0

This is already the case for the Windows frontend. This makes testing easier. Signed-off-by: David Scott <dave.scott@docker.com>

hostnet: use Host.Time rather than OS.Time

3e45460

We're not using the mirage-unix "platform" when in Uwt mode. Therefore we shouldn't use the OS.Time module. Signed-off-by: David Scott <dave.scott@docker.com>

ofs: functorise over V1_LWT.FLOW and V1_LWT.TIME

87cec66

Previously the module was hardcoded to use Lwt_unix via Flow_lwt_unix. This API change allows the client to choose the FLOW and TIME implementations. Signed-off-by: David Scott <dave.scott@docker.com>

Add support for Windows named-pipes to Host_lwt_unix

bfac30e

Note that Host_uwt already supports these via libuv. Signed-off-by: David Scott <dave.scott@docker.com>

ofs: add dependencies on astring and named-pipe

0550d7e

Signed-off-by: David Scott <dave.scott@docker.com>

hostnet: add dependency on uwt

138d642

Signed-off-by: David Scott <dave.scott@docker.com>

com.docker.slirp.exe: update to new interfaces

bb51c98

This is mostly straightforward, but it did require extending the `Host.Main` interface with `run_in_main` so the hyper-V sockets implementation can use the correct function. Signed-off-by: David Scott <dave.scott@docker.com>

test: functorise unit tests, run them twice

85c9b26

We run the unit tests with both Lwt_unix and Uwt. Signed-off-by: David Scott <dave.scott@docker.com>

opam: add a libuv package to build from source

ef4b137

This is needed because libuv is not in cygwin. Signed-off-by: David Scott <dave.scott@docker.com>

hostnet: test cases must link *.preemptive

3fdcadb

Signed-off-by: David Scott <dave.scott@docker.com>

opam: add win32/dev/uwt.999 and */dev/mirage-console.1.1.0

3cd8806

mirage-console.1.1.0 has a half-close fix, and uwt.999 has a modified dependency: rather than depending on conf-libuv, it depends on libuv which actually builds libuv. Signed-off-by: David Scott <dave.scott@docker.com>

libuv depends on conf-pkg-config

bd96bd1

Signed-off-by: David Scott <dave.scott@docker.com>

com.docker.slirp.exe: link against {lwt,uwt}.preemptive

eb95203

Although these are referenced in the hostnet library they aren't used and so probably aren't really included, but they are needed. Signed-off-by: David Scott <dave.scott@docker.com>

djs55 added 2 commits June 29, 2016 15:37

Use libuv by default, make --select an option

8fb1cf8

Since libuv seems to work, let's use it to work around the problems with select (notably: EINVAL with large fd sets) Signed-off-by: David Scott <dave.scott@docker.com>

win32: add hvsock.0.8

646dc1e

Signed-off-by: David Scott <dave.scott@docker.com>

GordonTheTurtle added status/0-triage group/windows labels Jun 30, 2016

avsm reviewed Jun 30, 2016
View reviewed changes

djs55 added 3 commits June 30, 2016 13:13

opam: fix typo (SWITCH rather than SWITH)

b4daec6

Signed-off-by: David Scott <dave.scott@docker.com>

AUTO: Update upstream packages

b3adcb4

Signed-off-by: David Scott <dave@recoil.org>

com.docker.slirp: use an Lwt.fail rather than raw failwith

1338356

We were already in an Lwt.t context, so Lwt.fail is more appropriate. Signed-off-by: David Scott <dave.scott@docker.com>

GordonTheTurtle added the dco/no label Jun 30, 2016

djs55 added 9 commits June 30, 2016 20:42

Revert "Windows: split OCaml patch back into 2"

4263b0b

This reverts commit 6d36ff1. Signed-off-by: David Scott <dave.scott@docker.com>

Revert "Add an OCaml patch to fix Windows select"

ecee761

This reverts commit f19ad8d. Signed-off-by: David Scott <dave.scott@docker.com>

com.docker.slirp: create a .tgz with the shared libraries

cafcf10

Now that we depend on libuv, we must include it in our package. Signed-off-by: David Scott <dave.scott@docker.com>

libuv: run ranlib on libuv.a

861fc76

Signed-off-by: David Scott <dave.scott@docker.com>

opam: build libuv from source on Mac

5c41d31

This is needed to respect the MACOS_DEPLOYMENT_TARGET Signed-off-by: David Scott <dave.scott@docker.com>

AUTO: Update upstream packages

25d926a

Signed-off-by: David Scott <dave.scott@docker.com>

libuv: make sure we're using the correct ar.exe

3b3e007

Signed-off-by: David Scott <dave.scott@docker.com>

libuv: set the same CFLAGS as ocaml

9752951

Signed-off-by: David Scott <dave.scott@docker.com>

libuv: use the full path for ranlib.exe

715d42a

Signed-off-by: David Scott <dave.scott@docker.com>

djs55 force-pushed the use-buffered-channels branch from c72dd87 to 715d42a Compare June 30, 2016 19:43

GordonTheTurtle removed the dco/no label Jun 30, 2016

djs55 merged commit 4cf14ab into moby:master Jul 1, 2016

djs55 deleted the use-buffered-channels branch July 1, 2016 10:15

avsm mentioned this pull request Jul 7, 2016

Add cplugins and add a configure option -fPIC ocaml/ocaml#668

Merged

mfp mentioned this pull request Mar 26, 2017

Proposal: (softly) replace Lwt_unix by libuv/uwt ocsigen/lwt#328

Open

avsm pushed a commit to avsm/vpnkit that referenced this pull request May 5, 2017

Merge pull request moby#69 from talex5/metadata

9b87c4e

Metadata support (executable files and symlinks)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use libuv for I/O #69

Use libuv for I/O #69

djs55 commented Jun 30, 2016

avsm Jun 30, 2016

djs55 commented Jul 1, 2016

yomimono commented Jul 1, 2016

djs55 commented Jul 1, 2016

avsm commented Jul 1, 2016

Use libuv for I/O #69

Use libuv for I/O #69

Conversation

djs55 commented Jun 30, 2016

avsm Jun 30, 2016

Choose a reason for hiding this comment

djs55 commented Jul 1, 2016

yomimono commented Jul 1, 2016

djs55 commented Jul 1, 2016

avsm commented Jul 1, 2016