Add some weekly VPNKit development reports

README.md

@@ -13,6 +13,7 @@ Binary artefacts are built by CI:
 VPNKit is a set of tools and services for helping [HyperKit](https://github.com/moby/hyperkit)
 VMs interoperate with host VPN configurations.
 
+
 Building on Unix
 ----------------
 
@@ -63,3 +64,13 @@ Design
 
 - [Using vpnkit as a default gateway](docs/ethernet.md): describes the flow of ethernet traffic to/from the VM
 - [Port forwarding](docs/ports.md): describes how ports are forwarded from the host into the VM
+
+Licensing
+---------
+
+VPNKit is licensed under the Apache License, Version 2.0. See
+[LICENSE](https://github.com/moby/vpnkit/blob/master/LICENSE.md) for the full
+license text.
+
+Contributions are welcome under the terms of this license. You may wish to browse
+the [weekly reports](reports) to read about overall activity in the repository.

reports/2017-04-23.md

@@ -0,0 +1,41 @@
+# Weekly VPNKit dev report for 2017-04-17 to 2017-04-23 (week 16)
+
+This report covers weekly developments in the [moby/vpnkit], [mirage/ocaml-dns], [mirage/ocaml-hvsock], [djs55/ocaml-osx-dnssd] and [mirage/mirage-tcpip] repositories.
+
+**TL;DR:** There were more DNS fixes around the handling of `localhost`, to address bugs
+reported downstream by Docker for Mac users. This involved disabling the use of multicast DNS on macOS.
+
+## PRs merged
+
+- Do not attempt to look up `localhost.local` via mDNS ([moby/vpnkit#211]), and then entirely disable mDNS on macOS ([moby/vpnkit#213] [@djs55]).
+- Rename the project from the `docker/` org to the `moby/` organisation ([moby/vpnkit#212] [@djs55]).
+
+### PRs with activity
+
+The [Lwt 3.0.0](https://github.com/ocsigen/lwt/releases/tag/3.0.0) release features several deprecated functions
+and a rearrangement of the packages to be more consistent. Several of the VPNKit packages are being updated to
+use the recommended interfaces and remain backwards compatible with older Lwt releases
+([mirage/ocaml-dns#136] [mirage/mirage-tcpip#308] [@aantron] [@djs55] [@yomimono]).
+
+The effort to add HTTP proxy support has stalled until upstream packages such as Cohttp integrated some
+improvements ([moby/vpnkit#20] [@dave-tucker] [@avsm]).
+Help welcome in this space; contact [@djs55] if interested!
+
+Other reports in this series can be browsed directly in the repository at [moby/vpnkit:/reports](https://github.com/moby/vpnkit/tree/master/reports/).
+
+[@aantron]: https://github.com/aantron
+[@dave-tucker]: https://github.com/dave-tucker
+[@djs55]: https://github.com/djs55
+[@yomimono]: https://github.com/yomimono
+[djs55/ocaml-osx-dnssd]: https://github.com/djs55/ocaml-osx-dnssd
+[mirage/mirage-tcpip]: https://github.com/mirage/mirage-tcpip
+[mirage/mirage-tcpip#308]: https://github.com/mirage/mirage-tcpip/pull/308
+[mirage/ocaml-dns]: https://github.com/mirage/ocaml-dns
+[mirage/ocaml-dns#136]: https://github.com/mirage/ocaml-dns/pull/136
+[mirage/ocaml-hvsock]: https://github.com/mirage/ocaml-hvsock
+[moby/vpnkit]: https://github.com/moby/vpnkit
+[moby/vpnkit#20]: https://github.com/moby/vpnkit/issues/20
+[moby/vpnkit#211]: https://github.com/moby/vpnkit/pull/211
+[moby/vpnkit#212]: https://github.com/moby/vpnkit/pull/212
+[moby/vpnkit#213]: https://github.com/moby/vpnkit/pull/213
+

reports/2017-04-30.md

@@ -0,0 +1,63 @@
+# Weekly VPNKit dev report for 2017-04-24 to 2017-04-30 (week 17)
+
+This report covers weekly developments in the [moby/vpnkit], [mirage/ocaml-dns],
+[mirage/ocaml-hvsock], [djs55/ocaml-osx-dnssd] and [mirage/mirage-tcpip] repositories.
+
+**TL;DR:** This week saw important bugfixes in preserving the order of DNS resource records and
+TCP connection tracking, which have been causing network issues for users of VPNKit
+via in Docker for Mac and Windows.
+
+The project now has a new home in the [Moby Project](http://mobyproject.org) and the main
+VPNKit repository was renamed to live in the new [moby/vpnkit] organisation. 
+Several library releases were cut this week that included these fixes, in [mirage/ocaml-hvsock:v0.14.0],
+[djs55/ocaml-osx-dnssd:v0.4] (and [djs55/ocaml-osx-dnssd:v0.3] as an intermediate release).
+
+## PRs merged
+
+- The port forwarding path has been made much more robust to connection floods.
+  Now connections are closed synchronously to ensure prompt resource cleanup and logging.
+  The number of maximum connections is also discovered dynamically rather than being hardcoded ([moby/vpnkit#214] [@djs55]).
+- Avoid reversing and duplicating DNS RRs when quering the local resolver on macOS ([moby/vpnkit#216] [djs55/ocaml-osx-dnssd#6] [@djs55])
+
+## PRs with activity
+
+- [@hannesm] did a comprehensive review of the state of DNS name handling as he is writing a recursive name resolver, and there has been an active discussion about how to improve the DNS library in future releases ([mirage/ocaml-dns#137] [@MagnusS] [@yomimono] [@stedolan] [@hannesm] [@dsheets] [@djs55] [@avsm]).
+- The MirageOS TCP/IP stack started sending packets with an oversized MTU on some backends such as virtio, so there has been discussion about how to fix it ([mirage/mirage-tcpip#309] [@hannesm] [@talex5] [@djs55] [@avsm] [@yomimono]).
+- To help debugging, we are also restoring Git history to the Mirage TCP/IP stack, but this requires a rebase ([mirage/mirage-tcpip#270] [@yomimono] [@balrajsingh] [@avsm] [@samoht]).
+- The Windows HyperV socket bindings are being prepared for a new release ([mirage/ocaml-hvsock#52]). 
+
+## External Links or Blogs
+
+- [Introducing the Moby Project](https://blog.docker.com/2017/04/introducing-the-moby-project/) to explain VPNKit's new home.
+
+Other reports in this series can be browsed directly in the repository at [moby/vpnkit:/reports](https://github.com/moby/vpnkit/tree/master/reports/).
+
+[@MagnusS]: https://github.com/MagnusS
+[@avsm]: https://github.com/avsm
+[@balrajsingh]: https://github.com/balrajsingh
+[@djs55]: https://github.com/djs55
+[@dsheets]: https://github.com/dsheets
+[@hannesm]: https://github.com/hannesm
+[@samoht]: https://github.com/samoht
+[@stedolan]: https://github.com/stedolan
+[@talex5]: https://github.com/talex5
+[@yomimono]: https://github.com/yomimono
+[djs55/ocaml-osx-dnssd]: https://github.com/djs55/ocaml-osx-dnssd
+[djs55/ocaml-osx-dnssd#6]: https://github.com/djs55/ocaml-osx-dnssd/pull/6
+[djs55/ocaml-osx-dnssd#7]: https://github.com/djs55/ocaml-osx-dnssd/pull/7
+[djs55/ocaml-osx-dnssd:v0.3]: https://github.com/djs55/ocaml-osx-dnssd/releases/tag/v0.3
+[djs55/ocaml-osx-dnssd:v0.4]: https://github.com/djs55/ocaml-osx-dnssd/releases/tag/v0.4
+[mirage/mirage-tcpip]: https://github.com/mirage/mirage-tcpip
+[mirage/mirage-tcpip#270]: https://github.com/mirage/mirage-tcpip/issues/270
+[mirage/mirage-tcpip#309]: https://github.com/mirage/mirage-tcpip/pull/309
+[mirage/mirage-tcpip#310]: https://github.com/mirage/mirage-tcpip/pull/310
+[mirage/ocaml-dns]: https://github.com/mirage/ocaml-dns
+[mirage/ocaml-dns#137]: https://github.com/mirage/ocaml-dns/issues/137
+[mirage/ocaml-hvsock]: https://github.com/mirage/ocaml-hvsock
+[mirage/ocaml-hvsock#52]: https://github.com/mirage/ocaml-hvsock/pull/52
+[mirage/ocaml-hvsock:v0.14.0]: https://github.com/mirage/ocaml-hvsock/releases/tag/v0.14.0
+[moby/vpnkit]: https://github.com/moby/vpnkit
+[moby/vpnkit#214]: https://github.com/moby/vpnkit/pull/214
+[moby/vpnkit#215]: https://github.com/moby/vpnkit/pull/215
+[moby/vpnkit#216]: https://github.com/moby/vpnkit/pull/216
+

reports/2017-05-07.md

@@ -0,0 +1,27 @@
+# Weekly dev report for 2017-05-01 to 2017-05-07 (week 18)
+
+This report covers weekly developments in the [moby/vpnkit], [mirage/ocaml-dns], [mirage/ocaml-hvsock],
+[djs55/ocaml-osx-dnssd] and [mirage/mirage-tcpip] repositories.
+
+**TL;DR:** A quiet week for VPNKit as work has focussed on support improvements via logging and an update to the documentation
+and architecture in the [README](README.md).
+
+## PRs merged
+
+- A minor improvement to the OCaml bindings to `SOMAXCONN` ([moby/vpnkit#217] [@yallop] [@djs55]).
+- Update the documentation to reflect the latest architecture ([moby/vpnkit#218] [@djs55])
+
+Other reports in this series can be browsed directly in the repository at [moby/vpnkit:/reports](https://github.com/moby/vpnkit/tree/master/reports/).
+
+[@avsm]: https://github.com/avsm
+[@yallop]: https://github.com/yallop
+[@djs55]: https://github.com/djs55
+[djs55/ocaml-osx-dnssd]: https://github.com/djs55/ocaml-osx-dnssd
+[mirage/mirage-tcpip]: https://github.com/mirage/mirage-tcpip
+[mirage/ocaml-dns]: https://github.com/mirage/ocaml-dns
+[mirage/ocaml-hvsock]: https://github.com/mirage/ocaml-hvsock
+[moby/vpnkit]: https://github.com/moby/vpnkit
+[moby/vpnkit#215]: https://github.com/moby/vpnkit/pull/215
+[moby/vpnkit#217]: https://github.com/moby/vpnkit/pull/217
+[moby/vpnkit#218]: https://github.com/moby/vpnkit/pull/218
+

reports/2017-05-14.md

@@ -0,0 +1,35 @@
+# Weekly VPNKit dev report for 2017-05-08 to 2017-05-14 (week 19)
+
+This report covers weekly developments in the [moby/vpnkit], [mirage/ocaml-dns], [mirage/ocaml-hvsock], [djs55/ocaml-osx-dnssd], [mirage/mirage-tcpip] and [djs55/ocaml-dns-forward] repositories. These form the basis of the VPNKit toolkit used to translate container network traffic into native platform socket calls on Windows and OSX.
+
+This week primarily saw fixes in the MirageOS TCP/IP stack relating to MTU handling.  There were a couple of library releases with relevant fixes: [tcpip-3.1.1](https://github.com/mirage/mirage-tcpip/releases/tag/v3.1.1) fixed parsing issues, and [tcpip-3.1.2](https://github.com/mirage/mirage-tcpip/releases/tag/v3.1.2) limited the TCP MSS to never exceed the local MTU, which was a problem on virtio-based backends.
+
+VPNKit also got a new feature to add builtin DNS names, which make it easier to map hostnames into virtual IP addresses being managed by VPNKit. This patch added a command-line argument `--host-names` that accepts a comma-separated list of DNS names which will be mapped to the current host virtual IP ([moby/vpnkit#220] [@djs55]).  The default name is `vpnkit.host` but this can be overriden or extended.
+
+## PRs merged
+
+- Compatibility with lwt >= 2.7.0 ([mirage/mirage-tcpip#308] [@djs55])
+- Stop sending overly large packets greater than the MTU ([mirage/mirage-tcpip#309] [@djs55] [@avsm] [@hannesm])
+- Prepare to release v3.1.1 [mirage/mirage-tcpip#311] [@yomimono] [@djs55]
+- Prepare to release v3.1.2 [mirage/mirage-tcpip#312] [@hannesm] [@djs55]
+
+Other reports in this series can be browsed directly in the repository at [moby/vpnkit:/reports](https://github.com/moby/vpnkit/tree/master/reports/).
+
+[@avsm]: https://github.com/avsm
+[@djs55]: https://github.com/djs55
+[@hannesm]: https://github.com/hannesm
+[@yomimono]: https://github.com/yomimono
+[djs55/ocaml-dns-forward]: https://github.com/djs55/ocaml-dns-forward
+[djs55/ocaml-osx-dnssd]: https://github.com/djs55/ocaml-osx-dnssd
+[mirage/mirage-tcpip]: https://github.com/mirage/mirage-tcpip
+[mirage/mirage-tcpip#308]: https://github.com/mirage/mirage-tcpip/pull/308
+[mirage/mirage-tcpip#309]: https://github.com/mirage/mirage-tcpip/pull/309
+[mirage/mirage-tcpip#311]: https://github.com/mirage/mirage-tcpip/pull/311
+[mirage/mirage-tcpip#312]: https://github.com/mirage/mirage-tcpip/pull/312
+[mirage/ocaml-dns]: https://github.com/mirage/ocaml-dns
+[mirage/ocaml-hvsock]: https://github.com/mirage/ocaml-hvsock
+[moby/vpnkit]: https://github.com/moby/vpnkit
+[moby/vpnkit#220]: https://github.com/moby/vpnkit/pull/220
+[moby/vpnkit#221]: https://github.com/moby/vpnkit/issues/221
+
+

reports/2017-05-28.md

@@ -0,0 +1,70 @@
+# Weekly VPNkit dev report for 2017-05-15 to 2017-05-28 (weeks 20 and 21)
+
+This report covers weekly developments in the [moby/vpnkit],
+[mirage/ocaml-dns], [mirage/ocaml-hvsock], [djs55/ocaml-osx-dnssd],
+[mirage/mirage-tcpip] and [djs55/ocaml-dns-forward] repositories. These form
+the basis of the VPNKit toolkit used to translate container network traffic
+into native platform socket calls on Windows and OSX.
+
+VPNKit now has an experimental transparent HTTP proxy builtin, which lets it
+proxy all outgoing traffic on port 80 at the HTTP level, and all outgoing
+traffic on port 443 via `HTTP CONNECT`.  This has the advantage that the client
+doesn't need to know it's being proxied so that if the client moves on the
+network everything continues to work. ([moby/vpnkit#225] [@djs55]).
+
+Scalability of VPNKit was also improved for a large number of connections,
+and is now configurable via the `--listen-backlog <n>` flag.
+
+Various spurious DNS failures were also fixed, leading to the release of
+[mirage/ocaml-dns:0.20.1].
+
+One area where we could use contributions in TCP/IP is with the test suite,
+to help spot regressions in performance and retransmission behaviour.  Please
+see [mirage/mirage-tcpip#140] or [mirage/mirage-tcpip#314] for more information,
+as well as the [testing tcpip](http://canopy.mirage.io/Projects/test-tcpip) page
+on the MirageOS dev wiki.
+
+## PRs merged
+
+- Add an experimental transparent HTTP proxy ([moby/vpnkit#225] [@djs55])
+- Avoid logging spurious DNS AAAA lookup failures ([moby/vpnkit#226] [@djs55])
+- Fix warnings, selectively re-enable `-warn-error` ([moby/vpnkit#222] [@djs55])
+- Add command-line argument `--listen-backlog <n>` ([moby/vpnkit#223] [@djs55])
+- Packaging: Update upstream packages ([moby/vpnkit#224] [@djs55])
+- Update to Lwt.3.0.0 ([mirage/ocaml-dns#136] [@samoht])
+- Update CHANGES ([mirage/ocaml-dns#140] [@samoht])
+- Add missing `doc` field in opam file ([mirage/ocaml-dns#141] [@samoht])
+- Test MTUs in TCP/IP ([mirage/mirage-tcpip#313] [@djs55])
+- Switch to jbuilder and topkg for fast builds in hvsock ([mirage/ocaml-hvsock#53])
+
+Other reports in this series can be browsed directly in the repository at
+[moby/vpnkit:/reports](https://github.com/moby/vpnkit/tree/master/reports/).
+
+[@djs55]: https://github.com/djs55
+[@samoht]: https://github.com/samoht
+[@talex5]: https://github.com/talex5
+[@yomimono]: https://github.com/yomimono
+[djs55/ocaml-dns-forward]: https://github.com/djs55/ocaml-dns-forward
+[djs55/ocaml-osx-dnssd]: https://github.com/djs55/ocaml-osx-dnssd
+[mirage/mirage-tcpip]: https://github.com/mirage/mirage-tcpip
+[mirage/mirage-tcpip#140]: https://github.com/mirage/mirage-tcpip/issues/140
+[mirage/mirage-tcpip#313]: https://github.com/mirage/mirage-tcpip/pull/313
+[mirage/mirage-tcpip#314]: https://github.com/mirage/mirage-tcpip/issues/314
+[mirage/mirage-tcpip#315]: https://github.com/mirage/mirage-tcpip/pull/315
+[mirage/mirage-tcpip:v3.1.3]: https://github.com/mirage/mirage-tcpip/releases/tag/v3.1.3
+[mirage/ocaml-dns]: https://github.com/mirage/ocaml-dns
+[mirage/ocaml-dns#136]: https://github.com/mirage/ocaml-dns/pull/136
+[mirage/ocaml-dns#139]: https://github.com/mirage/ocaml-dns/pull/139
+[mirage/ocaml-dns#140]: https://github.com/mirage/ocaml-dns/pull/140
+[mirage/ocaml-dns#141]: https://github.com/mirage/ocaml-dns/pull/141
+[mirage/ocaml-dns:0.20.1]: https://github.com/mirage/ocaml-dns/releases/tag/0.20.1
+[mirage/ocaml-hvsock]: https://github.com/mirage/ocaml-hvsock
+[mirage/ocaml-hvsock#53]: https://github.com/mirage/ocaml-hvsock/pull/53
+[moby/vpnkit]: https://github.com/moby/vpnkit
+[moby/vpnkit#222]: https://github.com/moby/vpnkit/pull/222
+[moby/vpnkit#223]: https://github.com/moby/vpnkit/pull/223
+[moby/vpnkit#224]: https://github.com/moby/vpnkit/pull/224
+[moby/vpnkit#219]: https://github.com/moby/vpnkit/pull/219
+[moby/vpnkit#225]: https://github.com/moby/vpnkit/pull/225
+[moby/vpnkit#226]: https://github.com/moby/vpnkit/pull/226
+