Adding IPv6 network support to docker daemon #8947

Merged
merged 1 commit into from Jan 8, 2015

Projects

None yet
@MalteJ
Contributor
MalteJ commented Nov 4, 2014

This PR adds IPv6 support to the docker daemon.
Features:

  • adding CLI flags --ipv6 and --fixed-cidr-v6
  • Setting the bridge IPv6 to fe80::1/64. This IP will be used by the containers as gateway.
  • registers the fixed-cidr-v6 in the ipallocator

Missing:
Test and Documentation.
Follow-up PR for enabling the container to use IPv6. (I have the code but still have to clean it up a bit and create a PR).
I wanted to add them but #8896 popped up and I thought it would be a good idea to provide my code so we can compare and pick the best of both.

Update 2014-11-04

  • Documentation is improving
  • Container IPv6 Implementation is included in this PR as a separate commit
@justinsb
justinsb commented Nov 4, 2014

Hi @MalteJ - this looks good to me - much more comprehensive than my patch (support for both IPv4 & IPv6 at the same time, whereas I was going to do that patch-by-patch).

The big thing that isn't in this patch seems to be the iptables stuff (which is mostly just because you have to use "ip6tables" for IPv6 tables). Do you want to incorporate that into this patch, or shall we try to land them separately?

Also, I skip the first IPv6 IP in the ip allocator, whereas you deliberately didn't. Are you OK with skipping the first IPv6?

@MalteJ
Contributor
MalteJ commented Nov 4, 2014

@justinsb thank you!
Yes, I would like to use your iptables code. Do you want to send a PR to me or should I just copy your code?

Why do you want to skip the first IP in the range? Is it used by your gateway?
I would like to use the link-local addresses as gateway address. Hence I am setting the bridge IPv6 to fe80::1. It is something like the default gateway IPv6 by convention.

@MalteJ
Contributor
MalteJ commented Nov 4, 2014

The additional container IPv6 implementation can be found at https://github.com/MalteJ/docker/commits/ipv6-container
I am still unsure how to organize the Pull Requests so they are small and easy to review. Create one large PR with multiple commits or going with this PR and sending a new PR with the container IPv6 code once this is merged?

Best,
Malte

@justinsb
justinsb commented Nov 4, 2014

Why do you want to skip the first IP in the range? Is it used by your gateway?

If I assign an IPv6 range to the bridge, I will typically use ::1 for the bridge IP. But I am then happy for the first docker instance to be ::2. (At least I don't know of a reason why not.) If we don't skip the first one then I have to specify both --bip and --fixed-cidr (and I lose some of the bip range because fixed-cidr has to be a subnet).

Does this make sense? Am I missing something?

I am still unsure how to organize the Pull Requests so they are small and easy to review.

I'm not sure either... @jfrazelle ?

@MalteJ
Contributor
MalteJ commented Nov 4, 2014

I just don't assign a global IPv6 to the bridge. In the container I am using the bridge's link-local address as gateway.
One thing that would argue for skipping the first IPv6 is that you can set this address to your host's eth0 and address the host from the outside via this address.
Talking about the ipallocator I would also like to generate the IPv6 addresses from the containers' MAC addresses once you have IPv6 subnets bigger than /80 to avoid ndp neighbor cache invalidation issues (see #8371).

@jfrazelle recommended to push the container IPv6 into this PR as a separate commit.
I have done that. You should be able to review the first commits separately as they do not override each other.

@MalteJ
Contributor
MalteJ commented Nov 4, 2014

Ohh, looks like the last commit has broken some tests.

@MalteJ
Contributor
MalteJ commented Nov 4, 2014

You should be able to test it anyway.

# remove your bridge
ifconfig docker0 down
brctl delbr docker0

# run docker daemon with IPv6
docker -d --ipv6 --fixed-cidr-v6="2a00:1450::/64"

# have a look at the new bridge
ifconfig docker0

and create a container

docker run -it --rm --global-ipv6 --name ipv6-test ubuntu bash
# in the container:
ifconfig eth0
route -A inet6

and on the docker host

docker inspect ipv6-test
@jessfraz
Contributor
jessfraz commented Nov 4, 2014

let me make sure the build server has ipv6 support

@MalteJ
Contributor
MalteJ commented Nov 4, 2014

I suspect the tests fail because there is something wrong with the code or the tests.

@jessfraz
Contributor
jessfraz commented Nov 5, 2014

hmm ya looks like its failing creating the bridge, same for me locally as well, and both the build server and my host have ipv6 support

@jessfraz jessfraz commented on an outdated diff Nov 5, 2014
daemon/networkdriver/bridge/driver_test.go
@@ -58,6 +59,9 @@ func TestAllocatePortDetection(t *testing.T) {
// Init driver
job := eng.Job("initdriver")
+ job.Stderr.Add(os.Stderr)
+ job.Stdout.Add(os.Stdout)
+ job.SetenvBool("EnableIPv6", true)
@jessfraz
jessfraz Nov 5, 2014 Contributor

we should probably have a test with and a test without, also this is the test that is failing

@MalteJ
Contributor
MalteJ commented Nov 5, 2014

I have reverted to the original test.
Testing the whole thing on another machine I could reproduce your issues when creating the bridge. I have committed a fix for that.
Are you now able to create an IPv6 enabled container on your machine using my instructions above?

I have added further documentation.
This PR is still missing tests and ip6tables.

@MalteJ
Contributor
MalteJ commented Nov 5, 2014

I tried to run this implementation on Digital Ocean but their IPv6 networking does not fit my approach.
I am using the Docker host as a gateway that routes IPv6 traffic of a specified subnet to its docker0 bridge. The containers use the Docker host's bridge interface link-local IPv6 as their gateway.

On Digital Ocean you use a part (16 addresses / 4 bits) of a /64 network. But this part of the /64 network is not a subnet. You are just allowed to use those 16 IPs. This means you are expected to respond to ndp neighbor solicitation requests for those 16 IPs. There is no route set up that uses your VM as a gateway for this subnet.

The result: My approach does not work on DigitalOcean.

How do you think about this?

There is a german IaaS provider (www.jiffybox.de) that gives you a /56 subnet for your VM. Here everything works fine.
I don't really like DigitalOcean's approach as in any way you would have to hack your network settings and connect the docker bridge to the host's eth0 or do ip6tables NAT stuff or foo like this.

@MalteJ
Contributor
MalteJ commented Nov 5, 2014

This said, I think I have forgotten to implement adding the route to the fixed-cidr-v6 on the docker host machine.
But that doesn't change anything on the DigitalOcean issues.

@justinsb
justinsb commented Nov 5, 2014

Hi @MalteJ - let's coordinate a plan here: I'm happy to merge my patch into yours - I guess I should propose a patch on your repo with my ip6tables changes. Does that sound good to you?

I have an OVH box which has a /64 which I can try this out on. I've also been trying this on EC2 using protocol 41.

I have a busy day, but hopefully I can get to this tonight.

Does that work?

@MalteJ
Contributor
MalteJ commented Nov 5, 2014

@justinsb sounds great! :)

@jessfraz
Contributor
jessfraz commented Nov 5, 2014

ping @skottler do you think digital ocean will be supporting this soon?

@discordianfish
Contributor

Yup, looks like digitalocean's IPv6 support is incomplete. Maybe something @progrium is interested in as well and could help to fix :)

@LK4D4
Contributor
LK4D4 commented Nov 17, 2014

@MalteJ @justinsb Hello, any news about this PR? Do you have plans to add more commits here or it is ready for review?

@SvenDowideit
Collaborator

Docs LGTM - @fredlf @jamtur01

I presume that getting the Docker daemon to listen on IPv6 port will need to be done in a separate PR? (It'll be useful in b2d)

@MalteJ
Contributor
MalteJ commented Nov 17, 2014

Hi @LK4D4
@justinsb wants to add ip6tables support.
I want to add some tests but hadn't have time to do so, yet.
From my side all functionality is implemented and ready to review.

I have merged the current master to this PR and resolved the conflicts. Hopefully the tests are still green.

@SvenDowideit yeah, I haven't thought of this until now.
I am okay with a separate PR. I think listening to an IPv6 port would even work right now, without this PR.

@jamtur01 jamtur01 and 1 other commented on an outdated diff Nov 17, 2014
daemon/networkdriver/bridge/driver.go
@@ -318,6 +394,25 @@ func configureBridge(bridgeIP string) error {
if netlink.NetworkLinkAddIp(iface, ipAddr, ipNet); err != nil {
return fmt.Errorf("Unable to add private network: %s", err)
}
+
+ if enableIPv6 {
+ // Enable IPv6 on the bridge
+ procFile := "/proc/sys/net/ipv6/conf/" + iface.Name + "/disable_ipv6"
+ if err := ioutil.WriteFile(procFile, []byte{'0', '\n'}, 0644); err != nil {
+ return fmt.Errorf("Unable to enable IPv6 addresses on bridge: %s\n", err)
@jamtur01
jamtur01 Nov 17, 2014 Contributor

Should output start with a lower case character?

@MalteJ
MalteJ Nov 17, 2014 Contributor

don't know!? should it?

@jamtur01 jamtur01 commented on an outdated diff Nov 17, 2014
docs/man/docker-run.1.md
@@ -137,6 +138,11 @@ ENTRYPOINT.
**--expose**=[]
Expose a port or a range of ports (e.g. --expose=3300-3310) from the container without publishing it to your host
+**--global-ipv6**=*true*|*false*
+ Assign a global IPv6 to the container and add a default route for IPv6. This
@jamtur01
jamtur01 Nov 17, 2014 Contributor

Assign a global IPv6... ? Assign a "what"? Address? Interface?

@jamtur01 jamtur01 commented on an outdated diff Nov 17, 2014
docs/man/docker-run.1.md
@@ -137,6 +138,11 @@ ENTRYPOINT.
**--expose**=[]
Expose a port or a range of ports (e.g. --expose=3300-3310) from the container without publishing it to your host
+**--global-ipv6**=*true*|*false*
+ Assign a global IPv6 to the container and add a default route for IPv6. This
+ requires the daemon to be started with --ipv6 and a global IPv6 subnet
+ defined via daemon's --fixed-cidr-v6 parameter. The default is *false*.
@jamtur01
jamtur01 Nov 17, 2014 Contributor

via the daemon's

@jamtur01 jamtur01 commented on an outdated diff Nov 17, 2014
docs/man/docker.1.md
@@ -65,6 +67,9 @@ unix://[/path/to/socket] to use.
**--iptables**=*true*|*false*
Disable Docker's addition of iptables rules. Default is true.
+**--ipv6**=*true*|*false*
+ Enable IPv6 support. Default is false. Docker will create an IPv6 enabled bridge with IP fe80::1 and allows to create IPv6 enabled containers.
@jamtur01
jamtur01 Nov 17, 2014 Contributor

and allows to create... Doesn't make sense.

which allows you to create... perhaps.

@jamtur01 jamtur01 commented on an outdated diff Nov 17, 2014
docs/sources/articles/networking.md
+By default the Docker server configures the container network for IPv4 only. You
+can enable IPv4/[IPv6](http://en.wikipedia.org/wiki/IPv6) dualstack support by
+running the Docker daemon with the `--ipv6` flag.
+Docker will set up the bridge `docker0` with the IPv6
+[link-local address](http://en.wikipedia.org/wiki/Link-local_address) `fe80::1`. You
+can configure Docker to assign global IPv6 addresses to containers by setting
+the `--fixed-cidr-v6` parameter:
+
+ docker -d --ipv6 --fixed-cidr-v6="2a00:1450::/64"
+
+To create a Docker container with a global IPv6 address execute the `docker run`
+command with the `--global-ipv6` flag:
+
+ docker run -it --global-ipv6 ubuntu bash -c "ifconfig eth0; route -A inet6"
+
+You will get an output like:
@jamtur01
jamtur01 Nov 17, 2014 Contributor

-an

@jamtur01 jamtur01 commented on an outdated diff Nov 17, 2014
docs/sources/articles/networking.md
+ RX bytes:110 (110.0 B) TX bytes:110 (110.0 B)
+
+ Kernel IPv6 routing table
+ Destination Next Hop Flag Met Ref Use If
+ 2a00:1450::/64 :: U 256 0 0 eth0
+ fe80::/64 :: U 256 0 0 eth0
+ ::/0 fe80::1 UG 1024 0 0 eth0
+ ::/0 :: !n -1 1 1 lo
+ ::1/128 :: Un 0 1 0 lo
+ ff00::/8 :: U 256 1 0 eth0
+ ::/0 :: !n -1 1 1 lo
+
+As you can see the Docker container will get a link-local address with the
+network prefix `/64` (here: `fe80::42:acff:fe11:2/64`) and a global IPv6 (here:
+ `2a00:1450::1/64`). The container will create connections to addresses outside
+of the `2a00:1450::/64` network via the link-local gateway `fe80::1` - the
@jamtur01
jamtur01 Nov 17, 2014 Contributor

via the link-local gateway fe80::1. This bridge was created by setting...

@jamtur01 jamtur01 commented on an outdated diff Nov 17, 2014
docs/sources/reference/api/docker_remote_api.md
@@ -10,7 +10,7 @@ page_keywords: API, Docker, rcli, REST, documentation
or `--tlsverify`) as with Boot2Docker 1.3.0, then you need to add extra
parameters to `curl` or `wget` when making test API requests:
`curl --insecure --cert ~/.docker/cert.pem --key ~/.docker/key.pem https://boot2docker:2376/images/json`
- or
@jamtur01
jamtur01 Nov 17, 2014 Contributor

Extraneous whitespace?

@jamtur01 jamtur01 commented on an outdated diff Nov 17, 2014
docs/sources/reference/commandline/cli.md
@@ -1247,6 +1248,7 @@ removed before the image is removed.
--entrypoint="" Overwrite the default ENTRYPOINT of the image
--env-file=[] Read in a line delimited file of environment variables
--expose=[] Expose a port or a range of ports (e.g. --expose=3300-3310) from the container without publishing it to your host
+ --global-ipv6=false Assign a global IPv6 to the container and add a default route for IPv6 (see [IPv6](/articles/networking#ipv6)).
@jamtur01
jamtur01 Nov 17, 2014 Contributor

Assign a global IPv6 ... what - as above.

@jamtur01 jamtur01 commented on an outdated diff Nov 17, 2014
docs/sources/reference/run.md
@@ -150,14 +150,15 @@ of the containers.
## Network settings
- --dns=[] : Set custom dns servers for the container
- --net="bridge" : Set the Network mode for the container
- 'bridge': creates a new network stack for the container on the docker bridge
- 'none': no networking for this container
- 'container:<name|id>': reuses another container network stack
- 'host': use the host network stack inside the container
- --add-host="" : Add a line to /etc/hosts (host:IP)
- --mac-address="" : Sets the container's Ethernet device's MAC address
+ --dns=[] : Set custom dns servers for the container
@jamtur01
jamtur01 Nov 17, 2014 Contributor

DNS

@jamtur01 jamtur01 commented on an outdated diff Nov 17, 2014
docs/sources/reference/run.md
@@ -172,6 +173,12 @@ By default a random MAC is generated. You can set the container's MAC address
explicitly by providing a MAC via the `--mac-address` parameter (format:
`12:34:56:78:9a:bc`).
+Global IPv6 networking is disabled by default. To assign a global IPv6 to the
@jamtur01
jamtur01 Nov 17, 2014 Contributor

To assign a global IPv6 ... as above

@jamtur01 jamtur01 and 1 other commented on an outdated diff Nov 17, 2014
docs/sources/reference/run.md
@@ -459,7 +466,7 @@ with `-P` or `-p,` or start the client container with `--link`.
If the operator uses `-P` or `-p` then Docker will make the exposed port
accessible on the host and the ports will be available to any client
-that can reach the host. When using `-P`, Docker will bind the exposed
@jamtur01
jamtur01 Nov 17, 2014 Contributor

More extraneous whitespace

@MalteJ
MalteJ Nov 17, 2014 Contributor

yeah, I have tried the Atom editor and it looks like it has removed some existing extranous whitespace.

@jamtur01 jamtur01 and 1 other commented on an outdated diff Nov 17, 2014
runconfig/parse.go
- OpenStdin: *flStdin,
- Memory: flMemory,
- CpuShares: *flCpuShares,
- Cpuset: *flCpuset,
- AttachStdin: attachStdin,
- AttachStdout: attachStdout,
- AttachStderr: attachStderr,
- Env: envVariables,
- Cmd: runCmd,
- Image: image,
- Volumes: flVolumes.GetMap(),
- MacAddress: *flMacAddress,
- Entrypoint: entrypoint,
- WorkingDir: *flWorkingDir,
- SecurityOpt: flSecurityOpt.GetAll(),
+ Hostname: hostname,
@jamtur01
jamtur01 Nov 17, 2014 Contributor

What's happening here with formatting?

@MalteJ
MalteJ Nov 17, 2014 Contributor

some lines below is a longer variable name. I have adjusted the formatting so everything is in line.

@jamtur01
Contributor

Nice work. Docs need some tweaks though.

@MalteJ
Contributor
MalteJ commented Nov 17, 2014

@jamtur01 thank you.
I'm always trying to improve my docs and am glad to have a proofreader like you :)

@jessfraz jessfraz added this to the 1.4 milestone Nov 18, 2014
@crosbymichael crosbymichael added the UX label Nov 18, 2014
@discordianfish
Contributor

@jfrazelle / @crosbymichael / @shykes Do we want to support also interactions with our central services via ipv6? If so, we should discuss that soon.

@fredlf fredlf commented on an outdated diff Nov 19, 2014
docs/man/docker.1.md
**--fixed-cidr**=""
- IPv4 subnet for fixed IPs (ex: 10.20.0.0/16); this subnet must be nested in the bridge subnet (which is defined by \-b or \-\-bip)
+ IPv4 subnet for fixed IPs (e.g.: 10.20.0.0/16); this subnet must be nested in the bridge subnet (which is defined by \-b or \-\-bip)
+
+**--fixed-cidr-v6**=""
+ IPv6 subnet for global IPv6 addresses (e.g.: 2a00:1450::/64)
@fredlf
fredlf Nov 19, 2014 Contributor

Comma after "e.g.", not colon, please.

@fredlf fredlf commented on an outdated diff Nov 19, 2014
docs/man/docker.1.md
@@ -65,6 +67,9 @@ unix://[/path/to/socket] to use.
**--iptables**=*true*|*false*
Disable Docker's addition of iptables rules. Default is true.
+**--ipv6**=*true*|*false*
+ Enable IPv6 support. Default is false. Docker will create an IPv6 enabled bridge with address fe80::1 which allows you to create IPv6 enabled containers.
@fredlf
fredlf Nov 19, 2014 Contributor

Parallelism: "... which will allow you to..." Also, IPv6-enabled needs a hyphen, please.

@fredlf fredlf commented on an outdated diff Nov 19, 2014
docs/sources/articles/networking.md
@@ -363,6 +369,54 @@ Again, this topic is covered without all of these low-level networking
details in the [Docker User Guide](/userguide/dockerlinks/) document if you
would like to use that as your port redirection reference instead.
+## IPv6
+
+<a name="ipv6"></a>
+
+By default the Docker server configures the container network for IPv4 only. You
@fredlf
fredlf Nov 19, 2014 Contributor

Comma after "default", please.

@fredlf fredlf commented on an outdated diff Nov 19, 2014
docs/sources/articles/networking.md
+<a name="ipv6"></a>
+
+By default the Docker server configures the container network for IPv4 only. You
+can enable IPv4/[IPv6](http://en.wikipedia.org/wiki/IPv6) dualstack support by
+running the Docker daemon with the `--ipv6` flag.
+Docker will set up the bridge `docker0` with the IPv6
+[link-local address](http://en.wikipedia.org/wiki/Link-local_address) `fe80::1`. You
+can configure Docker to assign global IPv6 addresses to containers by setting
+the `--fixed-cidr-v6` parameter:
+
+ docker -d --ipv6 --fixed-cidr-v6="2a00:1450::/64"
+
+To create a Docker container with a global IPv6 address execute the `docker run`
+command with the `--global-ipv6` flag:
+
+ docker run -it --global-ipv6 ubuntu bash -c "ifconfig eth0; route -A inet6"
@fredlf
fredlf Nov 19, 2014 Contributor

I found this confusing. I think we're missing some context. If I understand correctly, the --fixed-cidr-v6 parameter tells the Docker daemon to use IPv6 addresses in the specified range. The next parameter, --global-ipv6, tells a container to use an IPv6 address from the previously specified range. Can you please clarify the relationships between all three of these parameters as you introduce them? Thanks.

@fredlf fredlf commented on an outdated diff Nov 19, 2014
docs/sources/articles/networking.md
+
+ Kernel IPv6 routing table
+ Destination Next Hop Flag Met Ref Use If
+ 2a00:1450::/64 :: U 256 0 0 eth0
+ fe80::/64 :: U 256 0 0 eth0
+ ::/0 fe80::1 UG 1024 0 0 eth0
+ ::/0 :: !n -1 1 1 lo
+ ::1/128 :: Un 0 1 0 lo
+ ff00::/8 :: U 256 1 0 eth0
+ ::/0 :: !n -1 1 1 lo
+
+As you can see the Docker container will get a link-local address with the
+network prefix `/64` (here: `fe80::42:acff:fe11:2/64`) and a global IPv6 (here:
+ `2a00:1450::1/64`). The container will create connections to addresses outside
+of the `2a00:1450::/64` network via the link-local gateway `fe80::1`. The
+bridge was created by setting the `--ipv6` flag when executing the Docker
@fredlf
fredlf Nov 19, 2014 Contributor

Which bridge?

@fredlf fredlf commented on an outdated diff Nov 19, 2014
docs/sources/reference/api/docker_remote_api.md
@@ -55,7 +55,8 @@ total memory available (`MemTotal`).
`POST /containers/create`
**New!**
-You can set the new container's MAC address explicitly.
+- You can set the new container's MAC address explicitly.
+- Enable global IPv6 networking.
@fredlf
fredlf Nov 19, 2014 Contributor

For consistency, revise to "You can enable global IPv6 networking", please.

@fredlf fredlf commented on an outdated diff Nov 19, 2014
docs/sources/articles/networking.md
@@ -363,6 +369,54 @@ Again, this topic is covered without all of these low-level networking
details in the [Docker User Guide](/userguide/dockerlinks/) document if you
would like to use that as your port redirection reference instead.
+## IPv6
+
+<a name="ipv6"></a>
+
+By default the Docker server configures the container network for IPv4 only. You
+can enable IPv4/[IPv6](http://en.wikipedia.org/wiki/IPv6) dualstack support by
+running the Docker daemon with the `--ipv6` flag.
+Docker will set up the bridge `docker0` with the IPv6
+[link-local address](http://en.wikipedia.org/wiki/Link-local_address) `fe80::1`. You
+can configure Docker to assign global IPv6 addresses to containers by setting
@fredlf
fredlf Nov 19, 2014 Contributor

Not exactly sure what you mean by "global" here. Can you clarify and explain for the user?

@fredlf fredlf commented on an outdated diff Nov 19, 2014
docs/sources/reference/run.md
@@ -150,14 +150,15 @@ of the containers.
## Network settings
- --dns=[] : Set custom dns servers for the container
- --net="bridge" : Set the Network mode for the container
- 'bridge': creates a new network stack for the container on the docker bridge
- 'none': no networking for this container
- 'container:<name|id>': reuses another container network stack
- 'host': use the host network stack inside the container
- --add-host="" : Add a line to /etc/hosts (host:IP)
- --mac-address="" : Sets the container's Ethernet device's MAC address
+ --dns=[] : Set custom DNS servers for the container
+ --net="bridge" : Set the Network mode for the container
@fredlf
fredlf Nov 19, 2014 Contributor

lower-case "network"

@fredlf fredlf commented on an outdated diff Nov 19, 2014
docs/sources/reference/run.md
@@ -150,14 +150,15 @@ of the containers.
## Network settings
- --dns=[] : Set custom dns servers for the container
- --net="bridge" : Set the Network mode for the container
- 'bridge': creates a new network stack for the container on the docker bridge
- 'none': no networking for this container
- 'container:<name|id>': reuses another container network stack
- 'host': use the host network stack inside the container
- --add-host="" : Add a line to /etc/hosts (host:IP)
- --mac-address="" : Sets the container's Ethernet device's MAC address
+ --dns=[] : Set custom DNS servers for the container
+ --net="bridge" : Set the Network mode for the container
+ 'bridge': creates a new network stack for the container on the docker bridge
@fredlf
fredlf Nov 19, 2014 Contributor

Capitalize "Docker"

@fredlf fredlf commented on an outdated diff Nov 19, 2014
docs/sources/reference/run.md
@@ -253,7 +260,7 @@ the container exits**, you can add the `--rm` flag:
--security-opt="label:type:TYPE" : Set the label type for the container
--security-opt="label:level:LEVEL" : Set the label level for the container
--security-opt="label:disable" : Turn off label confinement for the container
- --secutity-opt="apparmor:PROFILE" : Set the apparmor profile to be applied
+ --secutity-opt="apparmor:PROFILE" : Set the apparmor profile to be applied
@fredlf
fredlf Nov 19, 2014 Contributor

camel case: AppArmor

@fredlf
Contributor
fredlf commented Nov 19, 2014

Still a few issues with docs content. Needs revision as suggested. Many thanks.

@LK4D4
Contributor
LK4D4 commented Nov 25, 2014

@MalteJ do you have time to fix docs issues? Thanks!

@MalteJ
Contributor
MalteJ commented Nov 25, 2014

@LK4D4 yes, I'll do that in the next hours.

@jessfraz
Contributor

awesome ping us when you are ready :)

@MalteJ
Contributor
MalteJ commented Nov 25, 2014

@fredlf thanks for your review. I have made some changes according to your remarks :-)

@LK4D4 @jfrazelle ping

@jessfraz
Contributor

awesome you are going to have to squash the merge commits before we merge, just a warning, but I am testing this out now thanks!

@MalteJ
Contributor
MalteJ commented Nov 25, 2014

yeah, I haven't squashed them yet so you can see the increments during review.

@jessfraz
Contributor
Nov 25 11:06:05 debian docker[18361]: bridge IPv6 does not match existing bridge configuration fe80::1
Nov 25 11:06:05 debian docker[18361]: time="2014-11-25T11:06:02-08:00" level="info" msg="-job init_networkdriver() = ERR (1)"
Nov 25 11:06:05 debian docker[18361]: time="2014-11-25T11:06:02-08:00" level="fatal" msg="bridge IPv6 does not match existing bridge configuration fe80::1"

EDIT: I know I need to delete old bridge

@LK4D4
Contributor
LK4D4 commented Nov 25, 2014

@jfrazelle remove existing docker0

@MalteJ
Contributor
MalteJ commented Nov 25, 2014

@jfrazelle That's docker's default. If you have an existing bridge that doesn't fit the defined config, the daemon logs a fatal message saying there is something wrong.
You have two options:

  1. change your bridge config manually e.g. ifconfig docker0 inet6 add fe80::1/64
  2. delete the existing bridge brctl delbr docker0
@jessfraz
Contributor
  1. Even if I delete the old bride and set up new, it works only once, so restarting daemon I need to delete it again to not have that error
  2. starting a container with ipv6 enabed (docker run -it --rm --global-ipv6 --name ipv6-test ubuntu bash) gives me a panic
DEBU[0013] Calling POST /containers/{name:.*}/start     
INFO[0013] POST /v1.16/containers/95bb1f61df1ff74ff5a7059304d5a735e1415d46a9694d0ede0a0b6ca4a1474a/start 
INFO[0013] +job start(95bb1f61df1ff74ff5a7059304d5a735e1415d46a9694d0ede0a0b6ca4a1474a) 
INFO[0013] +job allocate_interface(95bb1f61df1ff74ff5a7059304d5a735e1415d46a9694d0ede0a0b6ca4a1474a) 
INFO[0013] Allocate: IPv6 enabled                       
ERRO[0013] ERROR: Allocate: globalIPv6Network = nil!

INFO[0013] -job allocate_interface(95bb1f61df1ff74ff5a7059304d5a735e1415d46a9694d0ede0a0b6ca4a1474a) 
INFO[0013] -job start(95bb1f61df1ff74ff5a7059304d5a735e1415d46a9694d0ede0a0b6ca4a1474a) 
2014/11/25 11:19:44 http: panic serving @: runtime error: invalid memory address or nil pointer dereference
goroutine 50 [running]:
net/http.func·011()
    /usr/local/go/src/pkg/net/http/server.go:1100 +0xb7
runtime.panic(0xbe6240, 0x1279af3)
    /usr/local/go/src/pkg/runtime/panic.c:248 +0x18d
net.networkNumberAndMask(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/ip.go:433 +0x202
net.(*IPNet).String(0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/ip.go:483 +0x4a
github.com/docker/docker/daemon/networkdriver/ipallocator.RequestIP(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/docker/docker/daemon/networkdriver/ipallocator/allocator.go:87 +0xa8
github.com/docker/docker/daemon/networkdriver/bridge.Allocate(0xc208254500, 0x7fe4ef8e97d8)
    /go/src/github.com/docker/docker/daemon/networkdriver/bridge/driver.go:500 +0x450
github.com/docker/docker/engine.(*Job).Run(0xc208254500, 0x0, 0x0)
    /go/src/github.com/docker/docker/engine/job.go:83 +0x837
github.com/docker/docker/daemon.(*Container).AllocateNetwork(0xc2081e7040, 0x0, 0x0)
    /go/src/github.com/docker/docker/daemon/container.go:491 +0x25b
github.com/docker/docker/daemon.(*Container).initializeNetworking(0xc2081e7040, 0x0, 0x0)
    /go/src/github.com/docker/docker/daemon/container.go:1072 +0x5f6
github.com/docker/docker/daemon.(*Container).Start(0xc2081e7040, 0x0, 0x0)
    /go/src/github.com/docker/docker/daemon/container.go:338 +0x1e3
github.com/docker/docker/daemon.(*Daemon).ContainerStart(0xc2080d6580, 0xc208254480, 0x1)
    /go/src/github.com/docker/docker/daemon/start.go:38 +0x38b
github.com/docker/docker/daemon.*Daemon.ContainerStart·fm(0xc208254480, 0x7fe4ef8e97d8)
    /go/src/github.com/docker/docker/daemon/daemon.go:124 +0x30
github.com/docker/docker/engine.(*Job).Run(0xc208254480, 0x0, 0x0)
    /go/src/github.com/docker/docker/engine/job.go:83 +0x837
github.com/docker/docker/api/server.postContainersStart(0xc2080d7b80, 0xc20817da47, 0x4, 0x7fe4ef8f1d50, 0xc2081cbd60, 0xc2081255f0, 0xc2081c39e0, 0x0, 0x0)
    /go/src/github.com/docker/docker/api/server/server.go:788 +0x1ba
github.com/docker/docker/api/server.func·006(0x7fe4ef8f1d50, 0xc2081cbd60, 0xc2081255f0)
    /go/src/github.com/docker/docker/api/server/server.go:1224 +0x810
net/http.HandlerFunc.ServeHTTP(0xc2081a6e40, 0x7fe4ef8f1d50, 0xc2081cbd60, 0xc2081255f0)
    /usr/local/go/src/pkg/net/http/server.go:1235 +0x40
github.com/gorilla/mux.(*Router).ServeHTTP(0xc20807d9a0, 0x7fe4ef8f1d50, 0xc2081cbd60, 0xc2081255f0)
    /go/src/github.com/docker/docker/vendor/src/github.com/gorilla/mux/mux.go:98 +0x297
net/http.serverHandler.ServeHTTP(0xc2081cfce0, 0x7fe4ef8f1d50, 0xc2081cbd60, 0xc2081255f0)
    /usr/local/go/src/pkg/net/http/server.go:1673 +0x19f
net/http.(*conn).serve(0xc20818a600)
    /usr/local/go/src/pkg/net/http/server.go:1174 +0xa7e
created by net/http.(*Server).Serve
    /usr/local/go/src/pkg/net/http/server.go:1721 +0x313
DEBU[0013] Closing buffered stdin pipe                  
DEBU[0013] attach: stdin: end                           
DEBU[0013] attach: job 1 completed successfully         
DEBU[0013] attach: waiting for job 2/3                  
DEBU[0013] attach: stdout: end                          
DEBU[0013] attach: stderr: end                          
DEBU[0013] attach: job 2 completed successfully         
DEBU[0013] attach: waiting for job 3/3                  
DEBU[0013] attach: job 3 completed successfully         
DEBU[0013] attach: all jobs completed successfully      
INFO[0013] -job attach(95bb1f61df1ff74ff5a7059304d5a735e1415d46a9694d0ede0a0b6ca4a1474a) = OK (0) 
@LK4D4
Contributor
LK4D4 commented Nov 25, 2014

😿

@jessfraz
Contributor

i can't tell if it's just me tho.. I built a new kernel the other day... maybe i didn't enable all the modules although I believe they all are, its the exact .config I always use
but no matter what it shouldn't panic

@LK4D4
Contributor
LK4D4 commented Nov 25, 2014

I think we need to set --fixed-cidr-v6, but panic is not best idea in this case :)

@jessfraz
Contributor

ya I just realized that... there needs to be a default.

@jessfraz
Contributor

it works otherwise if I set the --fixed-cidr-v6 but I don't think most people will (or my instinct at least was to not)

@LK4D4
Contributor
LK4D4 commented Nov 25, 2014

My idea was that if you set --fixed-cidr-v6, than you have global IPv6 addresses and no need --global-.. flag. Actually I think that using network settings per-container is probably not good way.

@jessfraz
Contributor

I agree

@LK4D4
Contributor
LK4D4 commented Nov 25, 2014

I tried communication both through link-local and global addresses. It works well. But we need to figure out defaults for ui and add check if --fixed-cidr-v6 was changed.

@jessfraz
Contributor

So just to sum up (because this was a bit erratic):

  • It works when fixed-cidr is set and can ping container on ipv6 addr etc, awesome!

Things to fix:

  • no panic when --fixed-cidr-v6 is set
  • when --fixed-cird-v6 you do not need --global-.. flag
  • shouldn't have to delete bridge, when restarting daemon if --fixed-cidr changed or if I am using whatever default we come up with and didn't set --fixed-cidr (i understand the need to delete from ipv4 to ipv6 in case they have existing containers etc)
@MalteJ
Contributor
MalteJ commented Nov 25, 2014

@jfrazelle looks good.
Just one correction: The panic happens when you do docker run --global-ipv6 without docker -d --ipv6 --fixed-cidr-v6=...

@jessfraz
Contributor

yes when i run the daemon with just --ipv6 and no fixed-cidr
sorry about that

@MalteJ
Contributor
MalteJ commented Nov 25, 2014

To clarify the Docker IPv6 network design I have sketched it down.
The part below the cyan dashed line is part of this PR and gets initialized during daemon startup (docker0 bridge creation) or during container creation. The part above is up to the user and just an example for a simple network design:
docker-ipv6-network-design

@LK4D4
Contributor
LK4D4 commented Nov 25, 2014

@MalteJ thanks, very helpful.

@jessfraz
Contributor

ya this is awesome

@jessfraz jessfraz removed this from the 1.4 milestone Nov 25, 2014
@thaJeztah
Member

Would be nice to have such image in the docs once the IPv6 implementation is completed!

@fredlf
Contributor
fredlf commented Nov 27, 2014

Agreed, it would be terrific to add the diagram to the docs with a brief explanation (basically just what you say in the comment).

@SvenDowideit SvenDowideit changed the title from Adding IPv6 support to docker daemon to Adding IPv6 network support to docker daemon Nov 27, 2014
@SvenDowideit
Collaborator

I've changed the title of this PR to add 'network' support - as adding IPv6 support to the Docker daemon can also be interpreted as adding -H IPv6 support for the Docker socket.

@MalteJ
Contributor
MalteJ commented Nov 27, 2014

@thaJeztah @fredlf yeah, I will add it to the documentation.

@SvenDowideit thank you!

@thaJeztah
Member

@MalteJ if you do, perhaps you could provide the "source" file of that diagram? Not sure what you used to create it, but it may be useful to have that so that changes can be made in future (wrt styling/colour scheme, size or possibly change in content).

Perhaps @SvenDowideit can say how this should be handled (should it be included in the PR and in the repo, or sent separately?)

@MalteJ
Contributor
MalteJ commented Nov 27, 2014

@thaJeztah it's a gliffy.com diagram.

@thaJeztah
Member

Ah, nice! Didn't know that one. Looks interesting.

A quick look; it's able to export to SVG and a native 'gliffy' file format, think those could be the right format to exchange; SVG for portability and Gliffy to allow further editing (and of course a PNG for the docs).

But I'll leave that to the doc maintainers to decide what is needed :)

@MalteJ
Contributor
MalteJ commented Nov 27, 2014
@SvenDowideit
Collaborator

The syntax for adding an inline image looks like: ![](/article-images/something.png)

and if you put the svg and png into /docs/sources/article-images/, you should be able to see it when you run make docs

@MalteJ
Contributor
MalteJ commented Nov 30, 2014

OK, I have three new things for you:

  • I have removed the --global-ipv6 flag from docker run ( @jfrazelle @LK4D4 )
  • To prevent NDP neighbor cache invalidation issues IPv6 addresses are generated from the container's MAC address when the subnet has a minimum size of /80. The last 6 bytes simply are the MAC address.
  • I have added graphics to improve the docs and added some basic documentation about IPv6. Lots of people do not know the IPv6 basics so I thought it is a good idea to give a short introduction. But I am not sure if you want to have this IPv6 introduction under "Advanced Networking"!? Is it off-topic? You can find a compiled version of the docs on my server: http://docker-docs.janduda.net/articles/networking/ ( @SvenDowideit @jamtur01 @thaJeztah @fredlf )

I have a problem checking if the IPv6 routes are set correctly. The libcontainer method just supports getting IPv4 routes. I have to extend this functionality before I can add the IPv6 route check feature.

Have a nice sunday
Malte

@jamtur01 jamtur01 commented on the diff Nov 30, 2014
docs/sources/articles/networking.md
+
+- Route all traffic to `2000::/64` via `eth0`
+- Route all traffic to `2002::/64` via `docker0`
+- Route all traffic to `2001::/64` via Host1 with IP `2000::1`
+
+The difference to Host1 is that the network `2002::/64` is directly attached to
+the host via its `docker0` interface whereas it reaches `2001::/64` via Host1's
+IPv6 address `2000::1`.
+
+This way every container is able to contact every other container. The
+containers `Container1-*` share the same subnet and contact each other directly.
+The traffic between `Container1-*` and `Container2-*` will be routed via Host1
+and Host2 because those containers do not share the same subnet.
+
+In a switched environment every host has to know all routes to every subnet. You
+always have to update the hosts' routing tables once you add or remove a host
@jamtur01
jamtur01 Nov 30, 2014 Contributor

host's

@MalteJ
MalteJ Nov 30, 2014 Contributor

should be plural

@jamtur01 jamtur01 and 1 other commented on an outdated diff Nov 30, 2014
docs/sources/articles/networking.md
+- Route all traffic to `2001::/64` via Host1 with IP `2000::1`
+
+The difference to Host1 is that the network `2002::/64` is directly attached to
+the host via its `docker0` interface whereas it reaches `2001::/64` via Host1's
+IPv6 address `2000::1`.
+
+This way every container is able to contact every other container. The
+containers `Container1-*` share the same subnet and contact each other directly.
+The traffic between `Container1-*` and `Container2-*` will be routed via Host1
+and Host2 because those containers do not share the same subnet.
+
+In a switched environment every host has to know all routes to every subnet. You
+always have to update the hosts' routing tables once you add or remove a host
+to the cluster.
+
+Every configuration in the diagram that is visualized below the cyan colored
@jamtur01
jamtur01 Nov 30, 2014 Contributor

I'd make it clear which line - some people can't see cyan. :)

@MalteJ
MalteJ Nov 30, 2014 Contributor

done

@jamtur01 jamtur01 commented on an outdated diff Nov 30, 2014
docs/sources/articles/networking.md
+
+![](/article-img/ipv6_routed_network_example.svg)
+
+In this scenario containers of the same host can communicate directly with each
+other. The traffic between containers on different hosts will be routed via
+their hosts and the router. For example packet from `Container1-1` to
+`Container2-1` will be routed through `Host1`, `Router` and `Host2` until it
+arrives at `Container2-1`.
+
+To keep the IPv6 addresses short in this example a `/48` network is assigned to
+every host. The hosts use a `/64` subnet of this for its own services and one
+for Docker. When adding a third host you would add a route for the subnet
+`2001:db8:3::/48` in the router and configure Docker on Host3 with
+`--fixed-cidr-v6=2001:db8:3:1::/64`.
+
+Remember he subnet for Docker containers should at least have a size of `/80`.
@jamtur01
jamtur01 Nov 30, 2014 Contributor

he == the?

@jamtur01 jamtur01 and 1 other commented on an outdated diff Nov 30, 2014
docs/sources/articles/networking.md
+`Container2-1` will be routed through `Host1`, `Router` and `Host2` until it
+arrives at `Container2-1`.
+
+To keep the IPv6 addresses short in this example a `/48` network is assigned to
+every host. The hosts use a `/64` subnet of this for its own services and one
+for Docker. When adding a third host you would add a route for the subnet
+`2001:db8:3::/48` in the router and configure Docker on Host3 with
+`--fixed-cidr-v6=2001:db8:3:1::/64`.
+
+Remember he subnet for Docker containers should at least have a size of `/80`.
+This way an IPv6 address can end with the container's MAC address and you
+prevent ARP cache invalidation issues in the Docker layer. So if you have a
+`/64` for your whole environment use `/68` subnets for the hosts and `/80` for
+the containers. This way you can use 4096 hosts with 16 `/80` subnets each.
+
+Every configuration in the diagram that is visualized below the cyan colored
@jamtur01
jamtur01 Nov 30, 2014 Contributor

As above.

@MalteJ
MalteJ Nov 30, 2014 Contributor

done

@MalteJ
Contributor
MalteJ commented Nov 30, 2014

@jamtur01 thank you :)

@thaJeztah thaJeztah and 1 other commented on an outdated diff Nov 30, 2014
docs/sources/articles/networking.md
+`ff00::/8` are globally routable unicast addresses. In fact currently only
+`2000::/3` (`2000...` to `3fff...`) addresses are used for end-to-end IPv6
+global unicast communication.
+
+The subnet `2001:db8::/32` is reserved for documentation and experimental use.
+Feel free to setup a network within this subnet if you do not have got an own
+subnet.
+
+#### Unique Local Unicast Addresses
+For private addresses the subnet `fc00::/7` (`fc00...` to `fdff...`) is used.
+It is routable within your private network but should not be routed into the
+global space. Before you use it and for more information have a look at
+[RFC 4193](http://tools.ietf.org/html/rfc4193).
+
+#### NAT
+You should not use NAT in IPv6. It is be possible but be strongly discouraged to
@thaJeztah
thaJeztah Nov 30, 2014 Member

Something wrong with this sentence; "It is be possible but be strongly discouraged.."

Remove first "be", second one.
Not sure about "be discouraged", I'm more familiar with "it is discouraged" (but I'm no native English speaker!)

@MalteJ
MalteJ Nov 30, 2014 Contributor

you are right, the first "be" doesn't belong there.
I am not sure about the "be discouraged". It is an imparative but I don't know if I have used it correctly in this context.

@thaJeztah
Member

Wow, Malte! Really good stuff. I read the section on IPv6 and could understand most things (not very familiar with IPv6 yet, so I think this is a very welcome addition).

I didn't find big issues, but maybe one of the "real" doc-writers wants to tweak it a bit after merging just to make some sections a bit more "fluent".

Regarding the cyan line; even for people who can see cyan, I think it's a bit hard to see "at a glance". I think this could be tweaked/styled a bit as well,. (Different color? Have the line actually "split" the diagram?). I don't think this is urgent and perhaps there's an actual designer in the community that can make these changes.

Thanks a LOT for this.

@MalteJ
Contributor
MalteJ commented Nov 30, 2014

@thaJeztah thanks for the compliment. Nice to hear they have helped you :)
I have replaced the cyan dashed line by a black dashed line.
And yeah, I'd be glad if someone would revise my docs as I am not a native English speaker.

@thaJeztah
Member

@MalteJ hm not sure black in stead of cyan is making it stand out better, LOL. Erm, don't have a good suggestion at this moment, perhaps someone else has a bright idea :)

@MalteJ
Contributor
MalteJ commented Nov 30, 2014

@thaJeztah sure. But at least those who can't see cyan can see it now :D
I mean, it is not the most important part of this diagram. So it doesn't need to be the thing that you see first.

@jamtur01
Contributor

My point was I know at least two or three people in the community who are color-blind - describe the link/line rather than the color I guess.

@fredlf fredlf commented on an outdated diff Dec 2, 2014
docs/sources/articles/networking.md
@@ -363,6 +369,247 @@ Again, this topic is covered without all of these low-level networking
details in the [Docker User Guide](/userguide/dockerlinks/) document if you
would like to use that as your port redirection reference instead.
+## IPv6
+
+<a name="ipv6"></a>
+
+### Basics
+As we are [running out of IPv4 addresses](http://en.wikipedia.org/wiki/IPv4_address_exhaustion)
+the IETF has standardized an IPv4 successor, Internet Protocol Version 6, in
+[RFC 2460](https://www.ietf.org/rfc/rfc2460.txt). Both protocols, IPv4 and IPv6,
+reside on layer 3 of the [OSI model](http://en.wikipedia.org/wiki/OSI_model).
+
+#### Addresses and Subnets
+Apparently the most significant change is the IP address length. Where in IPv4
+the addresses have a length of 4 bytes (32 bit), e.g. `203.0.113.42`,
@fredlf
fredlf Dec 2, 2014 Contributor

Comma after "e.g.," please. Throughout.

@fredlf fredlf commented on an outdated diff Dec 2, 2014
docs/sources/articles/networking.md
+
+<a name="ipv6"></a>
+
+### Basics
+As we are [running out of IPv4 addresses](http://en.wikipedia.org/wiki/IPv4_address_exhaustion)
+the IETF has standardized an IPv4 successor, Internet Protocol Version 6, in
+[RFC 2460](https://www.ietf.org/rfc/rfc2460.txt). Both protocols, IPv4 and IPv6,
+reside on layer 3 of the [OSI model](http://en.wikipedia.org/wiki/OSI_model).
+
+#### Addresses and Subnets
+Apparently the most significant change is the IP address length. Where in IPv4
+the addresses have a length of 4 bytes (32 bit), e.g. `203.0.113.42`,
+IPv6 addresses have a length of 16 bytes (128 bits), e.g.
+`2001:0db8:0000:0000:0000:0000:0030:9a43`. Hence the theoretical number of IP
+addresses has raised from 4,294,967,296 (4.29e+9) to 3.40e+38. Due to
+conventions, restrictions and reserved blocks the number of usable IPv4
@fredlf
fredlf Dec 2, 2014 Contributor

comma needed after "blocks"

@fredlf fredlf commented on an outdated diff Dec 2, 2014
docs/sources/articles/networking.md
+- Route all traffic to `2001::/64` via Host1 with IP `2000::1`
+
+The difference to Host1 is that the network `2002::/64` is directly attached to
+the host via its `docker0` interface whereas it reaches `2001::/64` via Host1's
+IPv6 address `2000::1`.
+
+This way every container is able to contact every other container. The
+containers `Container1-*` share the same subnet and contact each other directly.
+The traffic between `Container1-*` and `Container2-*` will be routed via Host1
+and Host2 because those containers do not share the same subnet.
+
+In a switched environment every host has to know all routes to every subnet. You
+always have to update the hosts' routing tables once you add or remove a host
+to the cluster.
+
+Every configuration in the diagram that is visualized below the dashed line is
@fredlf
fredlf Dec 2, 2014 Contributor

that is visualized > shown

@fredlf fredlf commented on an outdated diff Dec 2, 2014
docs/sources/articles/networking.md
@@ -363,6 +369,247 @@ Again, this topic is covered without all of these low-level networking
details in the [Docker User Guide](/userguide/dockerlinks/) document if you
would like to use that as your port redirection reference instead.
+## IPv6
+
+<a name="ipv6"></a>
+
+### Basics
+As we are [running out of IPv4 addresses](http://en.wikipedia.org/wiki/IPv4_address_exhaustion)
+the IETF has standardized an IPv4 successor, Internet Protocol Version 6, in
+[RFC 2460](https://www.ietf.org/rfc/rfc2460.txt). Both protocols, IPv4 and IPv6,
+reside on layer 3 of the [OSI model](http://en.wikipedia.org/wiki/OSI_model).
+
+#### Addresses and Subnets
+Apparently the most significant change is the IP address length. Where in IPv4
@fredlf
fredlf Dec 2, 2014 Contributor

Omit "Apparently". Recast to: One significant difference between IPv4 and IPv6 is the length of an IPv6 address.

@fredlf fredlf commented on an outdated diff Dec 2, 2014
docs/sources/articles/networking.md
@@ -363,6 +369,247 @@ Again, this topic is covered without all of these low-level networking
details in the [Docker User Guide](/userguide/dockerlinks/) document if you
would like to use that as your port redirection reference instead.
+## IPv6
+
+<a name="ipv6"></a>
+
+### Basics
@fredlf
fredlf Dec 2, 2014 Contributor

I think this whole introductory section can be cut down and shortened. It's not really the job of Docker docs to explain what IPv6 is. I'd prefer to see an introductory sentence or two and then the links to resources for learning more (wikipedia, IETF spec, etc.).

@fredlf fredlf commented on an outdated diff Dec 2, 2014
docs/sources/articles/networking.md
+ example.com. 175 IN AAAA 2001:db8::1
+ example.com. 175 IN A 203.0.113.1
+
+An IPv6 address reverse lookup works similar to an IPv4 address reverse lookup.
+Instead of using the domain `in-addr.arpa` IPv6 lookups use `ip6.arpa`. Further
+the address is in hexadecimal format and every subdomain has a length of one
+character (4 bits of the address):
+
+ 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa. IN PTR example.com.
+
+You may add A and AAAA-records to a hostname at the same time. IPv4 clients will
+query the A-record and IPv6 clients the AAAA-record. If your client supports
+IPv4 and IPv6 it is up to the client which to use. Most modern operating systems
+prefer IPv6 by default.
+
+
@fredlf
fredlf Dec 2, 2014 Contributor

Again, I'm not sure we need all the explanatory/introductory material above. This is Docker documentation, not a tutorial in IPv6 conventions.

@fredlf
Contributor
fredlf commented Dec 2, 2014

Docs LGTM for now. I will need to go back and edit/clean things up after this gets merged. It needs quite a bit of polish and it would take too long to comment it all out.

@MalteJ
Contributor
MalteJ commented Dec 2, 2014

@fredlf thanks for your feedback and sorry my docs do not really pass your QA.
So we are removing the Basics part? Just wanted to help the people to get up to date. Yeah, I have noticed it got a bit too long ;)

@fredlf
Contributor
fredlf commented Dec 2, 2014

Oh no worries, @MalteJ. It just needs some cleaning up which is easier to do after the fact. Yes, I think we should remove the majority of the Basics section because it's not really specific or germane to Docker per se. Just a sentence or two summary plus the links to more info should suffice.

Thanks for all the work, it's much appreciated.

@jessfraz jessfraz added this to the next milestone Dec 4, 2014
@flecno flecno referenced this pull request in kylemanna/docker-openvpn Dec 8, 2014
Open

IPv6 support #20

@jessfraz
Contributor

awesome trying this one out again thanks for your patience @MalteJ

@jessfraz jessfraz self-assigned this Dec 12, 2014
@SvenDowideit
Collaborator

much awesomeness - Docs still LGTM

@jessfraz
Contributor

ok I think this is good now ping @LK4D4

@MalteJ
Contributor
MalteJ commented Dec 16, 2014

fine :)

just tell me when it's time to rebase

@jessfraz jessfraz modified the milestone: 1.5.0, next Dec 16, 2014
@MalteJ MalteJ referenced this pull request in kubernetes/kubernetes Dec 23, 2014
Open

Support and/or exploit ipv6 #1443

@shykes
Contributor
shykes commented Dec 30, 2014

Hi @MalteJ, first of all thanks for this contribution! A lot of people are very impatient to see ipv6 support in Docker, and I think if we do this right, we could leverage the Docker install base to make a real difference in IPV6 adoption world-wide. That would be pretty cool!

A few notes:

1: we want IPv6 support in Docker, and very much want to merge this.

2: you should be aware that there is a significant upgrade of Docker's networking stack underway, which might affect this code in the future. However, that doesn't mean we shouldn't merge this: it's just a factor to keep in mind.

3: we are actively looking for help with this upgrade. If you are interested we would love your help! @erikh is leading the effort, I recommend joining #docker-network on IRC/Freenode to get started.

4: before we merge this, we need to agree on the design implications of IPv6 support in Docker. There are 2 major implications: UX and the container's networking model (which we call CNM).

5: about UX: I think --ipv6 and --fixed-cidr-ipv6 are reasonable daemon flags. In the future we move move --fixed-cidr-ipv6 out of the core, and into a network driver configuration (different drivers might handle IP address allocation differently). But I think --ipv6 could remain in the core. Before we confirm this decision, I would like to get the opinion of @erikh @LK4D4 @unclejack @tianon @jfrazelle and other core maintainers who are interested.

6: about CNM. What should be the contract between the Docker runtime and the application regarding IPv6 connectivity? Will some application only work with ipv6 connectivity? If so, do we need to update the image manifest to carry this information? Are there other aspects of IPv6 connectivity which may affect the behavior of my application of deployment tools after we merge this? If so, what are they? Also interested in ideas and opinions from @erikh @LK4D4 @unclejack et al.

7: last question: will you be able to invest reasonable time to maintain this code in the future?

Thanks for reading, and thanks again for the contribution!

@shykes
Contributor
shykes commented Dec 30, 2014

One more note: if we get satisfactory answers to the questions above, then I think we should go ahead and merge, without blocking on the larger upgrade of the networking stack. If we find a design that we like, and take the time to think about implications for the users, then it's worth shipping it now, even if much of the underlying code will be refactored later.

I think this is good practice in general, we should avoid blocking small-to-medium useful contributions because of a huge long-running feature branch. It's frustrating for aspiring contributors and puts unnecessary pressure on the owners of the feature branch. It's better to encourage everyone to break down their patches and get them merged in little pieces as early as possible.

@jessfraz
Contributor

#5 SGTM, I like this patch it is small and awesome

@erikh
Contributor
erikh commented Dec 30, 2014

I'm just going to comment on this from the perspective of, "this will be replaced soon", which is what I think will happen given the internal networking effort.

But before that, I want to explicitly communicate that it wasn't my intent to sideline this patch. It just seems we're colliding on what code this touches. I hope no offense is felt, it certainly is not implied.

My main concern here is with the dual stack network interfaces. The problem with this (as others may have mentioned) is that ipv6 is still (yes, really) not available everywhere.

The new networking code makes zero assumptions (well, in a few cases it does, but we're getting rid of those) about the IP addressing mechanism when configuring an interface. It attempts to understand the kind of address it's working with (by checking if it's a valid v4 address first).

I would like to keep the model of "one interface, one network" and the dual stack approach defies that. This adds a lot of additional complexity for the goal of keeping one interface, which will no longer be a requirement of docker containers in the upcoming patches.

Merging this patch would require us to completely re-think how allocation is handled and interfaces are configured, in the NEW code. I'm not really ready to accept that this is something we need now that you can have containers create, join, and leave any network (with IPv4 or IPv6), or several of them.

It also has significant impact on the interfaces that we've been designing: https://github.com/docker/docker/pull/8947/files#diff-2fb7c74bdba3137445a0c8ca02bba557R86 for example is going to have to be catered to, where we currently just deal with net.IP and net.IPNet for most operations, which is sufficiently portable to achieve what I explained above.

I guess the bigger question would be if we should explicitly support dual-stack interfaces, and how urgently is that needed, e.g., in lieu of supporting multiple networks that are ipv4 or ipv6 enabled independently?

Anyone with strong opinions?

@shykes
Contributor
shykes commented Dec 30, 2014

@erikh I think a good starting point to talk about the planned upgrade, would be to explain how you currently plan on handling IPv6. Then we can compare the approach proposed here, with the approach proposed in the upgrade.

@tianon
Member
tianon commented Jan 3, 2015

Explicit IPv6 support or not, we're not going to get away without dual-stack support in some shape or form (as in, we already have dual stack today, because that's literally how IPv6 is designed to be implemented), and I think it's incredibly wasteful to create a second virtual ethernet device just for an explicitly Docker-added IPv6 address:

$ docker run -it --rm debian ip -6 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
4017: eth0: <NO-CARRIER,BROADCAST,UP,LOWER_UP> mtu 1500 
    inet6 fe80::42:acff:fe12:7d/64 scope link tentative 
       valid_lft forever preferred_lft forever

(ie, we're already dual-stack today -- it just isn't easy to use because Docker isn't explicitly aware of it / helping manage it or extract useful information about it, like providing easy access to that link-local address)

More specifically speaking to @shykes's points above, I'm +1 on IPv6, including and especially in dual-stack mode, and I think the new --fixed-cidr-ipv6 flag described is good enough for now while we work out more advanced networking (especially since it has nice parity with the existing IPv4 CIDR flag). As an active IPv6 user (on IPv6 connections that go up and down regularly while IPv4 is unaffected, so I get to see a lot of the rougher edges of connection handling first-hand), this sounds awesome, and is one of the biggest things I've been wanting to see Docker doing more of since I started working with/on the project.

Maybe at some point in the much farther future we can debate about when it would be appropriate to change the default of --ipv6 for the daemon to true. 👍

(As an interesting and related side-note, "Mac OS X has included support for IPv6 since Mac OS X v10.1. It's enabled it by default since Mac OS X v10.3.")

@MalteJ
Contributor
MalteJ commented Jan 3, 2015

+1 for the dual-stack approach.
I would like to keep the OSI Layers separated and don't create artificial dependencies between them. We should be able to use one ethernet device for multiple protocols and multiple IP addresses.
Further IPv6 compliance requires the support of multiple IPv6 addresses per device. This PR only supports one link-local and one global IPv6 address per device/container. In the future this should be extended to support multiple IPv6 addresses. Why do you need multiple IPv6 addresses per container? Think of SSL encryption. You need one IP address per certificate/CN. SNI tries to solve this but is probably only used for https.
If the new architecture limits ourselves to one IP version and one IP address per device we would prevent Docker to become IPv6 compliant anytime soon.

Docker could be a huge motivation to support IPv6 in the cloud. The routed networking model - like Kubernetes is using - fits perfectly with Docker and IPv6.
In lack of v4 addresses here in Germany the first cable ISPs provide their customers a /64 IPv6 subnet and only a NATed IPv4 address. Amazon has a similar problem. They introduced VPCs (virtual private clouds) to reduce the number of public IPv4 addresses they have to provide. Now just the internet facing VMs have public addresses. The others are NATed.
Btw NAT is bad: slow, expensive and falsely implies security ;-)
If we create a demand for IPv6 support the IaaS providers will implement it. At the moment they say, yeah, it's somewhere on the roadmap but we don't really have demand for that so we are not working on it.

@LK4D4
Contributor
LK4D4 commented Jan 5, 2015

@MalteJ now you have merging commit in your branch. Would you mind to rebase instead of merge? I think we need this merge before 1.5.

@MalteJ
Contributor
MalteJ commented Jan 6, 2015

rebased @LK4D4 @jfrazelle

@jessfraz
Contributor
jessfraz commented Jan 6, 2015

yayyy! so stoked to try it out!

@jessfraz
Contributor
jessfraz commented Jan 6, 2015

LGTM

@jessfraz
Contributor
jessfraz commented Jan 6, 2015

happening

@MalteJ
Contributor
MalteJ commented Jan 6, 2015

:D

@LK4D4
Contributor
LK4D4 commented Jan 6, 2015

I'll try tomorrow too

@tianon tianon commented on an outdated diff Jan 6, 2015
daemon/networkdriver/bridge/driver.go
if err != nil {
+ // No Bridge existant. Create one
@tianon
tianon Jan 6, 2015 Member

existant > existent

@tianon tianon commented on the diff Jan 6, 2015
daemon/networkdriver/bridge/driver.go
@@ -140,6 +194,16 @@ func InitDriver(job *engine.Job) engine.Status {
if err := ioutil.WriteFile("/proc/sys/net/ipv4/ip_forward", []byte{'1', '\n'}, 0644); err != nil {
job.Logf("WARNING: unable to enable IPv4 forwarding: %s\n", err)
}
+
+ if enableIPv6 {
+ // Enable IPv6 forwarding
+ if err := ioutil.WriteFile("/proc/sys/net/ipv6/conf/default/forwarding", []byte{'1', '\n'}, 0644); err != nil {
+ job.Logf("WARNING: unable to enable IPv6 default forwarding: %s\n", err)
+ }
+ if err := ioutil.WriteFile("/proc/sys/net/ipv6/conf/all/forwarding", []byte{'1', '\n'}, 0644); err != nil {
@tianon
tianon Jan 6, 2015 Member

Does it really make sense to enable both of these globally for all interfaces?

@MalteJ
MalteJ Jan 6, 2015 Contributor

For IPv4 it's the same.
You can manage it manually by setting -ip-forward=false

@tianon
tianon Jan 6, 2015 Member

I don't know what document I was reading last time I looked into this, but I guess this is reasonably sane from what I can find now: (from http://www.tldp.org/HOWTO/Linux+IPv6-HOWTO/proc-sys-net-ipv6..html#AEN1501)

Note: It is recommended to have the same setting on all interfaces; mixed router/host scenarios are rather uncommon.

@tianon
tianon Jan 6, 2015 Member

Oh wait, this is the bit that concerned me. This could actually break people's existing IPv6 setups:

  • Value TRUE: If local forwarding is enabled, Router behaviour is assumed. This means exactly the reverse from the above:
    1. Router Advertisements are ignored.
@tianon
tianon Jan 6, 2015 Member

(and by could, I mean that this PR as-is will break every deployment of IPv6 I've ever done on my local networks because of these two settings)

@tianon
tianon Jan 6, 2015 Member

It's not the same. IPv4 doesn't have router advertisements -- it usually uses DHCP. DHCPv6 is a lot less common, because it requires more setup. Router advertisements require zero configuration on any of the nodes -- it "just works", so it's the more common network setup solution.

@MalteJ
MalteJ Jan 6, 2015 Contributor

To describe the current behavior:
Router advertisement sets up a /128 IPv6 address for the host - let's say on eth0.
If you want to use Docker with IPv6 your network router should be set up to route a /64 to your host's eth0. Then you can assign an IPv6 address from this range manually to your eth0. Router advertisement does not work for this scenario.
If you have two ethernet devices, eth0 with a /64 subnet and eth1 with an /128 address assigned via RA, yes, this would be a problem. Docker would prevent eth1 to get an address via RA when started with the default -ip-forward=true.
If you have just one eth0 and want to use RA and a /128 address you cannot route IPv6 traffic to your virtual docker network. So you would have to start Docker with -ip-forward=false.
There is no IPv6 NAT.

@MalteJ
MalteJ Jan 6, 2015 Contributor

Your Docker host will effectively become a Router.

@tianon
tianon Jan 6, 2015 Member

Ok, so let me paint this picture a different way (from the perspective of a prospective user):

  1. user has IPv6; they don't like configuration hassle, so they go the easy route and setup RA
  2. user now goes to http://whatismyv6.com and sees IPv6, knows they have IPv6, are happy 🎉
  3. user starts using Docker and thinks "wow, this is cool, I wish it could use my IPv6!"
  4. user notices that it has a toggle on the daemon to enable IPv6, so of course they go turn it on immediately, and they even hand it a fixed-cidr because they have one -- it's the one their router is handing out via RA
  5. user no longer has IPv6 on their computer thanks to RA being disabled by Docker
@MalteJ
MalteJ Jan 6, 2015 Contributor

yeah, I get the problem.

If you want to realize the user flow you have sketched you would need to bridge the Docker containers into the Ethernet network. They have to be on the same ethernet link.

@tianon
tianon Jan 6, 2015 Member

Right. So I guess what I'm really trying to get at is that we need to document this carefully, and maybe even note on --ipv6 that it means you shouldn't enable this unless you're prepared to setup your host as an IPv6 router.

@tianon
tianon Jan 6, 2015 Member

(because I would've fallen into exactly this trap, and I can guarantee I won't be alone; following that, we'll get bug reports that "Docker broke my IPv6!")

@MalteJ
MalteJ Jan 6, 2015 Contributor

OK

@tianon
tianon Jan 6, 2015 Member

I also mention this because in reading the docs about IPv6 that are included here, I don't really see an explanation about how a user like me (whose setup is likely going to be one of the most common in the wild) would set that up, even in general terms.

@m1keil
m1keil commented Jan 6, 2015

Sorry to interrupt you guys, I'm following this PR for a few days while trying to create Docker deployment with IPv6. Two quick notes/ideas:

  1. accept_ra is not a boolean type anymore, it's integer and has additional value - 2 which will turn on RA even if forwarding is on. Maybe that's a possible solution for your potential issue?
  2. If I'm not mistaken, in order for this approach to work, you need to to have a prefix lower than /64 from upstream provider in order to have enough subnets with 64 bit host address space. I think this should be included in the docs?

P.S
Awesome work!

@MalteJ
Contributor
MalteJ commented Jan 7, 2015

@m1keil thanks for no. 1 I will look into that.
Regarding 2:
The Docker containers do not use stateless autoconfig for getting their addresses. The addresses are assigned by the Docker daemon. You can specify the address pool via --fixed-cidr-v6=... Here you may specify any prefix length. It does not need to be a /64. For example you may use a /80 while you have a /64 routed to the host.

@m1keil
m1keil commented Jan 7, 2015

@MalteJ Oh, I see, but unless I'm missing something, don't you need to set NDP proxy in order for this to work?
I mean, if you assign /80 to docker0 you will be able to ping the containers locally, but any traffic from outside the host will not pass since Neighbor Solicitation ICMP6 requests won't be answered.

I'm really sorry if I'm confusing something, thank you for your time!

@MalteJ
Contributor
MalteJ commented Jan 7, 2015

@m1keil every Docker network on every host is a separate ethernet link. They do not share the same subnet and no Neighbor Solicitation gets through. All traffic has to be routed.
Have a look at the diagrams in the IPv6 docs. Maybe it will become clear then :-)
http://docker-docs.janduda.net/articles/networking/#ipv6

@m1keil
m1keil commented Jan 7, 2015

Ahh yes of course, it's not same subnet.. (That's the Digital Ocean
approach that confused me ;-) ) thanks for clearing this up!
On Jan 7, 2015 2:32 AM, "Malte Janduda" notifications@github.com wrote:

@m1keil https://github.com/m1keil every Docker network on every host is
a separate ethernet link. They do not share the same subnet and no Neighbor
Solicitation gets through.
Have a look at the diagrams in the IPv6 docs. Maybe it will become clear
then :-)
http://docker-docs.janduda.net/articles/networking/#ipv6


Reply to this email directly or view it on GitHub
#8947 (comment).

@MalteJ
Contributor
MalteJ commented Jan 7, 2015

You are welcome :-)

@MalteJ
Contributor
MalteJ commented Jan 8, 2015

OK, I have tweaked the docs a bit to give some info about the accept_ra issue.

@LK4D4
Contributor
LK4D4 commented Jan 8, 2015

I tried also to run tests with daemon --ipv6 --fixed-cidr-v6=2a00:1450::/64.
All was Okay. I think we should run test daemon with this options and write tests for ipv6 after merge.

@LK4D4 LK4D4 commented on an outdated diff Jan 8, 2015
daemon/config.go
@@ -55,9 +57,11 @@ func (config *Config) InstallFlags() {
flag.BoolVar(&config.EnableIptables, []string{"#iptables", "-iptables"}, true, "Enable Docker's addition of iptables rules")
flag.BoolVar(&config.EnableIpForward, []string{"#ip-forward", "-ip-forward"}, true, "Enable net.ipv4.ip_forward")
@LK4D4
LK4D4 Jan 8, 2015 Contributor

This is now doing not only this.

@jessfraz
Contributor
jessfraz commented Jan 8, 2015

I can add a ipv6 server into our jenkins matrix as well :)

@tianon
Member
tianon commented Jan 8, 2015

Thanks @MalteJ ❤️

I think the information you've added would give me the appropriate warning (and information) I'd need before I jumped into this head-first. 👍

@LK4D4 LK4D4 and 1 other commented on an outdated diff Jan 8, 2015
docs/sources/reference/commandline/cli.md
@@ -75,6 +76,7 @@ expect an integer, and they can only be specified once.
--ip-forward=true Enable net.ipv4.ip_forward
@LK4D4
LK4D4 Jan 8, 2015 Contributor

Here too pls :)

@MalteJ
MalteJ Jan 8, 2015 Contributor

done

@MalteJ MalteJ Adding IPv6 network support to docker
Signed-off-by: Malte Janduda <mail@janduda.net>
813ff7f
@MalteJ
Contributor
MalteJ commented Jan 8, 2015

OK, merged, rebased and it's green again :)

@LK4D4
Contributor
LK4D4 commented Jan 8, 2015

LGTM

@LK4D4 LK4D4 merged commit 3fbf723 into docker:master Jan 8, 2015

1 check passed

default The build succeeded on drone.io
Details
@thaJeztah
Member

🎉 awesome work @MalteJ !

@MalteJ
Contributor
MalteJ commented Jan 8, 2015

Great to see this PR getting merged.

Thanks to all who participated!
Especially @jfrazelle and @LK4D4

@MalteJ MalteJ referenced this pull request Jan 9, 2015
Closed

Tests required for IPv6 #10001

@else
else commented Jan 9, 2015

👍 🎉

@gissehel gissehel commented on the diff May 14, 2015
daemon/networkdriver/bridge/driver.go
@@ -363,20 +458,34 @@ func generateMacAddr(ip net.IP) net.HardwareAddr {
return hw
}
+func linkLocalIPv6FromMac(mac string) (string, error) {
+ hx := strings.Replace(mac, ":", "", -1)
+ hw, err := hex.DecodeString(hx)
+ if err != nil {
+ return "", errors.New("Could not parse MAC address " + mac)
+ }
+
+ hw[0] ^= 0x2
+
+ return fmt.Sprintf("fe80::%x%x:%xff:fe%x:%x%x/64", hw[0], hw[1], hw[2], hw[3], hw[4], hw[5]), nil
@gissehel
gissehel May 14, 2015

Probable source of the problem #13211: "%x%x" with (0x1,0x2) will return the same ipv6 as (0x0,0x12)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment