Per-interface sysctls #47686

robmry · 2024-04-05T16:35:32Z

- What I did

Closes Per-interface sysctls #47639

Until now it's been possible to set per-interface sysctls using, for example, --sysctl net.ipv6.conf.eth0.accept_ra=2. But, the index in the interface name is allocated serially, and the numbering in a container with more than one interface may change when a container is restarted. The change to make it possible to connect a container to more than one network when it's created increased the ambiguity.

This change adds label com.docker.network.endpoint.sysctls to the DriverOpts in EndpointSettings. This option is explicitly associated with the interface.

Settings in --sysctl for eth0 are migrated to EndpointSettings.DriverOpts.

Because using --sysctl with any interface apart from eth0 would have unpredictable results, it is now an error to use any other interface name in the top level --sysctl option. The error message includes a hint at how to use the new per-interface setting.

The per-endpoint sysctl name has the interface name replaced by the string IFNAME, for example:
net.ipv6.conf.eth0.accept_ra=2
becomes:
net.ipv6.conf.IFNAME.accept_ra=2

(The string ifname is also accepted because, for some reason, the CLI converts names to lower-case.)

The value of DriverOpts["com.docker.network.endpoint.sysctls"] is a comma separated list of these sysctls.

Settings from --sysctl are applied by the runtime lib during task creation. So, task creation fails if the endpoint does not exist. Applying per-endpoint settings during interface configuration means the endpoint can be created later, which paves the way for removal of the SetKey OCI prestart hook.

Unlike other DriverOpts, the sysctl label itself is not driver-specific, but each driver has a chance to check settings/values and raise an error if a setting would cause it a problem - no such checks have been added in this initial version. As a future extension, if required, it would be possible for the driver to echo back valid/extended/modified settings to libnetwork for it to apply to the interface. (At that point, the syntax for the options could become driver specific to allow, for example, a driver to create more than one interface.)

- How I did it

migrate per-endpoint sysctls from the top-level into per-interface DriverOpts
pass those per-interface sysctls to the osl.Network, along with other config values for the interface
apply those sysctls during interface configuration (which currently still happens during the SetKey prestart callback, but needn't)

- How to verify it

New unit and integration tests.

And ...

Migration of one or two top level --sysctl settings ...

# docker run --rm -ti --name c1 --network mynet --sysctl=net.ipv6.conf.eth0.accept_ra=2 alpine
WARNING: Migrated sysctl "net.ipv6.conf.eth0.accept_ra" to DriverOpts{"com.docker.network.endpoint.sysctls":"net.ipv6.conf.IFNAME.accept_ra=2"}.
/ #

# docker run --rm -ti --name c1 --network mynet --sysctl=net.ipv6.conf.eth0.accept_ra=2 --sysctl=net.ipv6.conf.eth0.forwarding=1 alpine
WARNING: Migrated sysctl "net.ipv6.conf.eth0.accept_ra,net.ipv6.conf.eth0.forwarding" to DriverOpts{"com.docker.network.endpoint.sysctls":"net.ipv6.conf.IFNAME.accept_ra=2,net.ipv6.conf.IFNAME.forwarding=1"}.
/ #

Inspect output ...

# docker run --rm -ti --name c1 --network mynet --sysctl=net.ipv6.conf.eth0.accept_ra=2 --sysctl=net.ipv6.conf.eth0.forwarding=1 --sysctl=net.ipv6.conf.default.disable_ipv6=0 alpine
...

# docker inspect c1
[
    {
        ...
        "HostConfig": {
            ...
            "Sysctls": {
                "net.ipv6.conf.default.disable_ipv6": "0"
            },
        ...
        "NetworkSettings": {
             ...
            "Networks": {
                "mynet": {
                ...
                    "DriverOpts": {
                        "com.docker.network.endpoint.sysctls": "net.ipv6.conf.IFNAME.accept_ra=2,net.ipv6.conf.IFNAME.forwarding=1"
                    },
                ...
]

No migration for eth1 ...

# docker run --rm -ti --name c1 --network mynet --sysctl=net.ipv6.conf.eth1.accept_ra=2 alpine
docker: Error response from daemon: unable to determine network endpoint for sysctl net.ipv6.conf.eth1.accept_ra, use driver option 'com.docker.network.endpoint.sysctls' to set per-interface sysctls.
See 'docker run --help'.

Attempt to set a per-interface sysctl for an unknown protocol ...

# docker run --rm -ti --name c1 --network mynet --sysctl=net.blah.conf.eth0.input=1 alpine
docker: Error response from daemon: invalid config for network mynet: invalid endpoint settings:
unrecognised network interface sysctl 'net.blah.conf.IFNAME.input=1'; represent 'net.X.Y.ethN.Z=V' as 'net.X.Y.IFNAME.Z=V', 'X' must be 'ipv4', 'ipv6' or 'mpls'.
See 'docker run --help'.

Nonexistent sysctl (very verbose error message at the moment) ...

# docker run --rm -ti --name c1 --network mynet --sysctl=net.ipv6.conf.eth0.foo=1 alpine
WARNING: Migrated sysctl "net.ipv6.conf.eth0.foo" to DriverOpts{"com.docker.network.endpoint.sysctls":"net.ipv6.conf.IFNAME.foo=1"}.
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: failed to add interface veth93624eb to sandbox: /proc/sys/net/ipv6/conf/eth0/foo is not a sysctl file: unknown.

But, a lot of that waffle will go-away once the prestart hook is removed and settings are applied after task creation. It'll be more like ...

# docker run --rm -ti --name c1 --network mynet --sysctl=net.ipv6.conf.eth0.foo=1 alpine
WARNING: Migrated net.ipv6.conf.eth0.foo to DriverOpts{"com.docker.network.endpoint.sysctls":"ipv6.conf.foo=1"}.
docker: Error response from daemon: unable to write to '/proc/sys/net/ipv6/conf/eth0/foo': /proc/sys/net/ipv6/conf/eth0/foo is not a sysctl file: unknown.

- Description for the changelog

Allow sysctls to be set per-interface during container creation and network connection.

api/server/router/container/container_routes.go

akerouanton · 2024-04-15T14:52:20Z

api/server/router/container/container_routes.go

+			// Only try to migrate settings for "eth0", anything else would always
+			// have behaved unpredictably.
+			if spl[3] != "eth0" {
+				return "", fmt.Errorf(`unable to determine network endpoint for sysctl %s, use '--network=name=%s,sysctl=%s' or compose 'driver_opts: "%s":"%s"`,


I think we shouldn't put CLI-specific or Compose-specific remediation steps here -- the API could be called by other tools where those steps won't make any sense.

OTOH CLI error messages sometimes looks cryptic for users not familiar with our API. I think we don't have a plan for 'augmenting' CLI error messages with remediation steps. Maybe that's something we need to discuss.

Yes, it's not great, but I'm not sure how best to improve it.

I think it's quite important that we give good clues about how to specify per-interface sysctls - here, for the migration case below, and for when we refuse to migrate from the top level --sysctl in a future release.

A "for example" might help a little, since CLI and compose are probably the common cases. But, not really.

Maybe the best we can do is just delete the hints, and hope the user's able to find the right section of the docs.

I'm not sure how augmented CLI messages would work, perhaps the API would need to return some token that'd tell the client to explain how to set per-interface sysctls in its world (extended --network syntax for the CLI, or driver-opts in compose)? I'm probably missing the point (?!), but any sort of mechanism like that sounds like a big change that'd have to be out-of-scope here.

I changed the message (and updated the examples in PR description to show it) ... now it only mentions the driver-opt label - and the user will have to figure out how to use it.

But I've also updated the CLI PR docker/cli#4994 to get rid of the --network sysctl= option and document the use of [create|run] --network driver-opt=com.docker.network.endpoint.sysctls=[value] and network connect --driver-opt=com.docker.network.endpoint.sysctls= to set multiple sysctls for an endpoint.

akerouanton · 2024-04-15T14:59:46Z

api/server/router/container/container_routes.go

+		return "", nil
+	}
+
+	// TODO(robmry) - refuse to do the migration, generate an error if API > some-future-version.


Next API version should be fine.

This change should land in release 27.0 (along with re-removing the SetKey hook that requires it). So, we'd want to deprecate per-interface sysctls in --sysctl in 27.0, and remove the auto-migration in 28.0.

The current API version is 1.45, and it will be in the upcoming release 26.1. But, it might change in 27.0? In that case, if we make this code check for API version >1.45, we'll have accidentally removed the auto-migration in release 27.0.

So, it's probably best to raise a new issue to say a version check needs to be added, and mark it for milestone 28.0?

So, it's probably best to raise a new issue to say a version check needs to be added, and mark it for milestone 28.0?

#47651 bumped the API version for v27 but AFAICT there are no API changes committed yet. I'm fine with opening a new ticket for that.

Ok, good - I've restricted the migration to API <= 1.46.

api/server/router/container/container_routes.go

api/server/router/container/container_routes_test.go

corhere · 2024-05-15T16:34:29Z

api/server/router/container/container_routes.go

+	// TODO(robmry) - refuse to do the migration, generate an error if API > some-future-version.
+
+	newDriverOpt := strings.Join(netIfSysctls, ",")
+	warning := fmt.Sprintf(`Migrated %s to DriverOpts{"%s":"%s"}.`,


Suggested change

warning := fmt.Sprintf(`Migrated %s to DriverOpts{"%s":"%s"}.`,

warning := fmt.Sprintf(`Migrated sysctl %q to DriverOpts{%q:%q}.`,

The second example in the description:

# docker run --rm -ti --name c1 --network mynet --sysctl=net.ipv6.conf.eth0.accept_ra=2 --sysctl=net.ipv6.conf.eth0.forwarding=1 alpine WARNING: Migrated net.ipv6.conf.eth0.accept_ra,net.ipv6.conf.eth0.forwarding to DriverOpts{"com.docker.network.endpoint.sysctls":"ipv6.conf.accept_ra=2,ipv6.conf.forwarding=1"}.

Would become:

# docker run --rm -ti --name c1 --network mynet --sysctl=net.ipv6.conf.eth0.accept_ra=2 --sysctl=net.ipv6.conf.eth0.forwarding=1 alpine WARNING: Migrated sysctl "net.ipv6.conf.eth0.accept_ra,net.ipv6.conf.eth0.forwarding" to DriverOpts{"com.docker.network.endpoint.sysctls":"ipv6.conf.accept_ra=2,ipv6.conf.forwarding=1"}.

Maybe that's ok, or would be without the quotes around the list, and ignoring sysctl vs. sysctls.

Another option would be:

# docker run --rm -ti --name c1 --network mynet --sysctl=net.ipv6.conf.eth0.accept_ra=2 --sysctl=net.ipv6.conf.eth0.forwarding=1 alpine WARNING: Migrated sysctl "net.ipv6.conf.eth0.accept_ra" to DriverOpts{"com.docker.network.endpoint.sysctls":"ipv6.conf.accept_ra=2"}. WARNING: Migrated sysctl "net.ipv6.conf.eth0.forwarding" to DriverOpts{"com.docker.network.endpoint.sysctls":"ipv6.conf.forwarding=1"}.

But, that doesn't show the value that really needs to be passed to DriverOpts.

I think it's unambiguous as-is, the named sysctls won't be interpreted as anything but sysctls from the request.

But, could do Migrated sysctl %s to DriverOpts{%q:%q}. if you think it's necessary?

But, could do Migrated sysctl %s to DriverOpts{%q:%q}. if you think it's necessary?

Yes. You are formatting user-supplied garbage, with the expectation for users to copy-paste them back into docker commands. What if there are quote characters in the input strings?

Ah right, I see. Thank you.

daemon/container_operations.go

corhere · 2024-05-15T18:19:00Z

libnetwork/osl/interface_linux.go

+		scPath = append(scPath, sk[2:]...)
+
+		sysPath := filepath.Join(scPath...)
+		errC := make(chan error, 1)


n.InvokeFunc is a synchronous blocking call so there is no need to communicate through a channel.

var errF error f := func() { if err := ...; err != nil { errF = err return } } if err := n.InvokeFunc(f); err != nil { return err } if errF != nil { return err }

Signed-off-by: Rob Murray <rob.murray@docker.com>

api/swagger.yaml

api/server/router/container/container_routes_test.go

api/server/router/container/container_routes.go

corhere

I discovered some interesting prior art for per-iface sysctls: whatever CNI plugin OpenShift ships with accepts the full dotted sysctl name with the token IFNAME used as a placeholder for the interface name. It's an interesting contrast to our shortening scheme. Is our scheme too clever for users to figure out?

corhere · 2024-05-22T21:15:39Z

api/server/router/container/container_routes.go

+	// Append exiting per-endpoint sysctls to the migrated sysctls (give priority
+	// to per-endpoint settings).


Suggested change

// Append exiting per-endpoint sysctls to the migrated sysctls (give priority

// to per-endpoint settings).

// Append existing per-endpoint sysctls to the migrated sysctls (give priority

// to per-endpoint settings).

corhere · 2024-05-22T21:20:10Z

integration/networking/bridge_test.go

+
+	const scName = "net.ipv4.conf.eth0.forwarding"
+	for _, val := range []string{"0", "1"} {
+		t.Run("set to "+val, func(t *testing.T) {


Suggested change

t.Run("set to "+val, func(t *testing.T) {

t.Run("ipv4.conf.forwarding="+val, func(t *testing.T) {

I missed this - but changed it anyway for the new IFNAME schem, slightly-differently ... the CLI converts driver-opts to lower case (it probably shouldn't) - so the test now checks that both IFNAME and ifname are allowed, nesting loops with a structured test name.

corhere · 2024-05-22T21:38:59Z

daemon/container_operations.go

+	if sysctls, ok := epConfig.DriverOpts[netlabel.EndpointSysctls]; ok {
+		for _, sysctl := range strings.Split(sysctls, ",") {
+			scname := strings.SplitN(sysctl, ".", 3)
+			if len(scname) != 3 || (scname[0] != "ipv4" && scname[0] != "ipv6") {


On second thought, we should probably also allow mpls. As MPLS packets can transit Ethernet links, some users (particularly in datacenter and telecom environments) may have use cases for enabling MPLS on e.g. a macvlan iface.

corhere · 2024-05-22T21:51:38Z

libnetwork/osl/interface_linux.go

+		scPath := []string{"/proc/sys/net", sk[0], sk[1], ifName}
+		scPath = append(scPath, sk[2:]...)


Suggested change

scPath := []string{"/proc/sys/net", sk[0], sk[1], ifName}

scPath = append(scPath, sk[2:]...)

scPath := append([]string{"/proc/sys/net", sk[0], sk[1], ifName}, sk[2:]...)

Done, slightly differently, for the IFNAME scheme.

corhere · 2024-05-24T20:16:26Z

Maintainer call: consensus is to go forward with specifying the full sysctl name where an IFNAME token is used as a placeholder for the actual interface name, like how the tuning CNI plugin is configured to apply sysctls. We believe that users will have an easier time applying sysctls when the sysctl name is more closely aligned with the name which would be passed to the sysctl command or listed in kernel docs.

robmry · 2024-05-26T12:27:46Z

Maintainer call: consensus is to go forward with specifying the full sysctl name where an IFNAME token is used as a placeholder for the actual interface name, like how the tuning CNI plugin is configured to apply sysctls. We believe that users will have an easier time applying sysctls when the sysctl name is more closely aligned with the name which would be passed to the sysctl command or listed in kernel docs.

Done.

corhere

The commit message for abf6de3 needs to be updated to reflect the new IFNAME scheme.

robmry · 2024-05-28T08:58:12Z

The commit message for abf6de3 needs to be updated to reflect the new IFNAME scheme.

Thank you - fixed.

akerouanton

One small nit about an outdated comment but otherwise LGTM.

akerouanton · 2024-05-29T05:32:00Z

api/server/router/container/container_routes.go

+	for k, v := range hostConfig.Sysctls {
+		// If the sysctl name matches "net.*.*.eth0.*" ...
+		if spl := strings.SplitN(k, ".", 5); len(spl) == 5 && spl[0] == "net" && strings.HasPrefix(spl[3], "eth") {
+			// Transform the name to the endpoint-specific short form.


Suggested change

// Transform the name to the endpoint-specific short form.

Removed - thank you.

Until now it's been possible to set per-interface sysctls using, for example, '--sysctl net.ipv6.conf.eth0.accept_ra=2'. But, the index in the interface name is allocated serially, and the numbering in a container with more than one interface may change when a container is restarted. The change to make it possible to connect a container to more than one network when it's created increased the ambiguity. This change adds label "com.docker.network.endpoint.sysctls" to the DriverOpts in EndpointSettings. This option is explicitly associated with the interface. Settings in "--sysctl" for "eth0" are migrated to DriverOpts. Because using "--sysctl" with any interface apart from "eth0" would have unpredictable results, it is now an error to use any other interface name in the top level "--sysctl" option. The error message includes a hint at how to use the new per-interface setting. The per-endpoint sysctl name has the interface name replaced by "IFNAME". For example: net.ipv6.conf.eth0.accept_ra=2 becomes: net.ipv6.conf.IFNAME.accept_ra=2 The value of DriverOpts["com.docker.network.endpoint.sysctls"] is a comma separated list. Settings from '--sysctl' are applied by the runtime lib during task creation. So, task creation fails if the endpoint does not exist. Applying per-endpoint settings during interface configuration means the endpoint can be created later, which paves the way for removal of the SetKey OCI prestart hook. Unlike other DriverOpts, the sysctl label itself is not driver-specific, but each driver has a chance to check settings/values and raise an error if a setting would cause it a problem - no such checks have been added in this initial version. As a future extension, if required, it would be possible for the driver to echo back valid/extended/modified settings to libnetwork for it to apply to the interface. (At that point, the syntax for the options could become driver specific to allow, for example, a driver to create more than one interface). Signed-off-by: Rob Murray <rob.murray@docker.com>

akerouanton

LGTM. Thanks!

[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [docker/docker](https://togithub.com/docker/docker) | major | `26.1.4` -> `27.0.3` | --- ### Release Notes <details> <summary>docker/docker (docker/docker)</summary> ### [`v27.0.3`](https://togithub.com/moby/moby/releases/tag/v27.0.3) [Compare Source](https://togithub.com/docker/docker/compare/v27.0.2...v27.0.3) #### 27.0.3 For a full list of pull requests and changes in this release, refer to the relevant GitHub milestones: - [docker/cli, 27.0.3 milestone](https://togithub.com/docker/cli/issues?q=is%3Aclosed+milestone%3A27.0.3) - [moby/moby, 27.0.3 milestone](https://togithub.com/moby/moby/issues?q=is%3Aclosed+milestone%3A27.0.3) - Deprecated and removed features, see [Deprecated Features](https://togithub.com/docker/cli/blob/v27.0.3/docs/deprecated.md). - Changes to the Engine API, see [API version history](https://togithub.com/moby/moby/blob/v27.0.3/docs/api/version-history.md). ##### Bug fixes and enhancements - Fix a regression that incorrectly reported a port mapping from a host IPv6 address to an IPv4-only container as an error. [moby/moby#48090](https://togithub.com/moby/moby/pull/48090) - Fix a regression that caused duplicate subnet allocations when creating networks. [moby/moby#48089](https://togithub.com/moby/moby/pull/48089) - Fix a regression resulting in "fail to register layer: failed to Lchown" errors when trying to pull an image with rootless enabled on a system that supports native overlay with user-namespaces. [moby/moby#48086](https://togithub.com/moby/moby/pull/48086) ### [`v27.0.2`](https://togithub.com/moby/moby/releases/tag/v27.0.2) [Compare Source](https://togithub.com/docker/docker/compare/v27.0.1-rc.1...v27.0.2) #### 27.0.2 For a full list of pull requests and changes in this release, refer to the relevant GitHub milestones: - [docker/cli, 27.0.2 milestone](https://togithub.com/docker/cli/issues?q=is%3Aclosed+milestone%3A27.0.2) - [moby/moby, 27.0.2 milestone](https://togithub.com/moby/moby/issues?q=is%3Aclosed+milestone%3A27.0.2) - Deprecated and removed features, see [Deprecated Features](https://togithub.com/docker/cli/blob/v27.0.2/docs/deprecated.md). - Changes to the Engine API, see [API version history](https://togithub.com/moby/moby/blob/v27.0.2/docs/api/version-history.md). ##### Bug fixes and enhancements - Fix a regression that caused port numbers to be ignored when parsing a Docker registry URL. [docker/cli#5197](https://togithub.com/docker/cli/pull/5197), [docker/cli#5198](https://togithub.com/docker/cli/pull/5198) ##### Removed - api/types: deprecate `ContainerJSONBase.Node` field and `ContainerNode` type. These definitions were used by the standalone ("classic") Swarm API, but never implemented in the Docker Engine itself. [moby/moby#48055](https://togithub.com/moby/moby/pull/48055) ### [`v27.0.1`](https://togithub.com/moby/moby/releases/tag/v27.0.1) [Compare Source](https://togithub.com/docker/docker/compare/v26.1.4...v27.0.1-rc.1) #### 27.0.1 For a full list of pull requests and changes in this release, refer to the relevant GitHub milestones: - [docker/cli, 27.0.0 milestone](https://togithub.com/docker/cli/issues?q=is%3Aclosed+milestone%3A27.0.0) - [moby/moby, 27.0.0 milestone](https://togithub.com/moby/moby/issues?q=is%3Aclosed+milestone%3A27.0.0) - Deprecated and removed features, see [Deprecated Features](https://togithub.com/docker/cli/blob/v27.0.1/docs/deprecated.md). - Changes to the Engine API, see [API version history](https://togithub.com/moby/moby/blob/v27.0.1/docs/api/version-history.md). ##### New - containerd image store: Add `--platform` flag to `docker image push` and improve the default behavior when not all platforms of the multi-platform image are available locally. [docker/cli#4984](https://togithub.com/docker/cli/pull/4984), [moby/moby#47679](https://togithub.com/moby/moby/pull/47679) - Add support to `docker stack deploy` for `driver_opts` in a service's networks. [docker/cli#5125](https://togithub.com/docker/cli/pull/5125) - Consider additional `/usr/local/libexec` and `/usr/libexec` paths when looking up the userland proxy binaries by a name with a `docker-` prefix. [moby/moby#47804](https://togithub.com/moby/moby/pull/47804) ##### Bug fixes and enhancements - `*client.Client` instances are now always safe for concurrent use by multiple goroutines. Previously, this could lead to data races when the `WithAPIVersionNegotiation()` option is used. [moby/moby#47961](https://togithub.com/moby/moby/pull/47961) - Fix a bug causing the Docker CLI to leak Unix sockets in `$TMPDIR` in some cases. [docker/cli#5146](https://togithub.com/docker/cli/pull/5146) - Don't ignore a custom seccomp profile when used in conjunction with `--privileged`. [moby/moby#47500](https://togithub.com/moby/moby/pull/47500) - rootless: overlay2: support native overlay diff when using rootless-mode with Linux kernel version 5.11 and later. [moby/moby#47605](https://togithub.com/moby/moby/pull/47605) - Fix the `StartInterval` default value of healthcheck to reflect the documented value of 5s. [moby/moby#47799](https://togithub.com/moby/moby/pull/47799) - Fix `docker save` and `docker load` not ending on the daemon side when the operation was cancelled by the user, for example with <kbd>Ctrl+C</kbd>. [moby/moby#47629](https://togithub.com/moby/moby/pull/47629) - The `StartedAt` property of containers is now recorded before container startup, guaranteeing that the `StartedAt` is always before `FinishedAt`. [moby/moby#47003](https://togithub.com/moby/moby/pull/47003) - The internal DNS resolver used by Windows containers on Windows now forwards requests to external DNS servers by default. This enables `nslookup` to resolve external hostnames. This behaviour can be disabled via `daemon.json`, using `"features": { "windows-dns-proxy": false }`. The configuration option will be removed in a future release. [moby/moby#47826](https://togithub.com/moby/moby/pull/47826) - Print a warning when the CLI does not have permissions to read the configuration file. [docker/cli#5077](https://togithub.com/docker/cli/pull/5077) - Fix a goroutine and file-descriptor leak on container attach. [moby/moby#45052](https://togithub.com/moby/moby/pull/45052) - Clear the networking state of all stopped or dead containers during daemon start-up. [moby/moby#47984](https://togithub.com/moby/moby/pull/47984) - Write volume options JSON atomically to avoid "invalid JSON" errors after system crash. [moby/moby#48034](https://togithub.com/moby/moby/pull/48034) - Allow multiple macvlan networks with the same parent. [moby/moby#47318](https://togithub.com/moby/moby/pull/47318) - Allow BuildKit to be used on Windows daemons that advertise it. [docker/cli#5178](https://togithub.com/docker/cli/pull/5178) ##### Networking - Allow sysctls to be set per-interface during container creation and network connection. [moby/moby#47686](https://togithub.com/moby/moby/pull/47686) - In a future release, this will be the only way to set per-interface sysctl options. For example, on the command line in a `docker run` command,`--network mynet --sysctl net.ipv4.conf.eth0.log_martians=1` will be rejected. Instead, you must use `--network name=mynet,driver-opt=com.docker.network.endpoint.sysctls=net.ipv4.conf.IFNAME.log_martians=1`. ##### IPv6 - `ip6tables` is no longer experimental. You may remove the `experimental` configuration option and continue to use IPv6, if it is not required by any other features. - `ip6tables` is now enabled for Linux bridge networks by default. [moby/moby#47747](https://togithub.com/moby/moby/pull/47747) - This makes IPv4 and IPv6 behaviors consistent with each other, and reduces the risk that IPv6-enabled containers are inadvertently exposed to the network. - There is no impact if you are running Docker Engine with `ip6tables` enabled (new default). - If you are using an IPv6-enabled bridge network without `ip6tables`, this is likely a breaking change. Only published container ports (`-p` or `--publish`) are accessible from outside the Docker bridge network, and outgoing connections masquerade as the host. - To restore the behavior of earlier releases, no `ip6tables` at all, set `"ip6tables": false` in `daemon.json`, or use the CLI option `--ip6tables=false`. Alternatively, leave `ip6tables` enabled, publish ports, and enable direct routing. - With `ip6tables` enabled, if `ip6tables` is not functional on your host, Docker Engine will start but it will not be possible to create an IPv6-enabled network. ##### IPv6 network configuration improvements - A Unique Local Address (ULA) base prefix is automatically added to `default-address-pools` if this parameter wasn't manually configured, or if it contains no IPv6 prefixes. [moby/moby#47853](https://togithub.com/moby/moby/pull/47853) - Prior to this release, to create an IPv6-enabled network it was necessary to use the `--subnet` option to specify an IPv6 subnet, or add IPv6 ranges to `default-address-pools` in `daemon.json`. - Starting in this release, when a bridge network is created with `--ipv6` and no IPv6 subnet is defined by those options, an IPv6 Unique Local Address (ULA) base prefix is used. - The ULA prefix is derived from the Engine host ID such that it's unique across hosts and over time. - IPv6 address pools of any size can now be added to `default-address-pools`. [moby/moby#47768](https://togithub.com/moby/moby/pull/47768) - IPv6 can now be enabled by default on all custom bridge networks using `"default-network-opts": { "bridge": {"com.docker.network.enable_ipv6": "true"}}` in `daemon.json`, or `dockerd --default-network-opt=bridge=com.docker.network.enable_ipv6=true`on the comand line. [moby/moby#47867](https://togithub.com/moby/moby/pull/47867) - Direct routing for IPv6 networks, with `ip6tables` enabled. [moby/moby#47871](https://togithub.com/moby/moby/pull/47871) - Added bridge driver option `com.docker.network.bridge.gateway_mode_ipv6=<nat|routed>`. - The default behavior, `nat`, is unchanged from previous releases running with `ip6tables` enabled. NAT and masquerading rules are set up for each published container port. - When set to `routed`, no NAT or masquerading rules are configured for published ports. This enables direct IPv6 access to the container, if the host's network can route packets for the container's address to the host. Published ports will be opened in the container's firewall. - When a port mapping only applies to `routed` mode, only addresses `0.0.0.0` or `::` are allowed and a host port must not be given. - Note that published container ports, in `nat` or `routed` mode, are accessible from any remote address if routing is set up in the network, unless the Docker host's firewall has additional restrictions. For example: `docker network create --ipv6 -o com.docker.network.bridge.gateway_mode_ipv6=routed mynet`. - The option `com.docker.network.bridge.gateway_mode_ipv4=<nat|routed>` is also available, with the same behavior but for IPv4. - If firewalld is running on the host, Docker creates policy `docker-forwarding` to allow forwarding from any zone to the `docker` zone. This makes it possible to configure a bridge network with a routable IPv6 address, and no NAT or masquerading. [moby/moby#47745](https://togithub.com/moby/moby/pull/47745) - When a port is published with no host port specified, or a host port range is given, the same port will be allocated for IPv4 and IPv6. [moby/moby#47871](https://togithub.com/moby/moby/pull/47871) - For example `-p 80` will result in the same ephemeral port being allocated for `0.0.0.0` and `::`, and `-p 8080-8083:80` will pick the same port from the range for both address families. - Similarly, ports published to specific addresses will be allocated the same port. For example, `-p 127.0.0.1::80 -p '[::1]::80'`. - If no port is available on all required addresses, container creation will fail. - Environment variable `DOCKER_ALLOW_IPV6_ON_IPV4_INTERFACE`, introduced in release 26.1.1, no longer has any effect. [moby/moby#47963](https://togithub.com/moby/moby/pull/47963) - If IPv6 could not be disabled on an interface because of a read-only `/proc/sys/net`, the environment variable allowed the container to start anyway. - In this release, if IPv4 cannot be disabled for an interface, IPv6 can be explicitly enabled for the network simply by using `--ipv6` when creating it. Other workarounds are to configure the OS to disable IPv6 by default on new interfaces, mount `/proc/sys/net` read-write, or use a kernel with no IPv6 support. - For IPv6-enabled bridge networks, do not attempt to replace the bridge's kernel-assigned link local address with `fe80::1`. [moby/moby#47787](https://togithub.com/moby/moby/pull/47787) ##### Removed - Deprecate experimental GraphDriver plugins. [moby/moby#48050](https://togithub.com/moby/moby/pull/48050), [docker/cli#5172](https://togithub.com/docker/cli/pull/5172) - pkg/archive: deprecate `NewTempArchive` and `TempArchive`. These types were only used in tests and will be removed in the next release. [moby/moby#48002](https://togithub.com/moby/moby/pull/48002) - pkg/archive: deprecate `CanonicalTarNameForPath` [moby/moby#48001](https://togithub.com/moby/moby/pull/48001) - Deprecate pkg/dmesg. This package was no longer used, and will be removed in the next release. [moby/moby#47999](https://togithub.com/moby/moby/pull/47999) - Deprecate `pkg/stringid.ValidateID` and `pkg/stringid.IsShortID` [moby/moby#47995](https://togithub.com/moby/moby/pull/47995) - runconfig: deprecate `SetDefaultNetModeIfBlank` and move `ContainerConfigWrapper` to `api/types/container` [moby/moby#48007](https://togithub.com/moby/moby/pull/48007) - runconfig: deprecate `DefaultDaemonNetworkMode` and move to `daemon/network` [moby/moby#48008](https://togithub.com/moby/moby/pull/48008) - runconfig: deprecate `opts.ConvertKVStringsToMap`. This utility is no longer used, and will be removed in the next release. [moby/moby#48016](https://togithub.com/moby/moby/pull/48016) - runconfig: deprecate `IsPreDefinedNetwork`. [moby/moby#48011](https://togithub.com/moby/moby/pull/48011) ##### API - containerd image store: `POST /images/{name}/push` now supports a `platform` parameter (JSON encoded OCI Platform type) that allows selecting a specific platform-manifest from the multi-platform image. This is experimental and may change in future API versions. [moby/moby#47679](https://togithub.com/moby/moby/pull/47679) - `POST /services/create` and `POST /services/{id}/update` now support `OomScoreAdj`. [moby/moby#47950](https://togithub.com/moby/moby/pull/47950) - `ContainerList` api returns container annotations. [moby/moby#47866](https://togithub.com/moby/moby/pull/47866) - `POST /containers/create` and `POST /services/create` now take `Options` as part of `HostConfig.Mounts.TmpfsOptions` allowing to set options for tmpfs mounts. [moby/moby#46809](https://togithub.com/moby/moby/pull/46809) - The `Healthcheck.StartInterval` property is now correctly ignored when updating a Swarm service using API versions less than v1.44. [moby/moby#47991](https://togithub.com/moby/moby/pull/47991) - `GET /events` now supports image `create` event that is emitted when a new image is built regardless if it was tagged or not. [moby/moby#47929](https://togithub.com/moby/moby/pull/47929) - `GET /info` now includes a `Containerd` field containing information about the location of the containerd API socket and containerd namespaces used by the daemon to run containers and plugins. [moby/moby#47239](https://togithub.com/moby/moby/pull/47239) - Deprecate non-standard (config) fields in image inspect output. The `Config` field returned by this endpoint (used for `docker image inspect`) returned additional fields that are not part of the image's configuration and not part of the [Docker Image Spec] and the [OCI Image Spec]. These fields are never set (and always return the default value for the type), but are not omitted in the response when left empty. As these fields were not intended to be part of the image configuration response, they are deprecated, and will be removed in the future API versions. - Deprecate the daemon flag `--api-cors-header` and the corresponding `daemon.json` configuration option. These will be removed in the next major release. [moby/moby#45313](https://togithub.com/moby/moby/pull/45313) The following deprecated fields are currently included in the API response, but are not part of the underlying image's `Config`: [moby/moby#47941](https://togithub.com/moby/moby/pull/47941) - `Hostname` - `Domainname` - `AttachStdin` - `AttachStdout` - `AttachStderr` - `Tty` - `OpenStdin` - `StdinOnce` - `Image` - `NetworkDisabled` (already omitted unless set) - `MacAddress` (already omitted unless set) - `StopTimeout` (already omitted unless set) ##### Go SDK changes - Client API callback for the following functions now require a context parameter. [moby/moby#47536](https://togithub.com/moby/moby/pull/47536) - `client.RequestPrivilegeFunc` - `client.ImageSearchOptions.AcceptPermissionsFunc` - `image.ImportOptions.PrivilegeFunc` - Remove deprecated aliases for Image types. [moby/moby#47900](https://togithub.com/moby/moby/pull/47900) - `ImageImportOptions` - `ImageCreateOptions` - `ImagePullOptions` - `ImagePushOptions` - `ImageListOptions` - `ImageRemoveOptions` - Introduce `Ulimit` type alias for `github.com/docker/go-units.Ulimit`. The `Ulimit` type as used in the API is defined in a Go module that will transition to a new location in future. A type alias is added to reduce the friction that comes with moving the type to a new location. The alias makes sure that existing code continues to work, but its definition may change in future. Users are recommended to use this alias instead of the `units.Ulimit` directly. [moby/moby#48023](https://togithub.com/moby/moby/pull/48023) - Move and rename types, changing their import paths and exported names. [moby/moby#47936](https://togithub.com/moby/moby/pull/47936), [moby/moby#47873](https://togithub.com/moby/moby/pull/47873), [moby/moby#47887](https://togithub.com/moby/moby/pull/47887), [moby/moby#47882](https://togithub.com/moby/moby/pull/47882), [moby/moby#47921](https://togithub.com/moby/moby/pull/47921), [moby/moby#48040](https://togithub.com/moby/moby/pull/48040): - Move the following types to `api/types/container`: - `BlkioStatEntry` - `BlkioStats` - `CPUStats` - `CPUUsage` - `ContainerExecInspect` - `ContainerPathStat` - `ContainerStats` - `ContainersPruneReport` - `CopyToContainerOptions` - `ExecConfig` - `ExecStartCheck` - `MemoryStats` - `NetworkStats` - `PidsStats` - `StatsJSON` - `Stats` - `StorageStats` - `ThrottlingData` - Move the following types to `api/types/image`: - `ImagesPruneReport` - `ImageImportSource` - `ImageLoadResponse` - Move the `ExecStartOptions` type to `api/types/backend`. - Move the `VolumesPruneReport` type to `api/types/volume`. - Move the `EventsOptions` type to `api/types/events`. - Move the `ImageSearchOptions` type to `api/types/registry`. - Drop `Network` prefix and move the following types to `api/types/network`: - `NetworkCreateResponse` - `NetworkConnect` - `NetworkDisconnect` - `NetworkInspectOptions` - `EndpointResource` - `NetworkListOptions` - `NetworkCreateOptions` - `NetworkCreateRequest` - `NetworksPruneReport` - Move `NetworkResource` to `api/types/network`. ##### Packaging updates - Update Buildx to [v0.15.1](https://togithub.com/docker/buildx/releases/tag/v0.15.1). [docker/docker-ce-packaging#1029](https://togithub.com/docker/docker-ce-packaging/pull/1029) - Update BuildKit to [v0.14.1](https://togithub.com/moby/buildkit/releases/tag/v0.14.1). [moby/moby#48028](https://togithub.com/moby/moby/pull/48028) - Update runc to [v1.1.13](https://togithub.com/opencontainers/runc/releases/tag/v1.1.13) [moby/moby#47976](https://togithub.com/moby/moby/pull/47976) - Update Compose to [v2.28.1](https://togithub.com/docker/compose/releases/tag/v2.28.1). [moby/docker-ce-packaging#1032](https://togithub.com/docker/docker-ce-packaging/pull/1032) [Docker image spec]: https://togithub.com/moby/docker-image-spec/blob/v1.3.1/specs-go/v1/image.go#L19-L32 [OCI Image Spec]: https://togithub.com/opencontainers/image-spec/blob/v1.1.0/specs-go/v1/config.go#L24-L62 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6am on monday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/earthly/dind).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

robmry self-assigned this Apr 5, 2024

robmry added kind/enhancement Enhancements are not bugs or new features but can improve usability or performance. area/networking impact/changelog impact/documentation labels Apr 5, 2024

robmry mentioned this pull request Apr 8, 2024

Document CLI support for per interface sysctls docker/cli#4994

Merged

robmry removed the impact/documentation label Apr 8, 2024

robmry force-pushed the 47639_per-interface-sysctls branch from 79f4458 to 745a4c4 Compare April 8, 2024 09:31

robmry marked this pull request as ready for review April 8, 2024 10:12

robmry requested review from akerouanton and corhere April 8, 2024 10:12

robmry mentioned this pull request Apr 15, 2024

Docker 26 breaks kata networking kata-containers/kata-containers#9340

Open

akerouanton reviewed Apr 15, 2024

View reviewed changes

robmry force-pushed the 47639_per-interface-sysctls branch from 745a4c4 to 5d0ab3f Compare April 18, 2024 13:45

robmry force-pushed the 47639_per-interface-sysctls branch from 5d0ab3f to 5d70d23 Compare April 29, 2024 17:38

robmry force-pushed the 47639_per-interface-sysctls branch 2 times, most recently from 2edd300 to 2681c58 Compare May 8, 2024 16:07

robmry added this to the 27.0.0 milestone May 9, 2024

This was referenced May 14, 2024

Add driver_opts to services.networks compose-spec/compose-go#627

Merged

Add driver_opts to services.networks compose-spec/compose-spec#498

Merged

Per endpoint driver opts docker/compose#11815

Merged

corhere reviewed May 15, 2024

View reviewed changes

robmry force-pushed the 47639_per-interface-sysctls branch from 2681c58 to 93e43d5 Compare May 16, 2024 08:45

robmry added 2 commits May 17, 2024 11:30

Factor out selection of endpoint for config migration

a35716f

Signed-off-by: Rob Murray <rob.murray@docker.com>

Move EndpointSettings.DriverOpts from op-state to config

1e29f9b

Signed-off-by: Rob Murray <rob.murray@docker.com>

akerouanton reviewed May 17, 2024

View reviewed changes

api/swagger.yaml Show resolved Hide resolved

api/server/router/container/container_routes_test.go Show resolved Hide resolved

robmry force-pushed the 47639_per-interface-sysctls branch 4 times, most recently from b178729 to e29f2f0 Compare May 21, 2024 09:17

akerouanton reviewed May 21, 2024

View reviewed changes

api/server/router/container/container_routes.go Outdated Show resolved Hide resolved

robmry force-pushed the 47639_per-interface-sysctls branch from e29f2f0 to 058bf75 Compare May 21, 2024 10:26

akerouanton approved these changes May 21, 2024

View reviewed changes

corhere reviewed May 22, 2024

View reviewed changes

corhere approved these changes May 22, 2024

View reviewed changes

robmry force-pushed the 47639_per-interface-sysctls branch 3 times, most recently from 466bf3a to abf6de3 Compare May 26, 2024 12:25

robmry requested review from akerouanton and corhere May 26, 2024 12:27

corhere approved these changes May 27, 2024

View reviewed changes

robmry force-pushed the 47639_per-interface-sysctls branch from abf6de3 to a5dc638 Compare May 28, 2024 08:57

akerouanton approved these changes May 29, 2024

View reviewed changes

robmry force-pushed the 47639_per-interface-sysctls branch from a5dc638 to 0071832 Compare May 29, 2024 07:59

akerouanton approved these changes May 29, 2024

View reviewed changes

akerouanton merged commit 2ebf191 into moby:master May 29, 2024
126 checks passed

robmry deleted the 47639_per-interface-sysctls branch May 29, 2024 17:47

robmry mentioned this pull request Jun 6, 2024

Handle networks.driver_opts for a service docker/cli#5125

Merged

robmry mentioned this pull request Jul 11, 2024

Fix API version in TestSetInterfaceSysctl #48156

Merged

vvoland mentioned this pull request Jul 18, 2024

[27.1 backport] Fix API version in TestSetInterfaceSysctl #48181

Merged

This was referenced Aug 1, 2024

Migrate per-endpoint sysctls until 28.0.0 #48281

Merged

Stop migrating per-interface sysctls from '--sysctl' #48282

Closed

vvoland mentioned this pull request Aug 14, 2024

[27.x backport] Migrate per-endpoint sysctls until 28.0.0 #48332

Merged

robmry mentioned this pull request Oct 24, 2024

From 28.0.0, don't migrate per-endpoint sysctls #48746

Merged

	warning := fmt.Sprintf(`Migrated %s to DriverOpts{"%s":"%s"}.`,
	warning := fmt.Sprintf(`Migrated sysctl %q to DriverOpts{%q:%q}.`,

		// Append exiting per-endpoint sysctls to the migrated sysctls (give priority
		// to per-endpoint settings).

	t.Run("set to "+val, func(t *testing.T) {
	t.Run("ipv4.conf.forwarding="+val, func(t *testing.T) {

		scPath := []string{"/proc/sys/net", sk[0], sk[1], ifName}
		scPath = append(scPath, sk[2:]...)

	scPath := []string{"/proc/sys/net", sk[0], sk[1], ifName}
	scPath = append(scPath, sk[2:]...)
	scPath := append([]string{"/proc/sys/net", sk[0], sk[1], ifName}, sk[2:]...)

Per-interface sysctls #47686

Per-interface sysctls #47686

Uh oh!

Conversation

robmry commented Apr 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robmry May 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robmry Apr 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

corhere left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

corhere commented May 24, 2024

Uh oh!

robmry commented May 26, 2024

Uh oh!

corhere left a comment

Choose a reason for hiding this comment

Uh oh!

robmry commented May 28, 2024

Uh oh!

akerouanton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robmry commented Apr 5, 2024 •

edited

Loading

robmry May 8, 2024 •

edited

Loading

robmry Apr 18, 2024 •

edited

Loading