Issue in setting --sysctl net.core parameter with docker container.

[Mishi-999]

Description

I am facing a critical issue while running oracle rac database in docker container solely as net.core* parameters are not set in the container. I also tried with --net=host but that does not seem to help as it uses host network which limit connectivity between two rac containers and also hinders the installation.
I am using Docker-version 1.12.1 on ubuntu 4.4.0-59-generic.

Parameters are set in /etc/sysctl.conf and yet on running sysctl -a i am unable to grep "net.core.rmem_default = 262144".

Steps to reproduce the issue-

1. Pull basic image from a repository.
2. Run docker container in --priviledged mode with --sysctl parameter to set inside the container.

docker run \
--detach \
--privileged \
--name rac1 \
--hostname rac1 \
--sysctl net.core.rmem_default=262144 \
-p 1565:1521 \
--volume /srv/docker/rac_nodes/custom_services:/usr/lib/custom_services \
--volume /oracledata/stage:/stage \
--volume /sys/fs/cgroup:/sys/fs/cgroup:ro \
--volume /oraclenfs:/oraclenfs \
--shm-size 8000m \
--dns 10.10.10.10 \
--add-host=clu-121-scan.clu-121.example.com:10.10.10.15 \
--add-host=rac1:172.17.0.8 \
giinstalled \
/usr/lib/systemd/systemd --system --unit=multi-user.target

Error-
Inspite of running --sysctl net.core.rmem_default=262144 parameter in priviledged mode shows the error-

docker: Error response from daemon: oci runtime error: open /proc/sys/net/core/rmem_default: no such file or directory.

I also tried by manually setting this parameter in file-

[@ubuntu:~rac-test/12c-rac-docker]$ docker exec rac1 "touch /proc/sys/net/core/rmem_default"
rpc error: code = 2 desc = oci runtime error: exec failed: exec: "touch /proc/sys/net/core/rmem_default": stat touch /proc/sys/net/core/rmem_default: no such file or directory

[@ubuntu:~/rac-test/12c-rac-docker]$ docker exec rac1 "echo 262144 >> /proc/sys/net/core/rmem_default"
rpc error: code = 2 desc = oci runtime error: exec failed: exec: "echo 262144 >> /proc/sys/net/core/rmem_default": stat echo 262144 >> /proc/sys/net/core/rmem_default: no such file or directory.

But all in vain as it does not allow me to make a new file in /proc/sys/net/core in running container.

[rhatdan]

I have a feeling this is more of a kernel issue with the network namespace then a docker issue. We should probably bring this to the kernel list.

[cpuguy83]

Yep, docker is just trying to set this for you, if the setting doesn't exist there's not really anything we can do here.

[Mishi-999]

If i understood correctly then this means net.core.* parameters should be set on my host machine which is true ,means it is set on my machine.

[justincormack]

Yes, if it is not in the namespaced version, it is a global parameter you can set on the host and it will affect all containers.

[Mishi-999]

Yes it is set on my host but is not reflected inside my container.

[rhatdan]

This looks like a network namespace issue, with the kernel.

[eastpav]

I have the some issue. but I need modify net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes. I tried many images from docker hub, but none of them has the three parameters. all of the docker images dropped these parameters?

[rhatdan]

Openend a RHEL Bugzilla on the kernel on this issue.
https://bugzilla.redhat.com/show_bug.cgi?id=1461696

[thaJeztah]

@rhatdan Looks like the issue is "private"; could it be opened up so that people can follow the status on that ticket?

[thatsk]

docker run -d --privileged --name sysctl --sysctl net.core.netdev_max_backlog=30000 --net=example -v /sys/fs/cgroup:/sys/fs/cgroup:ro centos/systemd
d1e528e082f7f7b9f7f5e9976d4361325e78ef27d267e7529b047bb0ca728550
docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"write sysctl key net.core.netdev_max_backlog: open /proc/sys/net/core/netdev_max_backlog: no such file or directory\"": unknown.

[thatsk]

looks like centos latest image is missing this also

[vinujan59]

`
ubuntu@ip:~$ docker --version
Docker version 19.03.2, build 6a30dfc

ubuntu@ip:~$ uname -r
4.15.0-1051-aws

ubuntu@ip:~$ sysctl net.core.rmem_default
net.core.rmem_default = 212992

ubuntu@ip:~$ docker run --privileged -it ubuntu:16.04 uname -r
4.15.0-1051-aws

ubuntu@ip:~$ docker run --privileged -it ubuntu:16.04 sysctl net.core.rmem_default
sysctl: cannot stat /proc/sys/net/core/rmem_default: No such file or directory

ubuntu@ip:~$ docker run --privileged --sysctl net.core.rmem_default=524288 -it ubuntu:16.04 /bin/bash
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused "write sysctl key net.core.rmem_default: open /proc/sys/net/core/rmem_default: no such file or directory"": unknown.

ubuntu@ip:~$ docker run --privileged --network="host" -it ubuntu:16.04 sysctl net.core.rmem_default
net.core.rmem_default = 212992
`

• have the latest docker version
• the host has the parameter net.core.rmem_default
• the same kernel will be used by the running container
• the container doesn't have this parameter and not settable as well
• but with host network mode, it can be verified in the container, the value is being shared
  -- implies that net.core.rmem_default is namespaced

is docker doesn't support net.core.rmem_default parameter?

[crazy-canux]

any update?
Is there a walkaround for this?

[Asisranjan]

Hi @crazy-canux & @Mishi-999
Did you find a solution for it.
I am having this issue as well.

[vinujan59]

Updated the network mode to host which enabled us to modify the value

[crazy-canux]

mode=host means you modified for host which shared to container.
I give up this function.
No workaround for docker swarm.
k8s is the best solution.

[guillaumedsde]

Hi,

Does anyone know of a kernel which has these parameters namespaced?

I have tried with the following setup without sucess:

➜  ~ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 29
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.11.6-051106-generic
 Operating System: Ubuntu 20.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 31.19GiB
 Name: ████████████
 ID: ████████████
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://████████████/
 Live Restore Enabled: false
➜  ~ uname -a
Linux ████████████ 5.11.6-051106-generic #202103111435 SMP Thu Mar 11 14:42:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
➜  ~ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal

[thaJeztah]

(whoops, posted on the wrong ticket: removed my comment)

[tianon]

Ah, it looks like net.core.{r,w}mem_{default,max} maybe never actually got namespaced in the kernel? (https://patchwork.ozlabs.org/project/netdev/patch/20170726170333.24580-1-mcroce@redhat.com/)

[thaJeztah]

Yes, at least Dan opened a ticket for it; #30778 (comment)

Openend a RHEL Bugzilla on the kernel on this issue.
https://bugzilla.redhat.com/show_bug.cgi?id=1461696