Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong number of open file calculated #2157

Closed
dawez opened this issue Feb 28, 2018 · 5 comments · Fixed by #2241

Comments

@dawez
Copy link

@dawez dawez commented Feb 28, 2018

The current 0.11 version changed the way the calculation of maximum open files is done. #2050

Now the value is coming from the kernel parameter fs.file-max and this is probably not what we want. The value is shared among all containers if you run many multiple containers only a small fraction of the "open files" might be available to a single container. It make no sense to calculate the open files limit per nginx worker based on this parameter as if nginx would be only one running on the node.

Furthermore each container can have a specific limit that is capped by the kernel limit. If you set a limit for the "open files" in the Docker daemon that is way lower than the kernel parameter, the nginx ingress tries to setup a value [worker_rlimit_nofile] outside the pod's limits and causes an error in the log:

2018/02/28 08:37:57 [alert] 2415#2415: setrlimit(RLIMIT_NOFILE, 201374) failed (1: Operation not permitted)

When running ingress in non priviledged mode it is also not possible to change the kernel value anyway when calling 'sysctl -w fs.file-max=10000 in the initContainer.

What would make sense is to use instead the value coming from the userlimit as this is the real limit for the process open files.

@maxlaverse

This comment has been minimized.

Copy link
Contributor

@maxlaverse maxlaverse commented Feb 28, 2018

Actually, I think it was done right since #177 but #2050 broke the "open files" limit determination.

There was a discussion about it in the issue #2048 that led to PR #2050. The example given in that issue is a very specific use-cases. The calculation only makes sense if the Ingress is alone on the Kubernetes node and if no lower limit is set at a different level

Where the limit for the number of open files should be set is a tricky discussion as seen in kubernetes/kubernetes#3595.

But having the Nginx Ingress relying on the global kernel parameter fs.file-max to calculate how many open files there can be per Nginx worker sounds wrong. fs.file-max is "just" the limit on the kernel level and can be far away from how many files a process is allowed to open.

@bismarck

This comment has been minimized.

Copy link

@bismarck bismarck commented Mar 16, 2018

I just hit this issue. Any plans to revert #2050? Maybe add a configuration flag to opt into using fs.file-max instead of making it the default?

@aledbf

This comment has been minimized.

Copy link
Member

@aledbf aledbf commented Mar 16, 2018

@gyliu513 please check this

@shenshouer

This comment has been minimized.

Copy link

@shenshouer shenshouer commented Dec 18, 2018

No effect on sysctl -w fs.file-max=2097152 in initContainers, cannot change the value of worker_rlimit_nofile on 0.21.0 and 0.20.0

      initContainers:
      - command:
        - /bin/sh
        - -c
        - |
            sysctl -w net.netfilter.nf_conntrack_max=6553600
            sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=120
            sysctl -w net.ipv4.tcp_max_syn_backlog=2621440
            sysctl -w net.ipv4.tcp_max_tw_buckets=8000
            sysctl -w net.core.netdev_max_backlog=2621440
            sysctl -w net.core.rmem_default=33554432
            sysctl -w net.core.wmem_default=33554432
            sysctl -w net.core.rmem_max=33554432
            sysctl -w net.core.wmem_max=33554432
            sysctl -w net.ipv4.tcp_rmem="4096 87380 33554432"
            sysctl -w net.ipv4.tcp_wmem="4096 87380 33554432"
            sysctl -w net.ipv4.tcp_mem="94500000 915000000 927000000"
            sysctl -w net.ipv4.tcp_timestamps=0
            sysctl -w net.ipv4.tcp_synack_retries=1
            sysctl -w net.ipv4.tcp_syn_retries=1
            sysctl -w net.ipv4.tcp_tw_reuse=1
            sysctl -w net.ipv4.tcp_tw_recycle=1
            sysctl -w net.ipv4.ip_forward=1
            sysctl -w net.ipv4.tcp_fin_timeout=15
            sysctl -w net.ipv4.tcp_keepalive_time=60
            sysctl -w net.bridge.bridge-nf-call-iptables=1
            sysctl -w net.bridge.bridge-nf-call-ip6tables=1
            sysctl -w net.ipv4.conf.all.forwarding=1
            sysctl -w net.ipv4.ip_forward=1
            sysctl -w net.core.somaxconn=65535
            sysctl -w net.ipv4.ip_local_port_range="1024 65535"
            sysctl -w fs.file-max=2097152

            sysctl -w fs.inotify.max_user_instances=16384
            sysctl -w fs.inotify.max_user_watches=524288
            sysctl -w fs.inotify.max_queued_events=16384
            sysctl -w fs.suid_dumpable=2
            ulimit -c unlimited
            ulimit -n 1024000
            chmod -R 777 -R /var/log/nginx
        securityContext:
          privileged: true

and logs as:

# docker logs  0ce5e64d9c21
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.21.0
  Build:      git-b65b85cd9
  Repository: https://github.com/aledbf/ingress-nginx
-------------------------------------------------------------------------------

nginx version: nginx/1.15.6
built by gcc 8.2.0 (Debian 8.2.0-9)
built with OpenSSL 1.1.1  11 Sep 2018
TLS SNI support enabled
...
I1218 20:08:40.207903       8 status.go:148] new leader elected: nginx-ingress-controller-75d8d
I1218 20:08:40.208786       8 nginx.go:510] Adjusting ServerNameHashBucketSize variable to 64
I1218 20:08:40.208809       8 nginx.go:522] Number of worker processes: 48
I1218 20:08:40.208821       8 util.go:76] rlimit.max=65536
I1218 20:08:40.208833       8 nginx.go:527] Maximum number of open file descriptors: 341
...

when I exec to pods:

# kubectl -n ingress-nginx exec -it nginx-ingress-controller-8zld2 bash
Error from server (NotFound): pods "nginx-ingress-controller-8zld2" not found
# kubectl -n ingress-nginx exec -it nginx-ingress-controller-7tj8d bash
www-data@nginx-ingress-controller-7tj8d:/etc/nginx$ sysctl -a |grep file-max
sysctl: permission denied on key 'fs.protected_hardlinks'
sysctl: permission denied on key 'fs.protected_symlinks'
fs.file-max = 2097152
...
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Dec 18, 2018

@shenshouer: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.