Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

could not build server_names_hash, you should increase server_names_hash_bucket_size: 64 #34

Closed
rawlingsj opened this issue Jun 20, 2016 · 13 comments

Comments

@rawlingsj
Copy link

I'm deploying nginxdemos/nginx-ingress:0.3 as an RC but the pod goes into ERROR status.

From a bit of googling a suggested workaround is to edit the http{} nginx configuration..

http {
    server_names_hash_bucket_size 64;
    ...
}

but I can't figure out if I can do that. Even if the config isn't able to be customised yet yet I still don't know how I can build my own image adding in the http config. Any pointers?

Pod logs...

kubectl logs ingress-nginx-bq7ym
2016/06/20 11:00:31 [notice] 18#18: signal process started
2016/06/20 11:00:31 [notice] 23#23: signal process started
2016/06/20 11:00:31 [notice] 25#25: signal process started
2016/06/20 11:00:31 [notice] 27#27: signal process started
2016/06/20 11:00:31 [notice] 29#29: signal process started
2016/06/20 11:00:31 [notice] 31#31: signal process started
2016/06/20 11:00:31 [notice] 33#33: signal process started
2016/06/20 11:00:31 [emerg] 35#35: could not build server_names_hash, you should increase server_names_hash_bucket_size: 64
E0620 11:00:31.397974       1 nginx.go:207] Command nginx -s reload stdout: ""
E0620 11:00:31.398051       1 nginx.go:208] Command nginx -s reload stderr: "nginx: [emerg] could not build server_names_hash, you should increase server_names_hash_bucket_size: 64\n"
F0620 11:00:31.398064       1 nginx.go:209] Command nginx -s reload finished with error: exit status 1
@pleshakov
Copy link
Contributor

Thanks for the feedback. Yes, we don't have a mechanism to customize this parameter yet. I'll add it to our list of things.

To customize the server_names_hash_bucket_size parameter, you need to change the default nginx.conf file and add this file to the image, replacing the /etc/nginx/nginx.conf. I'll add this file to the repo and include adding it in the Dockerfile, so that people can easily customize if they need it.

@rawlingsj
Copy link
Author

Ok great - I'll give that a go, many thanks @pleshakov

@pleshakov
Copy link
Contributor

@rawlingsj please check https://github.com/nginxinc/kubernetes-ingress/blob/master/examples/customization/nginx-config.yaml .
#37 added support for customization of server_names_hash_max_size and server_names_hash_bucket_size

@rawlingsj
Copy link
Author

great thanks @pleshakov I'll give it a go as soon as I can, hopefully in the next day or so.

@rawlingsj
Copy link
Author

I've only just got around to testing this sorry.

I've added the config map which from the logs looks like it's read however the controller still seems to use the default nginx config rather than the overrides in the default/nginx-config configmap.

I've increased the client-max-body-size and server-names-hash-bucket-size in the ConfigMap.

I still get the error could not build server_names_hash, you should increase server_names_hash_bucket_size: 64 and the client_max_body_size is still being set to the default of 1m.

Am I doing something wrong?

ConfigMap created in the default namespace:

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
data:
  proxy-connect-timeout: "10s"
  proxy-read-timeout: "10s"
  client-max-body-size: "2000m"
  server-names-hash-bucket-size: "64"
  server-names-hash-max-size: "1024"

RC:

apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx-ingress-rc
  labels:
    app: nginx-ingress
spec:
  replicas: 1
  selector:
    app: nginx-ingress
  template:
    metadata:
      labels:
        app: nginx-ingress
    spec:
      containers:
      - image: nginxdemos/nginx-ingress:0.3
        imagePullPolicy: Always
        name: nginx-ingress
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        # Uncomment the lines below to enable extensive logging and/or customization of
        # NGINX configuration with configmaps
        args:
        - -v=3
        - -nginx-configmaps=default/nginx-config

The full log:

I0725 12:03:29.680146       1 nginx.go:230] Writing NGINX conf to /etc/nginx/nginx.conf

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    server_names_hash_max_size 512;


    include /etc/nginx/conf.d/*.conf;
}
I0725 12:03:29.681296       1 nginx.go:248] The main NGINX configuration file had been updated
I0725 12:03:29.681343       1 nginx.go:203] executing nginx
I0725 12:03:30.000081       1 controller.go:73] Adding Ingress: gogs
I0725 12:03:30.000104       1 controller.go:73] Adding Ingress: gogs-ssh
I0725 12:03:30.000111       1 controller.go:73] Adding Ingress: jenkins
I0725 12:03:30.000116       1 controller.go:73] Adding Ingress: jenkins-jnlp
I0725 12:03:30.000120       1 controller.go:73] Adding Ingress: nexus
I0725 12:03:30.000125       1 controller.go:73] Adding Ingress: fabric8-docker-registry
I0725 12:03:30.000130       1 controller.go:73] Adding Ingress: fabric8-forge
I0725 12:03:30.000136       1 utils.go:70] Syncing default/gogs
I0725 12:03:30.000165       1 controller.go:316] Syncing default/gogs
I0725 12:03:30.000179       1 controller.go:331] Adding or Updating Ingress: default/gogs
I0725 12:03:30.000209       1 controller.go:405] Error retriviend endpoints for the services gogs: Svc default/gogs doesn't exists
I0725 12:03:30.000242       1 nginx.go:106] Updating NGINX configuration
I0725 12:03:30.000501       1 controller.go:125] Adding endpoints: jenkins-jnlp
I0725 12:03:30.000519       1 controller.go:125] Adding endpoints: grafana
I0725 12:03:30.000525       1 controller.go:125] Adding endpoints: kibana
I0725 12:03:30.000547       1 controller.go:125] Adding endpoints: dnsmasq
I0725 12:03:30.000556       1 controller.go:125] Adding endpoints: kube-controller-manager
I0725 12:03:30.000561       1 controller.go:125] Adding endpoints: gogs
I0725 12:03:30.000566       1 controller.go:125] Adding endpoints: jenkins
I0725 12:03:30.000571       1 controller.go:125] Adding endpoints: kube-scheduler
I0725 12:03:30.000577       1 controller.go:125] Adding endpoints: fabric8-forge
I0725 12:03:30.000583       1 controller.go:125] Adding endpoints: fabric8
I0725 12:03:30.000588       1 controller.go:125] Adding endpoints: gogs-ssh
I0725 12:03:30.000592       1 controller.go:125] Adding endpoints: elasticsearch
I0725 12:03:30.000597       1 controller.go:125] Adding endpoints: elasticsearch-masters
I0725 12:03:30.000604       1 controller.go:125] Adding endpoints: prometheus
I0725 12:03:30.000610       1 controller.go:125] Adding endpoints: kubernetes
I0725 12:03:30.000615       1 controller.go:125] Adding endpoints: kube-dns
I0725 12:03:30.000620       1 controller.go:125] Adding endpoints: nexus
I0725 12:03:30.000620       1 nginx.go:152] Writing NGINX conf to /etc/nginx/conf.d/default-gogs.conf
I0725 12:03:30.000629       1 utils.go:70] Syncing default/jenkins-jnlp
I0725 12:03:30.000635       1 controller.go:257] Syncing endpoints default/jenkins-jnlp
I0725 12:03:30.000640       1 utils.go:70] Syncing default/grafana
I0725 12:03:30.000643       1 controller.go:257] Syncing endpoints default/grafana
I0725 12:03:30.000651       1 utils.go:70] Syncing default/kibana
I0725 12:03:30.000655       1 controller.go:257] Syncing endpoints default/kibana
I0725 12:03:30.000659       1 utils.go:70] Syncing kube-system/dnsmasq
I0725 12:03:30.000664       1 controller.go:257] Syncing endpoints kube-system/dnsmasq
I0725 12:03:30.000668       1 utils.go:70] Syncing kube-system/kube-controller-manager
I0725 12:03:30.000672       1 controller.go:257] Syncing endpoints kube-system/kube-controller-manager
I0725 12:03:30.000677       1 utils.go:70] Syncing default/gogs
I0725 12:03:30.000681       1 controller.go:257] Syncing endpoints default/gogs
I0725 12:03:30.000685       1 utils.go:70] Syncing default/jenkins
I0725 12:03:30.000690       1 controller.go:257] Syncing endpoints default/jenkins
I0725 12:03:30.000694       1 utils.go:70] Syncing kube-system/kube-scheduler
I0725 12:03:30.000699       1 controller.go:257] Syncing endpoints kube-system/kube-scheduler
I0725 12:03:30.000703       1 utils.go:70] Syncing default/fabric8-forge
I0725 12:03:30.000705       1 controller.go:257] Syncing endpoints default/fabric8-forge
I0725 12:03:30.000709       1 utils.go:70] Syncing default/fabric8
I0725 12:03:30.000711       1 controller.go:257] Syncing endpoints default/fabric8
I0725 12:03:30.000715       1 utils.go:70] Syncing default/gogs-ssh
I0725 12:03:30.000718       1 controller.go:257] Syncing endpoints default/gogs-ssh
I0725 12:03:30.000724       1 utils.go:70] Syncing default/elasticsearch
I0725 12:03:30.000726       1 controller.go:257] Syncing endpoints default/elasticsearch
I0725 12:03:30.000730       1 utils.go:70] Syncing default/elasticsearch-masters
I0725 12:03:30.000732       1 controller.go:257] Syncing endpoints default/elasticsearch-masters
I0725 12:03:30.000756       1 controller.go:350] ignoring service elasticsearch-masters: No ingress for service elasticsearch-masters
I0725 12:03:30.000762       1 utils.go:70] Syncing default/prometheus
I0725 12:03:30.000764       1 controller.go:257] Syncing endpoints default/prometheus
I0725 12:03:30.000768       1 utils.go:70] Syncing default/kubernetes
I0725 12:03:30.000771       1 controller.go:257] Syncing endpoints default/kubernetes
I0725 12:03:30.000775       1 utils.go:70] Syncing kube-system/kube-dns
I0725 12:03:30.000778       1 controller.go:257] Syncing endpoints kube-system/kube-dns
I0725 12:03:30.000784       1 utils.go:70] Syncing default/nexus
I0725 12:03:30.000786       1 controller.go:257] Syncing endpoints default/nexus
I0725 12:03:30.000740       1 controller.go:99] Adding service: elasticsearch-masters
I0725 12:03:30.000804       1 controller.go:350] ignoring service elasticsearch-masters: No ingress for service elasticsearch-masters
I0725 12:03:30.000822       1 controller.go:99] Adding service: dnsmasq
I0725 12:03:30.000830       1 controller.go:350] ignoring service dnsmasq: No ingress for service dnsmasq
I0725 12:03:30.000840       1 controller.go:99] Adding service: kube-dns
I0725 12:03:30.000848       1 controller.go:350] ignoring service kube-dns: No ingress for service kube-dns
I0725 12:03:30.000856       1 controller.go:99] Adding service: jenkinshift
I0725 12:03:30.000876       1 controller.go:350] ignoring service jenkinshift: No ingress for service jenkinshift
I0725 12:03:30.000892       1 controller.go:99] Adding service: gogs
I0725 12:03:30.000625       1 controller.go:125] Adding endpoints: jenkinshift
I0725 12:03:30.000907       1 controller.go:99] Adding service: jenkins-jnlp
I0725 12:03:30.000911       1 controller.go:125] Adding endpoints: fabric8-docker-registry
I0725 12:03:30.000920       1 controller.go:99] Adding service: nexus
I0725 12:03:30.000933       1 controller.go:99] Adding service: elasticsearch
I0725 12:03:30.000940       1 controller.go:350] ignoring service elasticsearch: No ingress for service elasticsearch

upstream default-gogs-gogs.default.beast.fabric8.io-gogs {

    server 127.0.0.1:8181;
}


server {
    listen 80;



    server_name gogs.default.beast.fabric8.io;





    location / {
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://default-gogs-gogs.default.beast.fabric8.io-gogs;
    }
}
I0725 12:03:30.000920       1 utils.go:70] Syncing default/jenkinshift
I0725 12:03:30.001273       1 controller.go:257] Syncing endpoints default/jenkinshift
I0725 12:03:30.001303       1 controller.go:350] ignoring service jenkinshift: No ingress for service jenkinshift
I0725 12:03:30.001319       1 utils.go:70] Syncing default/fabric8-docker-registry
I0725 12:03:30.001329       1 controller.go:257] Syncing endpoints default/fabric8-docker-registry
I0725 12:03:30.001184       1 controller.go:99] Adding service: kibana
I0725 12:03:30.001358       1 controller.go:350] ignoring service kibana: No ingress for service kibana
I0725 12:03:30.001376       1 controller.go:99] Adding service: prometheus
I0725 12:03:30.001390       1 controller.go:350] ignoring service prometheus: No ingress for service prometheus
I0725 12:03:30.001409       1 controller.go:99] Adding service: fabric8-docker-registry
I0725 12:03:30.001438       1 controller.go:99] Adding service: fabric8-forge
I0725 12:03:30.001467       1 controller.go:99] Adding service: fabric8
I0725 12:03:30.001487       1 controller.go:350] ignoring service fabric8: No ingress for service fabric8
I0725 12:03:30.001511       1 controller.go:99] Adding service: grafana
I0725 12:03:30.001557       1 controller.go:350] ignoring service grafana: No ingress for service grafana
I0725 12:03:30.001577       1 controller.go:99] Adding service: kubernetes
I0725 12:03:30.001600       1 controller.go:350] ignoring service kubernetes: No ingress for service kubernetes
I0725 12:03:30.001628       1 controller.go:99] Adding service: gogs-ssh
I0725 12:03:30.001655       1 controller.go:99] Adding service: jenkins
I0725 12:03:30.002063       1 nginx.go:172] NGINX configuration file had been updated
I0725 12:03:30.002093       1 nginx.go:203] executing nginx -s reload
2016/07/25 12:03:30 [notice] 22#22: signal process started
I0725 12:03:30.007728       1 utils.go:70] Syncing default/gogs-ssh
I0725 12:03:30.007745       1 controller.go:316] Syncing default/gogs-ssh
I0725 12:03:30.007762       1 controller.go:331] Adding or Updating Ingress: default/gogs-ssh
I0725 12:03:30.007777       1 nginx.go:106] Updating NGINX configuration
I0725 12:03:30.007963       1 nginx.go:152] Writing NGINX conf to /etc/nginx/conf.d/default-gogs-ssh.conf

upstream default-gogs-ssh-gogs-ssh.default.beast.fabric8.io-gogs-ssh {

    server 10.233.110.3:22;
}


server {
    listen 80;



    server_name gogs-ssh.default.beast.fabric8.io;





    location / {
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://default-gogs-ssh-gogs-ssh.default.beast.fabric8.io-gogs-ssh;
    }
}
I0725 12:03:30.008481       1 nginx.go:172] NGINX configuration file had been updated
I0725 12:03:30.008506       1 nginx.go:203] executing nginx -s reload
2016/07/25 12:03:30 [notice] 25#25: signal process started
I0725 12:03:30.018051       1 utils.go:70] Syncing default/jenkins
I0725 12:03:30.018067       1 controller.go:316] Syncing default/jenkins
I0725 12:03:30.018073       1 controller.go:331] Adding or Updating Ingress: default/jenkins
I0725 12:03:30.018091       1 nginx.go:106] Updating NGINX configuration
I0725 12:03:30.018267       1 nginx.go:152] Writing NGINX conf to /etc/nginx/conf.d/default-jenkins.conf

upstream default-jenkins-jenkins.default.beast.fabric8.io-jenkins {

    server 127.0.0.1:8181;
}


server {
    listen 80;



    server_name jenkins.default.beast.fabric8.io;





    location / {
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://default-jenkins-jenkins.default.beast.fabric8.io-jenkins;
    }
}
I0725 12:03:30.018559       1 nginx.go:172] NGINX configuration file had been updated
I0725 12:03:30.018572       1 nginx.go:203] executing nginx -s reload
I0725 12:03:30.021605       1 controller.go:160] Adding ConfigMap: nginx-config
I0725 12:03:30.021822       1 utils.go:70] Syncing default/nginx-config
I0725 12:03:30.022956       1 controller.go:279] Syncing configmap default/nginx-config
2016/07/25 12:03:30 [notice] 28#28: signal process started
I0725 12:03:30.024091       1 utils.go:70] Syncing default/jenkins-jnlp
I0725 12:03:30.024105       1 controller.go:316] Syncing default/jenkins-jnlp
I0725 12:03:30.024111       1 controller.go:331] Adding or Updating Ingress: default/jenkins-jnlp
I0725 12:03:30.024128       1 nginx.go:106] Updating NGINX configuration
I0725 12:03:30.024301       1 nginx.go:152] Writing NGINX conf to /etc/nginx/conf.d/default-jenkins-jnlp.conf

upstream default-jenkins-jnlp-jenkins-jnlp.default.beast.fabric8.io-jenkins-jnlp {

    server 10.233.86.4:50000;
}


server {
    listen 80;



    server_name jenkins-jnlp.default.beast.fabric8.io;





    location / {
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://default-jenkins-jnlp-jenkins-jnlp.default.beast.fabric8.io-jenkins-jnlp;
    }
}
I0725 12:03:30.024783       1 nginx.go:172] NGINX configuration file had been updated
I0725 12:03:30.024814       1 nginx.go:203] executing nginx -s reload
2016/07/25 12:03:30 [notice] 32#32: signal process started
I0725 12:03:30.030058       1 utils.go:70] Syncing default/nexus
I0725 12:03:30.030083       1 controller.go:316] Syncing default/nexus
I0725 12:03:30.030089       1 controller.go:331] Adding or Updating Ingress: default/nexus
I0725 12:03:30.030107       1 nginx.go:106] Updating NGINX configuration
I0725 12:03:30.030295       1 nginx.go:152] Writing NGINX conf to /etc/nginx/conf.d/default-nexus.conf

upstream default-nexus-nexus.default.beast.fabric8.io-nexus {

    server 127.0.0.1:8181;
}


server {
    listen 80;



    server_name nexus.default.beast.fabric8.io;





    location / {
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://default-nexus-nexus.default.beast.fabric8.io-nexus;
    }
}
I0725 12:03:30.030592       1 nginx.go:172] NGINX configuration file had been updated
I0725 12:03:30.030609       1 nginx.go:203] executing nginx -s reload
2016/07/25 12:03:30 [notice] 34#34: signal process started
I0725 12:03:30.036045       1 utils.go:70] Syncing default/fabric8-docker-registry
I0725 12:03:30.036061       1 controller.go:316] Syncing default/fabric8-docker-registry
I0725 12:03:30.036066       1 controller.go:331] Adding or Updating Ingress: default/fabric8-docker-registry
I0725 12:03:30.036084       1 nginx.go:106] Updating NGINX configuration
I0725 12:03:30.036260       1 nginx.go:152] Writing NGINX conf to /etc/nginx/conf.d/default-fabric8-docker-registry.conf

upstream default-fabric8-docker-registry-fabric8-docker-registry.default.beast.fabric8.io-fabric8-docker-registry {

    server 127.0.0.1:8181;
}


server {
    listen 80;



    server_name fabric8-docker-registry.default.beast.fabric8.io;





    location / {
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://default-fabric8-docker-registry-fabric8-docker-registry.default.beast.fabric8.io-fabric8-docker-registry;
    }
}
I0725 12:03:30.036522       1 nginx.go:172] NGINX configuration file had been updated
I0725 12:03:30.036550       1 nginx.go:203] executing nginx -s reload
2016/07/25 12:03:30 [emerg] 36#36: could not build server_names_hash, you should increase server_names_hash_bucket_size: 64
E0725 12:03:30.041781       1 nginx.go:216] Command nginx -s reload stdout: ""
E0725 12:03:30.041800       1 nginx.go:217] Command nginx -s reload stderr: "nginx: [emerg] could not build server_names_hash, you should increase server_names_hash_bucket_size: 64\n"
F0725 12:03:30.041811       1 nginx.go:218] Command nginx -s reload finished with error: exit status 1

@rawlingsj
Copy link
Author

I think I figured it out and have submitted the PR above, I ca certainly override server_names_hash_bucket_size using a config map now. I suspect the reason there was no default added before was because the nginx defaults depend on the size of the processor’s cache line see http://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_bucket_size

But without a default then the controller logic wasn't overriding the value when I added it to the config map.

Anyways, things are working here with that PR fix however may want to implement a different fix.

@pleshakov
Copy link
Contributor

From the logs I see that the controller didn't use the values from the configmap: it didn't update the main configuration file and when it was writing the config files to the conf.d directory, it didn't use the values specified in the configmap.

I think it was a timing issue: the controller wrote the config files before it started to use the values from the configmap. Do you see that the issue appears consistently?

Yes, the reason there is no default value for the server_names_hash_bucket_size is because the value depends on the host machine architecture.
The controller must update the value of the server_names_hash_bucket_size in the main configuration file regardless if the default value is provided in the nginx-controller/nginx/config.go and nginx-controller/nginx/nginx.go, as you suggested in your pull request.

The possible solutions I see:

  • create ingress resources after creating a configmap resource
  • change the controller logic: if a configmap is specified as an argument to the controller, don't update NGINX configuration before it picks up the values from the configmap, which we can add.

@rawlingsj
Copy link
Author

I think it was a timing issue: the controller wrote the config files before it started to use the values from the configmap. Do you see that the issue appears consistently?

Yeah it was happening every time.

Yes, the reason there is no default value for the server_names_hash_bucket_size is because the value depends on the host machine architecture.

Makes sense, I'll close that PR I submitted as it's not a fix.

The possible solutions I see:

  • create ingress resources after creating a configmap resource
  • change the controller logic: if a configmap is specified as an argument to the controller, don't update NGINX configuration before it picks up the values from the configmap, which we can add.

Ok cool - which ever route you prefer to go down I'm happy to build from a fork and test against my setup before you merge if that helps.

@kenden
Copy link

kenden commented Apr 12, 2017

The problem is still happening with Kubernetes 1.5.2 and nginx-ingress chart nginx-ingress-0.3.2.tgz (which uses image gcr.io/google_containers/nginx-ingress-controller:0.8.3), on an AWS EC2 instance t2.medium.
The logs of the nginx-controller pod show:

E0412 20:56:07.993434       1 command.go:87] failed to execute nginx -s reload: 2017/04/12 20:56:07 [emerg] 17#17: could not build server_names_hash, you should increase server_names_hash_bucket_size: 64
nginx: [emerg] could not build server_names_hash, you should increase server_names_hash_bucket_size: 64
W0412 20:56:07.993463       1 utils.go:91] requeuing default/default-nginx-ingress-default-backend, err error reloading nginx: exit status 1

Repro steps:

# Install nginx-controller with helm (less typing)
# we are adding a configMap including the values that should be changed
echo 'controller:
  config:
    server-names-hash-bucket-size: "1024"
    server-names-hash-max-size: "1024"
    server_names_hash_bucket_size: "1024" # test' > values.yaml
$ helm upgrade --install --debug default stable/nginx-ingress -f values.yaml
# view nginx-ingress-controller pod logs
$ kubectl logs $(kubectl get po -l app=nginx-ingress,component=controller --namespace default -o jsonpath='{.items[0].metadata.name}')
# ---> error shows in the logs

@pleshakov
Copy link
Contributor

@kenden
gcr.io/google_containers/nginx-ingress-controller:0.8.3 is based on this controller. Please check its configuration -- https://github.com/kubernetes/ingress/blob/master/controllers/nginx/configuration.md

@afoninsky
Copy link

@kenden looks like mistype in nginx ingress config map
should use server-name-hash-bucket-size instead of server-names-hash-bucket-size

@kenden
Copy link

kenden commented Apr 15, 2017

@afoninsky @pleshakov That was it, it works after setting server-name-hash-bucket-size: "1024" in the configMap. Thanks guys.

@dotw
Copy link

dotw commented Apr 25, 2017

@kenden Thanks. It works for me after setting server-name-hash-bucket-size: "1024" in the configMap. T

itskingori added a commit to Intellection/docker-nginx-proxy that referenced this issue Dec 11, 2020
Due to the long upstream server names we use, we were getting errors on
startup. The errors mentioned that we should increase this value. Error
looks something like this ...

```
2017/10/30 19:48:54 [emerg] 1#1: could not build server_names_hash, you should increase server_names_hash_bucket_size: 64
nginx: [emerg] could not build server_names_hash, you should increase server_names_hash_bucket_size: 64
```

Sources:

* http://nginx.org/en/docs/http/ngx_http_core_module.html#server_names_hash_bucket_size
* nginxinc/kubernetes-ingress#34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants