geoip2 enabling causes crash of controller v1.10 #11320

jlm0x017 · 2024-04-26T18:59:32Z

tl;dr: nginx fails to start in controller:

2024/04/26 18:32:18 [emerg] 23#23: unknown "geoip_country_code" variable
nginx: [emerg] unknown "geoip_country_code" variable
nginx: configuration file /tmp/nginx/nginx-cfg4113448354 test failed
-------------------------------------------------------------------------------
E0426 18:32:18.395175       7 queue.go:131] "requeuing" err=<
	-------------------------------------------------------------------------------
	Error: exit status 1
	2024/04/26 18:32:18 [emerg] 23#23: unknown "geoip_country_code" variable
	nginx: [emerg] unknown "geoip_country_code" variable
	nginx: configuration file /tmp/nginx/nginx-cfg4113448354 test failed
	-------------------------------------------------------------------------------
 > key="initial-sync"

What happened:

Using helm-chart 4.9.1 we experience no issues.

In updating to helm-chart 4.10.0 (and in 4.10.1) we have failures. The deployment for ingress-nginx-controller pods fail with these events:

Error reloading NGINX: ------------------------------------------------------------------------------- Error: exit status 1 2024/04/26 17:53:57 [emerg] 28#28: unknown "geoip_country_code" variable nginx: [emerg] unknown "geoip_country_code" variable nginx: configuration file /tmp/nginx/nginx-cfg3893559384 test failed -------------------------------------------------------------------------------

What you expected to happen:

I expect helm-chart versions to upgrade cleanly, or with well-advertised required configuration changes.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

version from running 4.9.1 helm chart:
$ /nginx-ingress-controller --version

NGINX Ingress controller
Release: v1.9.6
Build: 6a73aa3
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6

From failing 4.10.0 helm chart:

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.10.0
  Build:         71f78d49f0a496c31d4c19f095469f3f23900f8a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.3
-------------------------------------------------------------------------------

*Kubernetes version

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.8-eks-adc7111

Environment:

AWS EKS 1.28
Bottlerocket OS 1.19.4 (aws-k8s-1.28)
Kernel 6.1.82
Install tools:
- using argocd to deploy this chart
Basic cluster related info:
- client version listed above
- kubectl get nodes -o wide
How was the ingress-nginx-controller installed:
- argocd is deploying the helm chart on our behalf. The provided helm commands do not run from my workstation; Chart.yaml and values.yaml listed below.
Current State of the controller:
- n/a
Current state of ingress object, if applicable:
- n/a
Others:
- pod logs:

pod logs

logs from a failing pod:

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.10.0
  Build:         71f78d49f0a496c31d4c19f095469f3f23900f8a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.3
-------------------------------------------------------------------------------
W0426 18:32:16.956803       7 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0426 18:32:16.956993       7 main.go:205] "Creating API client" host="https://172.20.0.1:443"
I0426 18:32:16.973821       7 main.go:249] "Running in Kubernetes cluster" major="1" minor="28+" git="v1.28.8-eks-adc7111" state="clean" commit="d8d7a89760f6e2095d34d895e4f126c8a9a82c25" platform="linux/amd64"
I0426 18:32:17.074005       7 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0426 18:32:17.136991       7 ssl.go:536] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0426 18:32:17.154350       7 nginx.go:265] "Starting NGINX Ingress controller"
I0426 18:32:17.164074       7 store.go:535] "ignoring ingressclass as the spec.controller is not the same of this ingress" ingressclass="alb"
I0426 18:32:17.170329       7 event.go:364] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"nginx-private-controller", UID:"b71d7198-ce07-4ce4-bf3a-73c73cbef221", APIVersion:"v1", ResourceVersion:"56814598", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap kube-system/nginx-private-controller
I0426 18:32:18.258944       7 store.go:440] "Found valid IngressClass" ingress="argocd/argo-rollouts-dashboard" ingressclass="nginx-private"
I0426 18:32:18.259161       7 event.go:364] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"argocd", Name:"argo-rollouts-dashboard", UID:"ab517d5a-05ff-4322-a280-5bc66b240fb7", APIVersion:"networking.k8s.io/v1", ResourceVersion:"56807868", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0426 18:32:18.259224       7 store.go:440] "Found valid IngressClass" ingress="kube-system/pghero" ingressclass="nginx-private"
I0426 18:32:18.259339       7 event.go:364] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kube-system", Name:"pghero", UID:"c16b7141-106d-42bf-a5c4-8dc462541921", APIVersion:"networking.k8s.io/v1", ResourceVersion:"56807866", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0426 18:32:18.259370       7 store.go:436] "Ignoring ingress because of error while validating ingress class" ingress="loadpay/loadpay-web" error="no object matching key \"alb\" in local store"
I0426 18:32:18.259383       7 store.go:440] "Found valid IngressClass" ingress="monitoring/kube-prometheus-alertmanager" ingressclass="nginx-private"
I0426 18:32:18.259413       7 event.go:364] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"monitoring", Name:"kube-prometheus-alertmanager", UID:"9fee6ebe-3286-4f3a-a964-6124a74052b7", APIVersion:"networking.k8s.io/v1", ResourceVersion:"56807867", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0426 18:32:18.259582       7 store.go:440] "Found valid IngressClass" ingress="monitoring/kube-prometheus-prometheus" ingressclass="nginx-private"
I0426 18:32:18.259747       7 event.go:364] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"monitoring", Name:"kube-prometheus-prometheus", UID:"813cb49e-6708-4706-8630-b485bb5cd4bb", APIVersion:"networking.k8s.io/v1", ResourceVersion:"56807865", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0426 18:32:18.356142       7 nginx.go:308] "Starting NGINX process"
I0426 18:32:18.356213       7 leaderelection.go:250] attempting to acquire leader lease kube-system/nginx-private-controller-leader...
I0426 18:32:18.356572       7 nginx.go:328] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0426 18:32:18.356959       7 controller.go:190] "Configuration changes detected, backend reload required"
I0426 18:32:18.361240       7 status.go:84] "New leader elected" identity="nginx-private-controller-78f9cd6d7d-hwjq5"
E0426 18:32:18.395101       7 controller.go:205] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: exit status 1
2024/04/26 18:32:18 [emerg] 23#23: unknown "geoip_country_code" variable
nginx: [emerg] unknown "geoip_country_code" variable
nginx: configuration file /tmp/nginx/nginx-cfg4113448354 test failed
-------------------------------------------------------------------------------
E0426 18:32:18.395175       7 queue.go:131] "requeuing" err=<
	-------------------------------------------------------------------------------
	Error: exit status 1
	2024/04/26 18:32:18 [emerg] 23#23: unknown "geoip_country_code" variable
	nginx: [emerg] unknown "geoip_country_code" variable
	nginx: configuration file /tmp/nginx/nginx-cfg4113448354 test failed
	-------------------------------------------------------------------------------
 > key="initial-sync"
I0426 18:32:18.395280       7 event.go:364] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"nginx-private-controller-5bd766c8df-7lq2l", UID:"ff1ec4af-1ab4-4846-a17f-a4b946d394b4", APIVersion:"v1", ResourceVersion:"56814629", FieldPath:""}): type: 'Warning' reason: 'RELOAD' Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2024/04/26 18:32:18 [emerg] 23#23: unknown "geoip_country_code" variable
nginx: [emerg] unknown "geoip_country_code" variable
nginx: configuration file /tmp/nginx/nginx-cfg4113448354 test failed
-------------------------------------------------------------------------------
I0426 18:32:21.690444       7 controller.go:190] "Configuration changes detected, backend reload required"
E0426 18:32:21.713515       7 controller.go:205] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: exit status 1
2024/04/26 18:32:21 [emerg] 26#26: unknown "geoip_country_code" variable
nginx: [emerg] unknown "geoip_country_code" variable
nginx: configuration file /tmp/nginx/nginx-cfg3188325810 test failed
-------------------------------------------------------------------------------
E0426 18:32:21.713652       7 queue.go:131] "requeuing" err=<
	-------------------------------------------------------------------------------
	Error: exit status 1
	2024/04/26 18:32:21 [emerg] 26#26: unknown "geoip_country_code" variable
	nginx: [emerg] unknown "geoip_country_code" variable
	nginx: configuration file /tmp/nginx/nginx-cfg3188325810 test failed
	-------------------------------------------------------------------------------
 > key="nginx-private"
I0426 18:32:21.713799       7 event.go:364] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"nginx-private-controller-5bd766c8df-7lq2l", UID:"ff1ec4af-1ab4-4846-a17f-a4b946d394b4", APIVersion:"v1", ResourceVersion:"56814629", FieldPath:""}): type: 'Warning' reason: 'RELOAD' Error reloading NGINX:
-------------------------------------------------------------------------------
Error: exit status 1
2024/04/26 18:32:21 [emerg] 26#26: unknown "geoip_country_code" variable
nginx: [emerg] unknown "geoip_country_code" variable
nginx: configuration file /tmp/nginx/nginx-cfg3188325810 test failed
-------------------------------------------------------------------------------
I0426 18:32:25.023930       7 controller.go:190] "Configuration changes detected, backend reload required"
E0426 18:32:25.045902       7 controller.go:205] Unexpected failure reloading the backend:
-------------------------------------------------------------------------------
Error: exit status 1

How to reproduce this issue:

These should suffice to install working and non-working versions:

$ cat Chart.yaml. 
apiVersion: argoproj.io/v1alpha1
kind: application
name: nginx-internal
version: 0.0.1
dependencies:
- name: ingress-nginx
  alias: nginx-internal
  # https://github.com/kubernetes/ingress-nginx?tab=readme-ov-file#supported-versions-table
  version: 4.10.0
  repository: https://kubernetes.github.io/ingress-nginx

$ cat values.yaml
nginx-internal:
  controller:
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:444444444:certificate/eca520df-333333-3333

helm dependency build && helm template --name-template foo --namespace foo --values values.yaml . --debug > foo-4.10.0

kubectl apply -f foo-4.10.0 # you should see the pod under the RS in a crashloopbackoff state

kubectl delete -f foo-4.10.0

$ downgrade Chart.yaml to 4.9.1
helm dependency build && helm template --name-template foo --namespace foo --values values.yaml . --debug > foo-4.9.1
 kubectl apply -f foo.4.9.1 # all resources should be created successfully
kubectl delete -f foo-4.9.1 # final clean-up

Anything else we need to know:

Checking recent issues, this appears to be the only close complaint: #11254. That said, the versions are different. They're on controller-1.9.4 and a bump to 1.9.6 fixes their issue. I did not try providing an emptydir for geoip configuration, or other stub files, as he did.

** attempted work-arounds **
I tried to alternate specifcations:

in this attempt, I added the following to values.yaml:

config:
  use-geoip: "false"
  use-geoip2: "false"

the chart still failed with crashloopbackoff

in this attempt, I added the following to values.yaml:

 extraArgs:
   enable-metrics: true

the chart still failed with crashloopbackoff

The text was updated successfully, but these errors were encountered:

longwuyuan · 2024-04-27T07:25:01Z

/remove-kind bug

Lets add the bug label after triaging is completed

You have been succint and cryptic in your issue description. It does not help
Please answer the questions that are asked in a new issue template exactly as is because that info is input for a reader to understand and reproduce
For example, the simple question is how did you install so the expectation is you copy paste the exact command executed and the complete values file used. In your case, since you use ArgoCD, then it is expected that at least the complete values as is from the original install be presented as is. Providing the later modifications is great
At this stage for a problem like this, its not going to be possible to test ArgoCD in the CI. So please do a helm install command using the same values file and update the results. This will know if using ArgoCD introduces the problem or not

/triage needs-information
/kind support

ducnm0711 · 2024-04-27T17:47:49Z

Noted same issue - bump from helm-chart 4.6.1 to 4.10.0.
Update: chart 4.9.1 doesn't have this issue.
My current config:

controller:
  kind: DaemonSet
  maxmindLicenseKey: change-me
  config:
    use-geoip2: "true"
    log-format-escape-json: "true"
    log-format-upstream: '{
      ....
      "geoip_country_code": "$geoip_country_code"}'

It's appear that use-geoip2: "true" is not taken into nginx config, therefore throw error for additional log label geoip_country_code

2024/04/27 18:12:36 [emerg] 376#376: unknown "geoip_country_code" variable
nginx: [emerg] unknown "geoip_country_code" variable

longwuyuan · 2024-04-27T18:13:12Z

Any chance you can try to reproduce this problem on a minikube cluster or a kind cluster but only with geoip2 enabled and no other customization

ducnm0711 · 2024-04-27T18:23:41Z

Hi @longwuyuan
Thank you for replying.
Above is the minimum values.yaml config to reproduce this issue.

longwuyuan · 2024-04-27T18:27:51Z

Thanks. Can you ping me on slack. I am trying to figure out if it can also be reproduced only and only with geop2 enabled and no other customization.

longwuyuan · 2024-04-27T18:30:03Z

/remove-kind support
/kind bug
/triage accepted

longwuyuan · 2024-04-27T18:54:24Z

@ducnm0711 I don't have a licence to test so can you change the variable name and test

leev/ngx_http_geoip2_module#92 (comment)

We have removed the non geoip2 components but it will be a least effort test to do this. thanks

longwuyuan · 2024-04-27T19:12:06Z

/assign

longwuyuan · 2024-04-28T00:40:15Z

/retitle geoip2 enabling causes crash of controller v1.10

longwuyuan · 2024-04-28T00:42:53Z

i saw that there is a lite database for free so I will attempt to reproduce on minikube. meanwhile if you can also confirm that no variable no daemonset and no other customization, just enable geoip2, crashes the controller. If you have to use variable, then at least I will try to reproduce with var name as geop2_data_country_code , as shown in the docs and examples of geoip2 module

longwuyuan · 2024-04-28T00:43:40Z

cc @rikatz

longwuyuan · 2024-04-29T01:30:24Z

I don't get a crash when I install without the variable you used like this (licensekey redacted)

helm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace --set controller.maxmindLicenseKey=dslfhdfddOIUJJDFKDF&DSFDlkf --set controller.config.use-geoip2=true

I see the flag in the pod

k -n ingress-nginx describe po ingress-nginx-controller-7878f4b84-z5bxs | grep "Args" -A 10                     
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --maxmind-license-key=dslfhdfddOIUJJDFKDF&DSFDlkf

And I can see geoip2 in nginx.conf

% k -n ingress-nginx exec ingress-nginx-controller-7878f4b84-z5bxs -- cat /etc/nginx/nginx.conf | grep -i geoip2
load_module /etc/nginx/modules/ngx_http_geoip2_module.so;
        # https://github.com/leev/ngx_http_geoip2_module#example-usage
        geoip2 /etc/ingress-controller/geoip/GeoLite2-City.mmdb {
                $geoip2_city_country_code source=$remote_addr country iso_code;
                $geoip2_city_country_name source=$remote_addr country names en;
                $geoip2_city_country_geoname_id source=$remote_addr country geoname_id;
                $geoip2_city source=$remote_addr city names en;
                $geoip2_city_geoname_id source=$remote_addr city geoname_id;
                $geoip2_postal_code source=$remote_addr postal code;
                $geoip2_dma_code source=$remote_addr location metro_code;
                $geoip2_latitude source=$remote_addr location latitude;
                $geoip2_longitude source=$remote_addr location longitude;
                $geoip2_time_zone source=$remote_addr location time_zone;
                $geoip2_region_code source=$remote_addr subdivisions 0 iso_code;
                $geoip2_region_name source=$remote_addr subdivisions 0 names en;
                $geoip2_region_geoname_id source=$remote_addr subdivisions 0 geoname_id;
                $geoip2_subregion_code source=$remote_addr subdivisions 1 iso_code;
                $geoip2_subregion_name source=$remote_addr subdivisions 1 names en;
                $geoip2_subregion_geoname_id source=$remote_addr subdivisions 1 geoname_id;
                $geoip2_city_continent_code source=$remote_addr continent code;
                $geoip2_city_continent_name source=$remote_addr continent names en;one
        geoip2 /etc/ingress-controller/geoip/GeoLite2-ASN.mmdb {
                $geoip2_asn source=$remote_addr autonomous_system_number;
                $geoip2_org source=$remote_addr autonomous_system_organization;

This means that the variable name is the root-cause of the crash as reported by your error-message
I will try to play with the variable name and report

longwuyuan · 2024-04-29T01:55:41Z

I added the variable I see in the example (not the variable name that you used)
And I did not see a crash. I also see logs after a curl request. It was local so no real-country-code in logs

% cat values.yaml 
controller:
  maxmindLicenseKey: dslfhdfddOIUJJDFKDF&DSFDlkf
  config:
    use-geoip2: "true"
    log-format-escape-json: "true"
    log-format-upstream: '{
      ....
      "geoip_country_code": "$geoip2_city_country_code"}'

%  helm -n ingress-nginx upgrade ingress-nginx ingress-nginx/ingress-nginx -f values.yaml

% k -n ingress-nginx describe cm ingress-nginx-controller                               
Name:         ingress-nginx-controller
Namespace:    ingress-nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.10.1
              helm.sh/chart=ingress-nginx-4.10.1
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx

Data
====
use-geoip2:
----
true
allow-snippet-annotations:
----
false
log-format-escape-json:
----
true
log-format-upstream:
----
{ .... "geoip_country_code": "$geoip2_city_country_code"}

BinaryData
====

%  k create deployment test --image nginx:alpine
%  k expose deployment test --port 80
%  k create ing test --class nginx --rule test.mydomain.com/"*"=test:80
%  curl test.mydomain.com --resolve test.mydomain.com:80:172.19.0.3
% k -n ingress-nginx logs ingress-nginx-controller-7878f4b84-z5bxs
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.10.1
  Build:         4fb5aac1dd3669daa3a14d9de3e3cdb371b4c518
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.3

-------------------------------------------------------------------------------

I0429 01:10:32.218651      13 flags.go:387] "downloading maxmind GeoIP2 databases"
W0429 01:10:40.846551      13 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0429 01:10:40.846722      13 main.go:205] "Creating API client" host="https://10.96.0.1:443"
I0429 01:10:40.851276      13 main.go:248] "Running in Kubernetes cluster" major="1" minor="29" git="v1.29.2" state="clean" commit="4b8e819355d791d96b7e9d9efe4cbafae2311c88" platform="linux/amd64"
I0429 01:10:40.967775      13 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0429 01:10:40.978880      13 ssl.go:535] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0429 01:10:40.985792      13 nginx.go:264] "Starting NGINX Ingress controller"
I0429 01:10:40.989446      13 event.go:364] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"086e8c4c-6857-43b6-b2bf-9f42efabc582", APIVersion:"v1", ResourceVersion:"1964", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0429 01:10:42.188540      13 nginx.go:307] "Starting NGINX process"
I0429 01:10:42.188619      13 leaderelection.go:250] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader...
I0429 01:10:42.188924      13 nginx.go:327] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0429 01:10:42.189106      13 controller.go:190] "Configuration changes detected, backend reload required"
I0429 01:10:42.198572      13 leaderelection.go:260] successfully acquired lease ingress-nginx/ingress-nginx-leader
I0429 01:10:42.198631      13 status.go:84] "New leader elected" identity="ingress-nginx-controller-7878f4b84-z5bxs"
I0429 01:10:42.229832      13 controller.go:210] "Backend successfully reloaded"
I0429 01:10:42.229888      13 controller.go:221] "Initial sync, sleeping for 1 second"
I0429 01:10:42.229941      13 event.go:364] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7878f4b84-z5bxs", UID:"6b7659ae-8b6b-402c-bfd7-9c03f51d33a7", APIVersion:"v1", ResourceVersion:"2062", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0429 01:37:04.378551      13 event.go:364] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"086e8c4c-6857-43b6-b2bf-9f42efabc582", APIVersion:"v1", ResourceVersion:"4392", FieldPath:""}): type: 'Normal' reason: 'UPDATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0429 01:37:04.381893      13 controller.go:190] "Configuration changes detected, backend reload required"
I0429 01:37:04.423304      13 controller.go:210] "Backend successfully reloaded"
I0429 01:37:04.423585      13 event.go:364] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7878f4b84-z5bxs", UID:"6b7659ae-8b6b-402c-bfd7-9c03f51d33a7", APIVersion:"v1", ResourceVersion:"2062", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0429 01:42:06.903116      13 main.go:107] "successfully validated configuration, accepting" ingress="default/test"
I0429 01:42:06.909966      13 store.go:440] "Found valid IngressClass" ingress="default/test" ingressclass="nginx"
I0429 01:42:06.910106      13 event.go:364] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test", UID:"6d0542c6-c1e7-4d3f-9b4f-d6090aa80e7c", APIVersion:"networking.k8s.io/v1", ResourceVersion:"4890", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0429 01:42:06.910378      13 controller.go:190] "Configuration changes detected, backend reload required"
I0429 01:42:06.958640      13 controller.go:210] "Backend successfully reloaded"
I0429 01:42:06.958806      13 event.go:364] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7878f4b84-z5bxs", UID:"6b7659ae-8b6b-402c-bfd7-9c03f51d33a7", APIVersion:"v1", ResourceVersion:"2062", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0429 01:42:42.204168      13 status.go:304] "updating Ingress status" namespace="default" ingress="test" currentValue=null newValue=[{"ip":"172.19.0.3"}]
I0429 01:42:42.209251      13 event.go:364] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test", UID:"6d0542c6-c1e7-4d3f-9b4f-d6090aa80e7c", APIVersion:"networking.k8s.io/v1", ResourceVersion:"4946", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
{ .... "geoip_country_code": ""}
{ .... "geoip_country_code": ""}
[~]

longwuyuan · 2024-04-29T02:11:21Z

Finally when I use the variable name you used, I can reproduce the crash
values.yaml

% cat values.yaml 
controller:
  maxmindLicenseKey: dslfdfkjlkfdhdlkfhdlkhfdklhfKDF&DSFDlkf
  config:
    use-geoip2: "true"
    log-format-escape-json: "true"
    log-format-upstream: '{
      ....
      "geoip_country_code": "$geoip_country_code"}'
[~/Documents/ingressnnginx/issues/11320] 
% k -n ingress-nginx describe cm ingress-nginx-controller 
Name:         ingress-nginx-controller
Namespace:    ingress-nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.10.1
              helm.sh/chart=ingress-nginx-4.10.1
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx

Data
====
allow-snippet-annotations:
----
false
log-format-escape-json:
----
true
log-format-upstream:
----
{ .... "geoip_country_code": "$geoip_country_code"}
use-geoip2:
----
true

BinaryData
====

logs

 > key="ingress-nginx/ingress-nginx-controller-6j598"
I0429 02:05:02.205347      13 event.go:364] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7878f4b84-hjrth", UID:"3be1bafb-e7cc-4cb3-9c75-684b4556f28c", APIVersion:"v1", ResourceVersion:"6944", FieldPath:""}): type: 'Warning' reason: 'RELOAD' Error reloading NGINX: 
-------------------------------------------------------------------------------
Error: exit status 1
2024/04/29 02:05:02 [emerg] 59#59: unknown "geoip_country_code" variable
nginx: [emerg] unknown "geoip_country_code" variable
nginx: configuration file /tmp/nginx/nginx-cfg1742803439 test failed

-------------------------------------------------------------------------------
I0429 02:05:05.503847      13 controller.go:190] "Configuration changes detected, backend reload required"
E0429 02:05:05.533550      13 controller.go:205] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: exit status 1
2024/04/29 02:05:05 [emerg] 60#60: unknown "geoip_country_code" variable
nginx: [emerg] unknown "geoip_country_code" variable
nginx: configuration file /tmp/nginx/nginx-cfg3109804287 test failed

-------------------------------------------------------------------------------
E0429 02:05:05.533596      13 queue.go:131] "requeuing" err=<

        -------------------------------------------------------------------------------
        Error: exit status 1
        2024/04/29 02:05:05 [emerg] 60#60: unknown "geoip_country_code" variable
        nginx: [emerg] unknown "geoip_country_code" variable
        nginx: configuration file /tmp/nginx/nginx-cfg3109804287 test failed

        -------------------------------------------------------------------------------
 > key="ingress-nginx/ingress-nginx-controller-admission-vbpg5"
I0429 02:05:05.533673      13 event.go:364] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7878f4b84-hjrth", UID:"3be1bafb-e7cc-4cb3-9c75-684b4556f28c", APIVersion:"v1", ResourceVersion:"6944", FieldPath:""}): type: 'Warning' reason: 'RELOAD' Error reloading NGINX: 
-------------------------------------------------------------------------------
Error: exit status 1
2024/04/29 02:05:05 [emerg] 60#60: unknown "geoip_country_code" variable
nginx: [emerg] unknown "geoip_country_code" variable
nginx: configuration file /tmp/nginx/nginx-cfg3109804287 test failed

So please change the variable name

/remove-kind bug
/remove-triage accepted
/remove-triage needs-information
/kind support

longwuyuan · 2024-04-29T02:17:22Z

@jlm0x017 Please re-open the issue if you find a problem with the controller. For now I will close the issue as there is no problem found in the controller. Problem is just the variable name is invalid

/close

k8s-ci-robot · 2024-04-29T02:17:26Z

@longwuyuan: Closing this issue.

In response to this:

@jlm0x017 Please re-open the issue if you find a problem with the controller. For now I will close the issue as there is no problem found in the controller. Problem is just the variable name is invalid

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jlm0x017 · 2024-04-29T22:40:37Z

@longwuyuan Thanks for diving into this. You're exactly right, the variable name was being used in 'log-format-upstream:'; it was an artifact sticking around from prior versions. I identified where this was being set and removed it. 4.10.0+ are running just fine.

mykaua · 2024-08-09T14:47:07Z

Resolved the issue by updating maxmind license key

Pilotindream · 2024-08-20T10:14:07Z

Hello everyone. I faced same issue during upgrading to chart 4.11.2.
May someone explain how to resolve issue. Am i right that I can use the key that you provided here: maxmindLicenseKey: dslfhdfddOIUJJDFKDF&DSFDlkf or it should be generated somewhere?
Also in default values for chart 4.11.2 I don`t see variable use-geoip2 so I am not aware whether I can use it?

mykaua · 2024-08-21T06:59:17Z

@Pilotindream You may recreate maxmindLicesnsekey(https://support.maxmind.com/hc/en-us/articles/4407111582235-Generate-a-License-Key)
Please don't share any tokens or keys with the public.

Here is my config, for example:
config: use-gzip: "true" enable-brotli: "true" use-http2: "true" use-geoip: "false" use-geoip2: "true" log-format-escape-json: "true" log-format-upstream: |- {json code for logs output } enable-modsecurity: true enable-owasp-modsecurity-crs: true

jlm0x017 added the kind/bug Categorizes issue or PR as related to a bug. label Apr 26, 2024

k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority labels Apr 26, 2024

k8s-ci-robot added triage/needs-information Indicates an issue needs more information in order to work on it. kind/support Categorizes issue or PR as a support question. and removed kind/bug Categorizes issue or PR as related to a bug. labels Apr 27, 2024

k8s-ci-robot assigned longwuyuan Apr 27, 2024

k8s-ci-robot changed the title ~~helm-chart 4.10.0 (and 4.10.1) versions of ingress-nginx-controller pods stuck in crashloopbackoff with complaints about geoip configuration at the nginx level.~~ geoip2 enabling causes crash of controller v1.10 Apr 28, 2024

k8s-ci-robot removed the triage/needs-information Indicates an issue needs more information in order to work on it. label Apr 29, 2024

k8s-ci-robot closed this as completed Apr 29, 2024

longwuyuan mentioned this issue Aug 6, 2024

Having both libmaxminddb-dev and geoip-dev in Ingress-Nginx makes geo database not work with ModSecurity #11738

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

geoip2 enabling causes crash of controller v1.10 #11320

geoip2 enabling causes crash of controller v1.10 #11320

jlm0x017 commented Apr 26, 2024 •

edited

Loading

longwuyuan commented Apr 27, 2024

ducnm0711 commented Apr 27, 2024 •

edited

Loading

longwuyuan commented Apr 27, 2024 •

edited

Loading

ducnm0711 commented Apr 27, 2024

longwuyuan commented Apr 27, 2024

longwuyuan commented Apr 27, 2024

longwuyuan commented Apr 27, 2024 •

edited

Loading

longwuyuan commented Apr 27, 2024

longwuyuan commented Apr 28, 2024

longwuyuan commented Apr 28, 2024

longwuyuan commented Apr 28, 2024

longwuyuan commented Apr 29, 2024

longwuyuan commented Apr 29, 2024 •

edited

Loading

longwuyuan commented Apr 29, 2024 •

edited

Loading

longwuyuan commented Apr 29, 2024

k8s-ci-robot commented Apr 29, 2024

jlm0x017 commented Apr 29, 2024

mykaua commented Aug 9, 2024

Pilotindream commented Aug 20, 2024

mykaua commented Aug 21, 2024

geoip2 enabling causes crash of controller v1.10 #11320

geoip2 enabling causes crash of controller v1.10 #11320

Comments

jlm0x017 commented Apr 26, 2024 • edited Loading

version from running 4.9.1 helm chart: $ /nginx-ingress-controller --version

longwuyuan commented Apr 27, 2024

ducnm0711 commented Apr 27, 2024 • edited Loading

longwuyuan commented Apr 27, 2024 • edited Loading

ducnm0711 commented Apr 27, 2024

longwuyuan commented Apr 27, 2024

longwuyuan commented Apr 27, 2024

longwuyuan commented Apr 27, 2024 • edited Loading

longwuyuan commented Apr 27, 2024

longwuyuan commented Apr 28, 2024

longwuyuan commented Apr 28, 2024

longwuyuan commented Apr 28, 2024

longwuyuan commented Apr 29, 2024

longwuyuan commented Apr 29, 2024 • edited Loading

longwuyuan commented Apr 29, 2024 • edited Loading

longwuyuan commented Apr 29, 2024

k8s-ci-robot commented Apr 29, 2024

jlm0x017 commented Apr 29, 2024

mykaua commented Aug 9, 2024

Pilotindream commented Aug 20, 2024

mykaua commented Aug 21, 2024

jlm0x017 commented Apr 26, 2024 •

edited

Loading

version from running 4.9.1 helm chart:
$ /nginx-ingress-controller --version

ducnm0711 commented Apr 27, 2024 •

edited

Loading

longwuyuan commented Apr 27, 2024 •

edited

Loading

longwuyuan commented Apr 27, 2024 •

edited

Loading

longwuyuan commented Apr 29, 2024 •

edited

Loading

longwuyuan commented Apr 29, 2024 •

edited

Loading