New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nginx-ingress-controller crashes with SIGSEGV #3545

Open
addieter opened this Issue Dec 11, 2018 · 3 comments

Comments

Projects
None yet
3 participants
@addieter
Copy link

addieter commented Dec 11, 2018

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):


Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

NGINX Ingress controller version:
NGINX Ingress controller
Release: 0.20.0
Build: git-e8d8103
Repository: https://github.com/kubernetes/ingress-nginx.git
Image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller-arm:0.20.0

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.4", GitCommit:"bf9a868e8ea3d3a8fa53cbb22f566771b3f8068b", GitTreeState:"clean", BuildDate:"2018-10-25T19:06:30Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/arm"}

Environment:

  • Cloud provider or hardware configuration: Raspberry Pi 3 Model B+
  • OS (e.g. from /etc/os-release): Raspbian GNU/Linux 9 (stretch
  • Kernel (e.g. uname -a): Linux 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU/Linux
  • Install tools:
    Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
    Server: &version.Version{SemVer:"v2.11.0+unreleased", GitCommit:"158d6dbb746f525bad9a0aacb698af7d370ac3f5", GitTreeState:"dirty"}
  • Others:
    • Docker version 18.04.0-ce, build 3d479c0
    • metallb/controller:v0.7.3-arm

What happened:
The nginx-ingress-controller pod crashes with segfault after successful request to service.

  1. kubectl get pod nginx-ingress-controller-59cc49954d-b5kch
    nginx-ingress-controller-59cc49954d-b5kch   1/1       Running   26        3d
    
  2. Request to service
    curl --verbose ranger/towel/helloworld --header "Host: ranger"
    *   Trying 2003:c8:9f2e:2100:15d8:1b9f:5a9c:b715...
    * TCP_NODELAY set
    * Connection failed
    * connect to 2003:c8:9f2e:2100:15d8:1b9f:5a9c:b715 port 80 failed: Connection refused
    *   Trying 192.168.178.77...
    * TCP_NODELAY set
    * Connected to ranger (192.168.178.77) port 80 (#0)
    > GET /towel/helloworld HTTP/1.1
    > Host: ranger
    > User-Agent: curl/7.54.0
    > Accept: */*
    >
    < HTTP/1.1 200
    < Server: nginx/1.15.5
    < Date: Tue, 11 Dec 2018 07:46:08 GMT
    < Content-Type: application/json;charset=UTF-8
    < Transfer-Encoding: chunked
    < Connection: keep-alive
    < Vary: Accept-Encoding
    <
    * Connection #0 to host ranger left intact
    {"id":14,"message":"Hello, world!"}%
    
  3. nginx-ingress-controller crashes

kubectl get pod nginx-ingress-controller-59cc49954d-b5kc:

nginx-ingress-controller-59cc49954d-b5kch   0/1       Error     27        3d

kubectl logs nginx-ingress-controller-59cc49954d-b5kch:

I1211 07:46:20.803405       8 flags.go:180] Watching for Ingress class: nginx
nginx version: nginx/1.15.5
W1211 07:46:20.814264       8 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1211 07:46:20.817531       8 main.go:196] Creating API client for https://10.96.0.1:443
I1211 07:46:20.947391       8 main.go:240] Running in Kubernetes cluster version v1.11 (v1.11.4) - git (clean) commit bf9a868e8ea3d3a8fa53cbb22f566771b3f8068b - platform linux/arm
I1211 07:46:20.954249       8 main.go:101] Validated default/nginx-ingress-default-backend as the default backend.
I1211 07:46:23.600322       8 nginx.go:256] Starting NGINX Ingress controller
I1211 07:46:23.668833       8 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"default", Name:"nginx-ingress-controller", UID:"9dc046fd-fa2a-11e8-bc35-b827eb5c793c", APIVersion:"v1", Re
sourceVersion:"150947", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap default/nginx-ingress-controller
I1211 07:46:24.710633       8 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"towel-ingress", UID:"23aabc05-fc87-11e8-bc35-b827eb5c793c", APIVersion:"extensions/v1beta1",
 ResourceVersion:"573396", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/towel-ingress
I1211 07:46:24.801389       8 leaderelection.go:185] attempting to acquire leader lease  default/ingress-controller-leader-nginx...
I1211 07:46:24.801457       8 nginx.go:277] Starting NGINX process
I1211 07:46:24.813672       8 controller.go:177] Configuration changes detected, backend reload required.
I1211 07:46:24.825633       8 leaderelection.go:194] successfully acquired lease default/ingress-controller-leader-nginx
I1211 07:46:24.825894       8 status.go:197] new leader elected: nginx-ingress-controller-59cc49954d-b5kch
I1211 07:46:25.298432       8 controller.go:195] Backend successfully reloaded.
I1211 07:46:25.298820       8 controller.go:205] Initial synchronization of the NGINX configuration.
I1211 07:46:26.309819       8 controller.go:212] Dynamic reconfiguration succeeded.
W1211 07:46:50.597452       8 controller.go:824] Service "default/towel-tool" does not have any active Endpoint.
I1211 07:46:50.597887       8 controller.go:177] Configuration changes detected, backend reload required.
I1211 07:46:51.053256       8 controller.go:195] Backend successfully reloaded.
I1211 07:46:51.057831       8 controller.go:212] Dynamic reconfiguration succeeded.
W1211 07:47:18.153682       8 controller.go:824] Service "default/towel-tool" does not have any active Endpoint.
I1211 07:47:21.487755       8 controller.go:177] Configuration changes detected, backend reload required.
I1211 07:47:21.939385       8 controller.go:195] Backend successfully reloaded.
I1211 07:47:21.946202       8 controller.go:212] Dynamic reconfiguration succeeded.
192.168.178.50 - [192.168.178.50] - - [11/Dec/2018:07:55:42 +0000] "GET /towel/helloworld HTTP/1.1" 200 46 "-" "curl/7.54.0" 86 0.022 [default-towel-tool-8080] 10.244.2.27:8080 46 0.020 200 93bea19734a3da7dc93776b3c9a5d2ba
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11a70]

goroutine 1308 [running]:
runtime/internal/atomic.goXadd64(0x38a652c, 0x2, 0x0, 0x20c49ba, 0x3f96872b)
        /usr/local/go/src/runtime/internal/atomic/atomic_arm.go:96 +0x1c
k8s.io/ingress-nginx/vendor/github.com/prometheus/client_golang/prometheus.(*histogram).Observe(0x38a64d0, 0x20c49ba, 0x3f96872b)
        /go/src/k8s.io/ingress-nginx/vendor/github.com/prometheus/client_golang/prometheus/histogram.go:272 +0x68
k8s.io/ingress-nginx/internal/ingress/metric/collectors.(*SocketCollector).handleMessage(0x38f7c00, 0x407ac00, 0x146, 0x600)
        /go/src/k8s.io/ingress-nginx/internal/ingress/metric/collectors/socket.go:269 +0xb8c
k8s.io/ingress-nginx/internal/ingress/metric/collectors.(*SocketCollector).handleMessage-fm(0x407ac00, 0x146, 0x600)
        /go/src/k8s.io/ingress-nginx/internal/ingress/metric/collectors/socket.go:317 +0x34
k8s.io/ingress-nginx/internal/ingress/metric/collectors.handleMessages(0x66c40d00, 0x3d3adb8, 0x3d3adc0)
        /go/src/k8s.io/ingress-nginx/internal/ingress/metric/collectors/socket.go:435 +0xa8
created by k8s.io/ingress-nginx/internal/ingress/metric/collectors.(*SocketCollector).Start
        /go/src/k8s.io/ingress-nginx/internal/ingress/metric/collectors/socket.go:317 +0xc0
  1. Controller then restarts and runs until next request
nginx-ingress-controller-59cc49954d-b5kch   0/1       CrashLoopBackOff   27        3d
nginx-ingress-controller-59cc49954d-b5kch   0/1       Running   28        3d
nginx-ingress-controller-59cc49954d-b5kch   1/1       Running   28        3d

What you expected to happen:
The controller not to crash.

How to reproduce it (as minimally and precisely as possible):

ingress.yml used:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: towel-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - host: ranger
    http:
      paths:
        - path: /towel
          backend:
            serviceName: towel-tool
            servicePort: 8080

Service:

apiVersion: v1
kind: Service
metadata:
  name: towel-tool
  labels:
    app: towel-tool
spec:
  ports:
    - port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    app: towel-tool

Deployment:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: towel-tool
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: towel-tool
    spec:
      containers:
      - name: towel-tool
        image: aqube/towel-tool-arm:1.1-SNAPSHOT
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          protocol: TCP

The towel-tool image runs a spring app returning a hello world message at /helloworld.

Anything else we need to know:
Only happens when a host is specified in the ingress rule.

@aledbf aledbf added the kind/bug label Dec 14, 2018

@aledbf

This comment has been minimized.

Copy link
Member

aledbf commented Dec 14, 2018

@addieter please update to 0.21.0

@pvormittag

This comment has been minimized.

Copy link

pvormittag commented Dec 17, 2018

@aledbf - I'm experiencing the same issue, however there doesn't appear to be a 0.21.0 build for arm?

https://quay.io/repository/kubernetes-ingress-controller/nginx-ingress-controller-arm

@aledbf

This comment has been minimized.

Copy link
Member

aledbf commented Dec 22, 2018

@pvormittag that's correct. The luajit dependency (from openresty) only works on amd64 and arm64. I 've been trying to build from the original luajit source code but the NGINX compilation fails (because of lua-nginx-module)
If I cannot find a way to fix this, we will drop support for arm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment