k8s prometheus pod crashed with the image built from the master branch

[reachlin]

What did you do?
trying to run prometheus build verification on k8s

What did you expect to see?
prometheus pod is up and running

What did you see instead? Under which circumstances?
prometheus pod crashed. but the same code works fine on local ubuntu box or Travis with the latest image from quay.io/coreos/prometheus. so there must be sth. wrong with the image built from the master branch?

https://travis-ci.org/reachlin/prometheus/builds/243513392

$ docker logs $uuid2 2>&1

panic: standard_init_linux.go:175: exec user process caused "no such file or directory" [recovered]

	panic: standard_init_linux.go:175: exec user process caused "no such file or directory"

goroutine 1 [running, locked to thread]:

panic(0x7ddea0, 0xc820115500)

	/usr/local/go/src/runtime/panic.go:481 +0x3e6

github.com/urfave/cli.HandleAction.func1(0xc8200ad2f8)

	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x38e

panic(0x7ddea0, 0xc820115500)

	/usr/local/go/src/runtime/panic.go:443 +0x4e9

github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc8200acc08, 0xc82001a018, 0xc8200acd18)

	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x136

github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc820065630, 0x7fab75d006c0, 0xc820115500)

	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x5b1

main.glob.func8(0xc820086a00, 0x0, 0x0)

	/go/src/github.com/opencontainers/runc/main_unix.go:26 +0x68

reflect.Value.call(0x7448a0, 0x8f0bd0, 0x13, 0x839998, 0x4, 0xc8200ad278, 0x1, 0x1, 0x0, 0x0, ...)

	/usr/local/go/src/reflect/value.go:435 +0x120d

reflect.Value.Call(0x7448a0, 0x8f0bd0, 0x13, 0xc8200ad278, 0x1, 0x1, 0x0, 0x0, 0x0)

	/usr/local/go/src/reflect/value.go:303 +0xb1

github.com/urfave/cli.HandleAction(0x7448a0, 0x8f0bd0, 0xc820086a00, 0x0, 0x0)

	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x2ee

github.com/urfave/cli.Command.Run(0x83c828, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8d0180, 0x51, 0x0, ...)

	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xfec

github.com/urfave/cli.(*App).Run(0xc820001200, 0xc820062060, 0x2, 0x2, 0x0, 0x0)

	/go/src/github.com/opencontainers/runc/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0xaa4

main.main()

	/go/src/github.com/opencontainers/runc/main.go:137 +0xe24

Environment

   Travis

• System information:

  Travis

• Prometheus version:

  https://github.com/reachlin/prometheus/blob/travis/VERSION

• Alertmanager version:

  n/a

• Prometheus configuration file:

https://github.com/reachlin/prometheus/blob/travis/documentation/examples/prometheus.yml

• Alertmanager configuration file:
  n/a

• Logs:
  https://travis-ci.org/reachlin/prometheus/builds/243513392

[reachlin]

i verified using the stable prometheus image pulled, the pod is up on Travis logs, but build from the master branch still fails

$ docker ps -a

CONTAINER ID        IMAGE                                                                                                           COMMAND                  CREATED             STATUS              PORTS               NAMES

eda9c996b962        quay.io/coreos/prometheus@sha256:da23334bab007b01790fe7437276302d2f25b94452078bb0484623a8c1c1fc0e               "/bin/prometheus -con"   10 seconds ago      Up 9 seconds                            k8s_prometheus_prometheus-994847654-7s8qn_default_6d31149d-5244-11e7-997f-42010a0a1417_0

[reachlin]

on local box, the master branch image also crashed with similar errors:

lincai@montana:~/go/src/github.com/prometheus/prometheus$ make k8s 
...
Removing intermediate container de2aa85b7ff7
Successfully built b7f16b4de978
>> running in k8s on Travis
configmap "prometheus" created
service "prometheus" created
deployment "prometheus" created
lincai@montana:~/go/src/github.com/prometheus/prometheus$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                        READY     STATUS             RESTARTS   AGE
default       prometheus-994847654-b23nb                  0/1       CrashLoopBackOff   4          1m
...
lincai@montana:~/go/src/github.com/prometheus/prometheus$ docker ps -a|grep prometheus
7c36c4b6ca58        b7f16b4de978                                                                                                    "/bin/prometheus -..."   37 seconds ago      Exited (1) 36 seconds ago                       k8s_prometheus_prometheus-994847654-b23nb_default_5455ccd2-525e-11e7-b6a9-080027200f75_4
fd9cbec18287        pause-amd64:3.0                                                                                                 "/pause"                 2 minutes ago       Up 2 minutes                                    k8s_POD_prometheus-994847654-b23nb_default_5455ccd2-525e-11e7-b6a9-080027200f75_0
lincai@montana:~/go/src/github.com/prometheus/prometheus$ docker logs 7c36c4b6ca58
standard_init_linux.go:178: exec user process caused "no such file or directory"

[reachlin]

i did further investigation and login to that container, and found a very strange problem:

/prometheus # /bin/prometheus -config.file=/etc/prometheus/prometheus.yml
sh: /bin/prometheus: not found
/prometheus # ls -al /bin/prometheus 
-rwxrwxr-x    1 root     root      61239613 Jun 16 06:37 /bin/prometheus
/prometheus # /bin/prometheus 
sh: /bin/prometheus: not found
/prometheus # ls /etc/prometheus/prometheus.yml 
/etc/prometheus/prometheus.yml
/prometheus # 

[matthiasr]

This could be a missing dynamic library

…

On Fri, Jun 16, 2017, 09:07 Charles ***@***.***> wrote: i did further investigation and login to that container, and found a very strange problem: /prometheus # /bin/prometheus -config.file=/etc/prometheus/prometheus.yml sh: /bin/prometheus: not found /prometheus # ls -al /bin/prometheus -rwxrwxr-x 1 root root 61239613 Jun 16 06:37 /bin/prometheus /prometheus # /bin/prometheus sh: /bin/prometheus: not found /prometheus # ls /etc/prometheus/prometheus.yml /etc/prometheus/prometheus.yml /prometheus # — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2848 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAICBqDSc2rmVB_zvwhS-4fNRMHXa_YUks5sEinIgaJpZM4N7-ga> .

[reachlin]

@matthiasr very likely, because if i run the same binary from my host, not in that container, it seems fine, so i guess, sth. changed in the parent image or lib depended?

[matthiasr]

or the library exists on your host? I'll take a look later

…

On Fri, Jun 16, 2017, 09:16 Charles ***@***.***> wrote: @matthiasr <https://github.com/matthiasr> very likely, because if i run the same binary from my host, not in that container, it seems fine, so i guess, sth. changed in the parent image or lib depended? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2848 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAICBhMEJlIiRPaTBYfIGaOjccNSykZSks5sEivYgaJpZM4N7-ga> .

[brancz]

The binaries built on release are statically linked binaries using promu crossbuild. That is also the binary that is then put into the container image. I've seen these errors before when just building the binary with the build make target and then putting it into a container.

[matthiasr]

so, I'm confused what image the issue is with. quay.io/coreos/prometheus is ancient (Prometheus 1.1.1). But from the linked Travis build it seems you're actually building your own images and these don't work? Is running Docker and K8s on top of Travis CI supported at all? IIRC we use CircleCI because it is not.

[matthiasr]

The official image works just fine:

% docker run quay.io/prometheus/prometheus:master --version
Unable to find image 'quay.io/prometheus/prometheus:master' locally
master: Pulling from prometheus/prometheus
4b0bc1c4050b: Already exists
a3ed95caeb02: Pull complete
d6ab6c75ce17: Pull complete
7c778741ebcf: Pull complete
c1904f8c4b54: Pull complete
c75af369dfe3: Pull complete
ffde608c21ef: Pull complete
bc35b50df43c: Pull complete
f71ae0ab7bff: Pull complete
Digest: sha256:a2372bff444a6e9a4bf4ec7cfdf7c3a83b27fece486faac407491424ecd26b84
Status: Downloaded newer image for quay.io/prometheus/prometheus:master
prometheus, version 1.7.0 (branch: master, revision: 9ba61df45aeea9808d5ccd57a0dd9d64833f174f)
  build user:       root@801638fbcd49
  build date:       20170612-14:17:51
  go version:       go1.8.3

I'm afraid the issue is with your build, not the mainline one.

[reachlin]

@matthiasr so the image build from 'make docker' won't work? did you try image build from the clone of master branch? I made k8s on Travis working and fine with other containers, check it here: https://github.com/reachlin/k8s0/

[brancz]

@reachlin the docker make target just uses whatever prometheus binary is in the root directory. The binary built with the build make target does not produce a statically linked binary, which is expected for the binary in the docker image. I recommend running promu crossbuild to build the statically linked binary as we do for releases. Then move the respective binary from .build/linux-amd64/prometheus to the root of the repo and then build the docker image with the docker make target.

[reachlin]

@brancz thanks, i will try that. please update the readme if possible, it's confusing

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.