Can't use stackdriver container

[polycaster]

Hi,

I have been trying to make this work for a while now and i've come to a few conclusions.

I can't run any of the containers present in your public registry. Both locally and inside the clusters attempting to run your container results in this:
standard_init_linux.go:190: exec user process caused "permission denied"

If I go and make the image myself, using make docker in the root directory, the generated container no longer has the error and I can start it up.

However, running your container from the gcloud console works and i'm able to get that one up straight away.

Docker info for local:

Containers: 56
 Running: 0
 Paused: 0
 Stopped: 56
Images: 1173
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version:  (expected: 468a545b9edcd5932818eb9de8e72413e616e86e)
runc version: N/A (expected: 69663f0bd4b60df09991c08812a60108003fa340)
init version: v0.18.0 (expected: fec3683b971d9c3ef73f284f176672c44b448662)
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-43-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 31.25GiB
Name: xxx
ID: HMKY:IVL6:OCB6:3OU5:PDDK:2XB7:2DJX:Q35N:D6DL:Z4ZB:JZBW:6DLF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Docker info for gcp console:

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.14.74+
Operating System: Debian GNU/Linux 9 (stretch) (containerized)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.656GiB
Name: cs-6000-devshell-vm-382718db-b6dd-4808-aaec-daaec04bd062
ID: 74SW:2GLN:6MQ2:IKIY:IGYQ:ZRRP:BSX3:JHHR:ALJO:LD3H:DLKH:SGI5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 https://eu-mirror.gcr.io/
Live Restore Enabled: false

The ideal scenario would be being able to simply pull your container and run it.
Have you seen this issue before?

Thanks.

[jkohen]

Thanks for the report. What commands are you using to run the container in the case where you get an error?

[polycaster]

The way I've been running it is Docker run $IMAGE_TAG

If it doesn't work it will output the string I've mentioned, if it works it outputs the help menu from the stackdriver sidecar binary

[jkohen]

What IMAGE_TAG are you using? If it's gcr.io/prometheus-to-sd/opencensus-operator, please try gcr.io/stackdriver-prometheus/opencensus-operator.

We recently corrected our documentation, but it takes some time for documentaiton changes to become public.

[polycaster]

I'm currently attempting to use your latest from the gcr repo:
gcr.io/stackdriver-prometheus/stackdriver-prometheus-sidecar@sha256:18a16114d7856e224697d2157e8791f07664d14ea39d61a625f1f1d7495aa9b9 which is tagged with release-0.3.1

I've also tried all other images present on that repo which also didn't work

[igorpeshansky]

Is this moby/moby#26495 ?

[polycaster]

@igorpeshansky The symptoms seem to be the same, but I have the difference that if I am the one building the container it then works.

I'm just trying to avoid builds from source and to use the official release version

[igorpeshansky]

Hmm, if your self-built container works and the official one doesn't, is there a difference in the docker history output for the two containers? Are you able to reproduce the error on a fresh GCE instance (just to rule out the possibility that there's something funky with your local Docker setup)?

[polycaster]

There are no differences other than some hashes and a slight size difference.

Local built container:

IMAGE                                                                     CREATED              CREATED BY                                                                                                                             SIZE                COMMENT
sha256:f085c70dd790397d08c38a46ce6ecdefca7c0dc35336c3202752ee3d1cbb230e   About a minute ago   /bin/sh -c #(nop)  ENTRYPOINT ["/bin/stackdriver-prometheus-sidecar"]                                                                  0B                  
sha256:09110fcdf3a2c92a61e01b42f8020dce2755ffa189cbeedde6429d9f8a3c5907   About a minute ago   /bin/sh -c #(nop)  EXPOSE 9091                                                                                                         0B                  
sha256:cb49f848f6024a9c19d8654b34510762b45d520d9d3b8a7635241e865316706f   About a minute ago   /bin/sh -c #(nop)  USER nobody                                                                                                         0B                  
sha256:5cd5c368ab1f6fcd37fe2011c52e747788168892bfac1d10727e356974bf2247   About a minute ago   /bin/sh -c #(nop) COPY file:f5029bb2cb7b96a7d581b1b2fe511953721121daa7a00d8fe83421aab7050034 in /bin/stackdriver-prometheus-sidecar    45.4MB              
sha256:efa93f4e70de7d08bb78963ebe7f55096c206f57da90df61bb47e6956a381a0a   6 days ago           /bin/sh -c #(nop)  LABEL maintainer=Stackdriver Engineering <engineering@stackdriver.com>                                              0B                  
sha256:773509dd27217af7c85ce2ba6e95e3b3d5b20575bb9fda29ae2ef70269834c25   3 weeks ago          /bin/sh -c #(nop) COPY dir:804962465d9e5fc189af7359027a4655d97e1d19685416af190f8a3ade612bbe in /                                       1.45MB              
<missing>                                                                 3 weeks ago          /bin/sh -c #(nop)  MAINTAINER The Prometheus Authors <prometheus-developers@googlegroups.com>                                          0B                  
<missing>                                                                 4 weeks ago          /bin/sh -c #(nop)  CMD ["sh"]                                                                                                          0B                  
<missing>                                                                 4 weeks ago          /bin/sh -c #(nop) ADD file:7ba585d92672740a4831cb5ebe70f1cc102065fe30b39339a48bfc0ce42e2c9e in /                                       1.2MB    

Image gcr.io/stackdriver-prometheus/stackdriver-prometheus:release-0.3.1:

sha256:02f33082bcb90012903fcca685352227b0a0dacd100ab3f67c202da23af5c15f   3 weeks ago         /bin/sh -c #(nop)  ENTRYPOINT ["/bin/stackdriver-prometheus-sidecar"]                                                                  0B                  
<missing>                                                                 3 weeks ago         /bin/sh -c #(nop)  EXPOSE 9091                                                                                                         0B                  
<missing>                                                                 3 weeks ago         /bin/sh -c #(nop)  USER nobody                                                                                                         0B                  
<missing>                                                                 3 weeks ago         /bin/sh -c #(nop) COPY file:962b134641f1d664c53397e5f8df1602ee21bbced026d45eaa5e150ff1eeb28b in /bin/stackdriver-prometheus-sidecar    45.4MB              
<missing>                                                                 12 months ago       /bin/sh -c #(nop)  LABEL maintainer=Stackdriver Engineering <engineering@stackdriver.com>                                              0B                  
<missing>                                                                 16 months ago       /bin/sh -c #(nop) ADD file:9ef349fb594fab97851f3bd8635373b325f796de5b07384fb34a04500c130272 in /                                       1.46MB              
<missing>                                                                 16 months ago       /bin/sh -c #(nop) MAINTAINER The Prometheus Authors <prometheus-developers@googlegroups.com>                                           0B                  
<missing>                                                                 17 months ago       /bin/sh -c #(nop)  CMD ["sh"]                                                                                                          0B                  
<missing>                                                                 17 months ago       /bin/sh -c #(nop) ADD file:645231abe6e10e7282a6e78b49723a3ba35b62741fc08228b4086ffb95128f98 in /                                       1.13MB    

At first I thought it was my local setup too, but i've got the same results from running it in a non-managed AWS based kubernetes cluster and in a GKE cluster in GCP.

I've also attempted getting someone else to try to run it, ended up with the same result

[polycaster]

Attempted spinning up different kinds of instances, results were the same:

Debian GNU/Linux 9 (stretch): Failed
Ubuntu 18.04: Failed
Container-Optimized OS 71-11151.71.0 stable: Failed

But, like I said previously, running it from the google cloud console will work.

This was done by running docker run gcr.io/stackdriver-prometheus/stackdriver-prometheus-sidecar@sha256:18a16114d7856e224697d2157e8791f07664d14ea39d61a625f1f1d7495aa9b9

[igorpeshansky]

The two histories are different: layer 4 ("COPY file") has a different file hash, layer 6 is a "COPY dir" in your build and an "ADD file" in the official build, and layer 9 ("ADD file") has a different file hash and size. My guess is that they are using a different base image, since the Dockerfile says latest.
This, to me, suggests that rebuilding the official image today would likely fix the problem. @jkohen, WDYT?

[polycaster]

Would it be worth locking the version of the base container and upgrading deliberately?

[jkohen]

@igorpeshansky sorry, that's not my area of expertise. Let's make sure that we can provide a reliable and predictable experience. Is that what other projects tend to do?

I originally copied the latest tag from Prometheus, assuming that it was a good practice.

[igorpeshansky]

It doesn't look like the base container we're using provides a nice way to track versions. For now, we'll simply rebuild from the latest to fix the issue.