gitlab-ce image cannot be executed from inside OpenShift

[jpechane]

Follow the procedure described here - https://blog.openshift.com/getting-any-docker-image-running-in-your-own-openshift-cluster/

The net result is that GitLab will start but then the pod crashes with a message in the log

================================================================================
Error executing action `restart` on resource 'service[gitlab-workhorse]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/chpst -u root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-workhorse ----
STDOUT: timeout: run: /opt/gitlab/service/gitlab-workhorse: (pid 275) 35s, got TERM
STDERR: 
---- End output of /opt/gitlab/embedded/bin/chpst -u root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-workhorse ----
Ran /opt/gitlab/embedded/bin/chpst -u root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitlab-workhorse returned 1

I even tried to give it privileged scc but the issue persists.
If I run the image via docker run then it starts correctly.

The same image runs in OSE 3.1 or in CDK flawlessly.

oc version
oc v3.2.0.3
kubernetes v1.2.0-origin-41-g91d3e75

docker version
Client:
 Version:         1.9.1
 API version:     1.21
 Package version: docker-1.9.1-19.el7.x86_64
 Go version:      go1.4.2
 Git commit:      185277d/1.9.1
 Built:           
 OS/Arch:         linux/amd64

Server:
 Version:         1.9.1
 API version:     1.21
 Package version: docker-1.9.1-19.el7.x86_64
 Go version:      go1.4.2
 Git commit:      185277d/1.9.1
 Built:           
 OS/Arch:         linux/amd64

[bparees]

you ran this command oadm policy add-scc-to-user anyuid -z default to grant anyuid permissions? I'm able to run the image with no problem on the latest origin master code.

[jpechane]

yes I did - just different project name

oc get scc anyuid -o yaml
users:
- system:serviceaccount:gitlab:default

And the error is different to one that happens when you do not run as a root

[bparees]

@jpechane can you make sure your node pulled the latest version of the image?

[jpechane]

pod

     from:
        kind: ImageStreamTag
        name: gitlab-ce:8.5.7-ce.0

imagestream

spec:
  dockerImageRepository: gitlab/gitlab-ce
  tags:
  - from:
      kind: DockerImage
      name: gitlab/gitlab-ce:8.5.5-ce.0
    name: 8.5.5-ce.0
  - from:
      kind: DockerImage
      name: gitlab/gitlab-ce:8.5.5-rc1.ce.0
    name: 8.5.5-rc1.ce.0
  - from:
      kind: DockerImage
      name: gitlab/gitlab-ce:8.5.6-ce.0
    name: 8.5.6-ce.0
  - from:
      kind: DockerImage
      name: gitlab/gitlab-ce:8.5.7-ce.0
    name: 8.5.7-ce.0
  - from:
      kind: DockerImage
      name: gitlab/gitlab-ce:latest
    name: latest

[bparees]

after some difficulties i got this running using:
privileged: true
anyuid

are you sure you successfully configured the pod to run as privileged?

here is my pod yaml:

apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      kubernetes.io/created-by: |
        {"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicationController","namespace":"p1","name":"gitlab-ce-4","uid":"f6a4a75c-f085-11e5-b086-507b9d27b5d9","apiVersion":"v1","resourceVersion":"5058"}}
      openshift.io/container.gitlab-ce.image.entrypoint: '["/assets/wrapper"]'
      openshift.io/deployment-config.latest-version: "4"
      openshift.io/deployment-config.name: gitlab-ce
      openshift.io/deployment.name: gitlab-ce-4
      openshift.io/generated-by: OpenShiftNewApp
      openshift.io/scc: privileged
    creationTimestamp: 2016-03-22T23:29:50Z
    generateName: gitlab-ce-4-
    labels:
      app: gitlab-ce
      deployment: gitlab-ce-4
      deploymentconfig: gitlab-ce
    name: gitlab-ce-4-trow3
    namespace: p1
    resourceVersion: "5070"
    selfLink: /api/v1/namespaces/p1/pods/gitlab-ce-4-trow3
    uid: f8d9cc36-f085-11e5-b086-507b9d27b5d9
  spec:
    containers:
    - image: gitlab/gitlab-ce@sha256:a06f9c7e25c5af13bfbda6ab83771c959f60a077e4da1c38ed9a3b8bf031a14b
      imagePullPolicy: Always
      name: gitlab-ce
      ports:
      - containerPort: 22
        protocol: TCP
      - containerPort: 80
        protocol: TCP
      - containerPort: 443
        protocol: TCP
      resources: {}
      securityContext:
        privileged: true
      terminationMessagePath: /dev/termination-log
      volumeMounts:
      - mountPath: /etc/gitlab
        name: gitlab-ce-volume-1
      - mountPath: /var/log/gitlab
        name: gitlab-ce-volume-2
      - mountPath: /var/opt/gitlab
        name: gitlab-ce-volume-3
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: default-token-rlurn
        readOnly: true
    dnsPolicy: ClusterFirst
    host: localhost.localdomain
    imagePullSecrets:
    - name: default-dockercfg-rwb0x
    nodeName: localhost.localdomain
    restartPolicy: Always
    securityContext: {}
    serviceAccount: default
    serviceAccountName: default
    terminationGracePeriodSeconds: 30
    volumes:
    - emptyDir: {}
      name: gitlab-ce-volume-1
    - emptyDir: {}
      name: gitlab-ce-volume-2
    - emptyDir: {}
      name: gitlab-ce-volume-3
    - name: default-token-rlurn
      secret:
        secretName: default-token-rlurn
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: 2016-03-22T23:29:52Z
      status: "True"
      type: Ready
    containerStatuses:
    - containerID: docker://4b0f0143a3863b06d44da49ebfc3923aa15c8fb72136af6a2b37c7fb43872e3d
      image: gitlab/gitlab-ce@sha256:a06f9c7e25c5af13bfbda6ab83771c959f60a077e4da1c38ed9a3b8bf031a14b
      imageID: docker://e0573a99c8c60377be4f947080b7db93ddc38b72e33d073a545eb3c22a28777f
      lastState: {}
      name: gitlab-ce
      ready: true
      restartCount: 0
      state:
        running:
          startedAt: 2016-03-22T23:29:52Z
    hostIP: 10.192.213.11
    phase: Running
    podIP: 172.17.0.6
    startTime: 2016-03-22T23:29:50Z
kind: List
metadata: {}

[jpechane]

Thanks my mistake. I put the serviceAccount into privileged SCC but set only runAsUser, not privileged. If I switch the container into privileged ode it works now. But it leads me to the question why it does not require privileged for OSE 3.1 and requires it for 3.2? What has changed?

[bparees]

I suspect it has to do w/ the work we've done to further lock down what can be done as root user within a container. @rhatdan or @smarterclayton might know more details about what permissions have been locked down for root users in non-privileged containers.

[smarterclayton]

I wonder if this is due to the CAP drop. We drop {"KILL", "MKNOD", "SYS_CHROOT"}. I would not expect gitlab (as a well functioning container) to do be doing some of the things it's doing. It's basically a VM at this point, expecting to run as root with no restrictions.

[rhatdan]

   CAP_KILL
          Bypass  permission  checks  for  sending  signals (see kill(2)).
          This includes use of the ioctl(2) KDSIGACCEPT operation.

This would only apply if the process was running as root trying to kill a non root process. I believe

MKNOD would block a container process from trying to create devnodes that are allowed in standard docker things like /dev/null, /dev/none ...

SYS_CHROOT prevents a priv process run runninc chroot /rootfs

Do we think gitlab-ce is doing any of these.

[smarterclayton]

They're calling "chpst -u root /opt/gitlab/embedded/bin/sv restart
/opt/gitlab/service/gitlab-workhorse" when they fail. That may very well
be KILL.

On Wed, Mar 23, 2016 at 3:57 PM, Daniel J Walsh notifications@github.com
wrote:

CAP_KILL
Bypass permission checks for sending signals (see kill(2)).
This includes use of the ioctl(2) KDSIGACCEPT operation.

This would only apply if the process was running as root trying to kill a
non root process. I believe

MKNOD would block a container process from trying to create devnodes that
are allowed in standard docker things like /dev/null, /dev/none ...

SYS_CHROOT prevents a priv process run runninc chroot /rootfs

Do we think gitlab-ce is doing any of these.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#8083 (comment)

[bparees]

@rhatdan @smarterclayton assuming it is the KILL perm, do we view this behavior as desirable, or does it perhaps indicate we should not be dropping KILL? (I assume we're dropping KILL to ensure the uid inside the container can't manage to kill host processes if it escaped the container)