Doesn't work on OpenShift when using the restricted SCC

[rakesh-4321]

$helm install --name foo stable/ibm-mqadvanced-server-dev --set license=accept
$ helm status foo
LAST DEPLOYED: Tue Aug 21 16:17:24 2018
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
foo-ibm-mq-0 0/1 CrashLoopBackOff 5 7m

Container keeps restarting and the status stays "CrashLoopBackOff". Below listed errors are what I see in the kubernetes pod logs

$ docker logs aa94173b74e7
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4e8e46]

goroutine 1 [running]:
github.com/ibm-messaging/mq-container/internal/logger.(*Logger).Debugf(0x0, 0x58be27, 0x1f, 0xc420055e30, 0x1, 0x1)
/go/src/github.com/ibm-messaging/mq-container/internal/logger/logger.go:127 +0x26
main.logTermination(0xc4200103a0, 0x1, 0x1)
/go/src/github.com/ibm-messaging/mq-container/cmd/runmqdevserver/main.go:100 +0x16e
main.doMain(0xc4200781e0, 0xc420055f78)
/go/src/github.com/ibm-messaging/mq-container/cmd/runmqdevserver/main.go:111 +0xae
main.main()
/go/src/github.com/ibm-messaging/mq-container/cmd/runmqdevserver/main.go:163 +0x26

[arthurbarr]

That's a fairly catastrophic failure, and there's not much info to go on in that log (as it's failing to even write a debug log entry). Can you please provide some more details about the Kubernetes environment you're installing into (including things like which PodSecurityPolicy is the default). Also the results of kubectl describe pod foo-ibm-mq-0 and kubectl get -o yaml statefulset foo-ibm-mq would be useful.

[rakesh-4321]

Thanks for looking into this @arthurbarr
I'm installing this to the openshift container platform. I'm installing the IBM chart unaltered.
Below is the pod and sfs description. Let me know if you need anything else.

kubectl describe pod foo-ibm-mq-0
Name: foo-ibm-mq-0
Namespace: default
Node: m1.oc.local/172.17.4.3
Start Time: Wed, 22 Aug 2018 11:25:40 -0400
Labels: app=ibm-mq
chart=ibm-mqadvanced-server-dev
controller-revision-hash=foo-ibm-mq-569f8cf899
heritage=Tiller
release=foo
statefulset.kubernetes.io/pod-name=foo-ibm-mq-0
Annotations: openshift.io/scc=restricted
productID=98102d16795c4263ad9ca075190a2d4d
productName=IBM MQ Advanced for Developers
productVersion=9.1.0.0
Status: Running
Controlled By: StatefulSet/foo-ibm-mq
Containers:
qmgr:
Container ID: docker://d417f10989823d13a62765c03e4c147bf39b8c375b15f25389626a1ac3ed1e35
Image: ibmcom/mq:9.1.0.0
Image ID: docker-pullable://docker.io/ibmcom/mq@sha256:144f087d9b6162037d7eae69fb0ecc38598dfa865673a1927ac39c05b7465bfd
Ports: 1414/TCP, 9443/TCP, 9157/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Wed, 22 Aug 2018 11:26:29 -0400
Finished: Wed, 22 Aug 2018 11:26:29 -0400
Ready: False
Restart Count: 3
Limits:
cpu: 500m
memory: 512Mi
Requests:
cpu: 500m
memory: 512Mi
Liveness: exec [chkmqhealthy] delay=60s timeout=5s period=10s #success=1 #failure=1
Readiness: exec [chkmqready] delay=10s timeout=3s period=5s #success=1 #failure=1
Environment:
LICENSE: accept
MQ_QMGR_NAME: foo
LOG_FORMAT: basic
MQ_ENABLE_METRICS: true
MQ_ADMIN_PASSWORD: <set to the key 'adminPassword' in secret 'foo-ibm-mq'> Optional: false
Mounts:
/mnt/mqm from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-dcws5 (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-foo-ibm-mq-0
ReadOnly: false
default-token-dcws5:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-dcws5
Optional: false
QoS Class: Guaranteed
Node-Selectors:
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule
Events:
Type Reason Age From Message

---

Normal Scheduled 1m default-scheduler Successfully assigned foo-ibm-mq-0 to m1.oc.local
Normal SuccessfulMountVolume 1m kubelet, m1.oc.local MountVolume.SetUp succeeded for volume "default-token-dcws5"
Normal SuccessfulMountVolume 1m kubelet, m1.oc.local MountVolume.SetUp succeeded for volume "pvc-381bf927-a57f-11e8-b65f-080027ce0b4f"
Normal Pulled 17s (x4 over 1m) kubelet, m1.oc.local Container image "ibmcom/mq:9.1.0.0" already present on machine
Normal Created 17s (x4 over 1m) kubelet, m1.oc.local Created container
Normal Started 17s (x4 over 1m) kubelet, m1.oc.local Started container
Warning BackOff 9s (x8 over 1m) kubelet, m1.oc.local Back-off restarting failed container

kubectl get -o yaml statefulset foo-ibm-mq
apiVersion: apps/v1
kind: StatefulSet
metadata:
creationTimestamp: 2018-08-22T15:25:39Z
generation: 1
labels:
app: ibm-mq
chart: ibm-mqadvanced-server-dev
heritage: Tiller
release: foo
name: foo-ibm-mq
namespace: default
resourceVersion: "40913"
selfLink: /apis/apps/v1/namespaces/default/statefulsets/foo-ibm-mq
uid: a05cdf29-a61f-11e8-b65f-080027ce0b4f
spec:
podManagementPolicy: OrderedReady
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: ibm-mq
chart: ibm-mqadvanced-server-dev
heritage: Tiller
release: foo
serviceName: qm
template:
metadata:
annotations:
productID: 98102d16795c4263ad9ca075190a2d4d
productName: IBM MQ Advanced for Developers
productVersion: 9.1.0.0
creationTimestamp: null
labels:
app: ibm-mq
chart: ibm-mqadvanced-server-dev
heritage: Tiller
release: foo
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
weight: 2
- preference:
matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- ppc64le
weight: 2
- preference:
matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- s390x
weight: 2
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
- ppc64le
- s390x
containers:
- env:
- name: LICENSE
value: accept
- name: MQ_QMGR_NAME
value: foo
- name: LOG_FORMAT
value: basic
- name: MQ_ENABLE_METRICS
value: "true"
- name: MQ_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
key: adminPassword
name: foo-ibm-mq
image: ibmcom/mq:9.1.0.0
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- chkmqhealthy
failureThreshold: 1
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: qmgr
ports:
- containerPort: 1414
protocol: TCP
- containerPort: 9443
protocol: TCP
- containerPort: 9157
protocol: TCP
readinessProbe:
exec:
command:
- chkmqready
failureThreshold: 1
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 3
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 500m
memory: 512Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /mnt/mqm
name: data
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
volumeClaimTemplates:

• metadata:
  creationTimestamp: null
  labels:
  app: ibm-mq
  chart: ibm-mqadvanced-server-dev
  heritage: Tiller
  release: foo
  name: data
  spec:
  accessModes:
  • ReadWriteOnce
    resources:
    requests:
    storage: 2Gi
    status:
    phase: Pending
    status:
    collisionCount: 0
    currentReplicas: 1
    currentRevision: foo-ibm-mq-569f8cf899
    observedGeneration: 1
    replicas: 1
    updateRevision: foo-ibm-mq-569f8cf899

[arthurbarr]

Thanks for the info. I can see you're using the default OpenShift restricted SCC (openshift.io/scc=restricted). I'm afraid that the MQ image doesn't currently work with that SCC. We are planning some changes which should improve the situation, but for now, the easiest thing to do is to use the privileged SCC.

[mariusreusch]

Same issue here, we are really looking forward to the improvements!

[cra1gl1]

we are also looking for this feature. @arthurbarr when can we expect this?

[andrejpetras]

Hi,

I tried the Non-Root Pull-Request #276 on the Openshift and I get this errors:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x50a546]

goroutine 1 [running]:
github.com/ibm-messaging/mq-container/internal/logger.(*Logger).Debugf(0x0, 0x87a609, 0x1f, 0xc420085df8, 0x1, 0x1)
	/go/src/github.com/ibm-messaging/mq-container/internal/logger/logger.go:127 +0x26
main.logTermination(0xc4201ce070, 0x1, 0x1)
	/go/src/github.com/ibm-messaging/mq-container/cmd/runmqserver/logging.go:45 +0x16e
main.doMain(0x0, 0x0)
	/go/src/github.com/ibm-messaging/mq-container/cmd/runmqserver/main.go:40 +0x178
main.main()
	/go/src/github.com/ibm-messaging/mq-container/cmd/runmqserver/main.go:190 +0x22

  | panic: runtime error: invalid memory address or nil pointer dereference
-- | --
  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4e8e46]
  |  
  | goroutine 1 [running]:
  | github.com/ibm-messaging/mq-container/internal/logger.(*Logger).Debugf(0x0, 0x58be53, 0x1f, 0xc42006fe20, 0x1, 0x1)
  | /go/src/github.com/ibm-messaging/mq-container/internal/logger/logger.go:127 +0x26
  | main.logTermination(0xc4200103b0, 0x1, 0x1)
  | /go/src/github.com/ibm-messaging/mq-container/cmd/runmqdevserver/main.go:103 +0x16e
  | main.doMain(0x5241ea, 0xc42009e210)
  | /go/src/github.com/ibm-messaging/mq-container/cmd/runmqdevserver/main.go:114 +0xae
  | main.main()
  | /go/src/github.com/ibm-messaging/mq-container/cmd/runmqdevserver/main.go:166 +0x26

[alephnot0]

Is there a timeframe that can be communicated for a fix? I am also running into this issue.

[arthurbarr]

This is not a bug, but a current limitation. We are working on improvements. The master branch now runs as the "mqm" user. The restricted SCC requires the image to run as a random UID. Also, the image doesn't work under the default Red Hat SELinux container policy. You can change the SELinux type to spc_t, but that is not allowed under the restricted SCC. MQ's current support statement for SELinux is here