Cannot create directory

[srimukh]

Image not starting properly in kubernetes after I bind the image to a volume. Following are the exception logs

2019-05-06T21:46:50.682Z Image tag: ibm-mqadvanced-server-dev:9.1.2.0-x86_64-ubuntu-16.04
2019-05-06T21:46:50.996Z MQ version: 9.1.2.0
2019-05-06T21:46:50.996Z MQ level: p912-L190313.DE
2019-05-06T21:46:50.996Z MQ license: Developer
2019-05-06T21:46:51.665Z Starting web server
2019-05-06T21:46:51.665Z Creating queue manager ibmmqdeployment177776554449gkbq
2019-05-06T21:46:52.116Z crtmqm returned 71
2019-05-06T21:46:52.116Z AMQ6109S: An internal IBM MQ error has occurred.

2019-05-06T21:46:52.116Z /usr/bin/crtmqm: exit status 71
2019-05-06T21:46:52.775Z Error 22 starting web server: 
mkdir: cannot create directory '/var/mqm/web/installations/Installation1/servers/.pid': Permission denied
mkdir: cannot create directory '/var/mqm/web/installations/Installation1/servers/mqweb/workarea': Permission denied
/opt/mqm/web/bin/server: 353: /opt/mqm/web/bin/server: cannot create /var/mqm/web/installations/Installation1/servers/mqweb/workarea/.sLock: Directory nonexistent
/opt/mqm/web/bin/server: 800: /opt/mqm/web/bin/server: cannot create /var/mqm/web/installations/Installation1/servers/mqweb/logs/console.log: Permission denied
JVMSHRC559E Failed to create a directory "/var/mqm/web/installations/Installation1/servers/.classCache" for the shared class cache
JVMSHRC686I Failed to startup shared class cache. Continue without using it as -Xshareclasses:nonfatal is specified
java.io.FileNotFoundException: /var/mqm/web/installations/Installation1/servers/mqweb/logs/start.log (Permission denied)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:286)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:226)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:173)
	at com.ibm.ws.kernel.boot.logging.DefaultFileStreamFactory.createOutputStream(DefaultFileStreamFactory.java:27)
	at com.ibm.ws.kernel.boot.logging.TextFileOutputStreamFactory$1.run(TextFileOutputStreamFactory.java:72)
	at com.ibm.ws.kernel.boot.logging.TextFileOutputStreamFactory$1.run(TextFileOutputStreamFactory.java:69)
	at java.security.AccessController.doPrivileged(AccessController.java:703)
	at com.ibm.ws.kernel.boot.logging.TextFileOutputStreamFactory.createOutputStream(TextFileOutputStreamFactory.java:69)
	at com.ibm.ws.kernel.boot.Debug.open(Debug.java:35)
	at com.ibm.ws.kernel.boot.Launcher.createPlatform(Launcher.java:93)
	at com.ibm.ws.kernel.boot.cmdline.EnvCheck.main(EnvCheck.java:59)
	at com.ibm.ws.kernel.boot.cmdline.EnvCheck.main(EnvCheck.java:35)
Starting server mqweb.
Server mqweb start failed. Check server logs for details.

[arthurbarr]

You have to be careful about volume permissions. Different storage providers handle this differently. Can you provide some more details about how you're running this (e.g. Kubernetes YAML, or through the IBM MQ Cloud Pak)?

For example, some storage classes will dynamically add a GID which is authorised to write files to the volume, and add that GID to the container. In other storage classes, you might have to add a different GID manually.

[srimukh]

Hi thanks for the reply. I am running it through kubernetes yaml only. I am in fact using an Azure file share as my volume mount.

i am creating a persistent volume and a persistent volume claim using the following files.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mqfilesharepv
  labels:
    usage: mqfilesharepv
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  azureFile:
    # Replace with your secret name
    secretName: mq-secret
    # Replace with correct storage share name
    shareName: mqfileshare
    # In case the secret is stored in a different namespace
    #shareNamespace: default
    readOnly: false
  mountOptions:
  - dir_mode=0755
  - file_mode=0755
  #- uid=1000
  #- gid=1000

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mqfilesharepvc
  annotations:
    volume.beta.kubernetes.io/storage-class: ""
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi
  selector:
    matchLabels:
      usage: mqfilesharepv

and following file is my deployment yaml which i am using to deploy

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ibmmq-deployment1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: ibmmq1
        version: v4
    spec:
      securityContext:
        fsGroup: 999
        #runAsGroup: 999
        runAsUser: 0
      containers:
      - name: ibmmq1
        #image: miraclelabsregistry008.azurecr.io/ibmmq:9.1.1.0
        image: ibmcom/mq:9.1.2.0
        env:
         - name: LICENSE
           value: accept
        imagePullPolicy: IfNotPresent
        ports:
        - name: port1414
          containerPort: 1414
        - name: port9443
          containerPort: 9443
        volumeMounts:
        - mountPath: "/mnt/mqm"
          name: test-storage
      volumes:
      - name: test-storage
        persistentVolumeClaim:
         claimName: mqfilesharepvc

[parrobe]

Looking at the above you're giving your PV files/dirs the permissions 755 which means only the owner uid will be able to write. What uid/gid ownership will the directories be given by your PV supplier? If it isn't 0 then your container (Which is running as uid 0) will not have permission to write to the volume.

[srimukh]

I removed the folder permission while creating pv and as per mey setting in my yml you can see it is starting as root. Only diffrence i have noticed with giving 755 is in both cases issue is permission only but with 755 it is creating the basic folders like queue manager and failing after that but from this logs you can see it is failing at the start itself.

2019-05-20T19:07:54.689Z CPU architecture: amd64
2019-05-20T19:07:54.689Z Linux kernel version: 4.15.0-1040-azure
2019-05-20T19:07:54.689Z Container runtime: kube
2019-05-20T19:07:54.689Z Base image: Ubuntu 16.04.6 LTS
2019-05-20T19:07:54.691Z Running as user ID 0 (root) with primary group 0, and supplementary groups 999
2019-05-20T19:07:54.691Z Capabilities (bounding set): chown,dac_override,fowner,fsetid,kill,setgid,setuid,setpcap,net_bind_service,net_raw,sys_chroot,mknod,audit_write,setfcap
2019-05-20T19:07:54.691Z seccomp enforcing mode: disabled
2019-05-20T19:07:54.691Z Process security attributes: docker-default (enforce)
2019-05-20T19:07:54.741Z Set password for "admin" user
2019-05-20T19:07:54.749Z Using queue manager name: ibmmqdeployment1ccc6c99865cvhg
2019-05-20T19:11:01.543Z Created directory structure under /var/mqm
2019-05-20T19:11:01.543Z Image created: 2019-04-24T09:10:17+00:00
2019-05-20T19:11:01.543Z Image revision: a9718af
2019-05-20T19:11:01.543Z Image source: https://github.com/ibm-messaging/mq-container.git
2019-05-20T19:11:01.543Z Image tag: ibm-mqadvanced-server-dev:9.1.2.0-x86_64-ubuntu-16.04
2019-05-20T19:11:06.808Z MQ version: 9.1.2.0
2019-05-20T19:11:06.808Z MQ level: p912-L190313.DE
2019-05-20T19:11:06.808Z MQ license: Developer
2019-05-20T19:11:13.035Z Creating queue manager ibmmqdeployment1ccc6c99865cvhg
2019-05-20T19:11:13.035Z Starting web server
2019-05-20T19:11:37.479Z crtmqm returned 25
2019-05-20T19:11:37.479Z AMQ6245E: Error executing system call 'mkdir' on file
'/var/mqm/qmgrs/ibmmqdeployment1ccc6c99865cvhg/ssl' error '1'.
AMQ7019E: An error occurred while creating or checking the directory structure
for the queue manager.

2019-05-20T19:11:37.479Z /usr/bin/crtmqm: exit status 25

[punitporwal07]

@srimukh were you able to fix this issue ?

I am getting ditto error when using ibm-mq-9.1.5 image on my generic k8s-cluster, after attaching pv & pvc

[k8s@002 k8s]$ kl po/mqlv-deployment-6fffc84f5c-l5hnb
2020-10-29T08:50:19.355Z CPU architecture: amd64
2020-10-29T08:50:19.356Z Linux kernel version: 3.10.0-1062.9.1.el7.x86_64
2020-10-29T08:50:19.356Z Container runtime: kube
2020-10-29T08:50:19.356Z Base image: Red Hat Enterprise Linux 8.1 (Ootpa)
2020-10-29T08:50:19.356Z Running as user ID 1001 with primary group 0, and supplementary groups 20000
2020-10-29T08:50:19.356Z Capabilities (bounding set): chown,dac_override,fowner,fsetid,kill,setgid,setuid,setpcap,net_bind_service,net_raw,sys_chroot,mknod,audit_write,setfcap
2020-10-29T08:50:19.356Z seccomp enforcing mode: disabled
2020-10-29T08:50:19.356Z Process security attributes: system_u:system_r:spc_t:s0
2020-10-29T08:50:19.356Z No volume detected. Persistent messages may be lost
2020-10-29T08:50:19.475Z Using queue manager name: QM1
2020-10-29T08:50:19.475Z mkdir /mnt/mqm/data: permission denied

plz suggest!

[preethikarthi]

I think there is no way to run this image in kubernetes.. PV and PVC needs root permission to modify the folder structure. But IBM MQ requires non root user (mqm or uid: 999)to start the web app. This leads to non functional in K8s.!!! May be try add the static disk to deployment.
I din't try this. But will post the output sooner...

Example:

 volumeMounts:
  - name: azure
    mountPath: /mnt/mqm

volumes:
- name: azure
azureDisk:
kind: Managed
diskName: myAKSDisk
diskURI: /subscriptions//resourceGroups/MC_myAKSCluster_myAKSCluster_eastus/providers/Microsoft.Compute/disks/myAKSDisk

[preethikarthi]

Yes, i manage to run the "mq" in AKS. Here is the snippet of deployment which i am using.

apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Values.app.name }}
labels:
app: {{ .Values.app.name }}
group: {{ .Values.app.name }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: {{ .Values.app.name }}
template:
metadata:
labels:
app: {{ .Values.app.name }}
group: {{ .Values.app.name }}
spec:
securityContext:
fsGroup: 999
runAsNonRoot: false
containers:
- name: {{ .Values.app.name }}
image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
resources:
limits:
cpu: {{ .Values.limits.cpu }}
memory: {{ .Values.limits.memory }}
requests:
cpu: {{ .Values.requests.cpu }}
memory: {{ .Values.requests.memory }}
env:
- name: LICENSE
value: "accept"
volumeMounts:
- name: wmq
mountPath: /mnt/mqm

        - name: config-volume
          mountPath: /etc/mqm/config.mqsc
          subPath: config.mqsc

  volumes:
    - name: wmq
      azureDisk:
        kind: Managed
        diskName: myDisk
        diskURI: /subscriptions/<subscription>/resourcegroups/<node_resource_group>/myDisk

You may need to create static Azure managed disk with desired size. And need to attach with your POD with the above security context.

NOTE: This wont work, if you are creating storageClass with Azure-file. Since it requires root permission to modify the directories. So its the best approach to create managed disk and attach them.

Thanks

[arthurbarr]

This image can definitely be run in Kubernetes, as many people have done so. There are a few things to consider:

1. Different versions of MQ work slightly differently, and the image code has changed over time to match, so you need to make sure you're using the right combination. Prior to MQ 9.2.0, MQ needed to run as the "mqm" user. This was changed to be more secure, and MQ 9.2.0 shipped a new "no-install" package which doesn't use setuid bits at all, and can be run as any user. The current code assumes you are using this package. When running in this way, we test using the "root" group (GID 0) — note that the root group has no special permissions, but it gives a consistent group ID to use in OpenShift, which runs containers with random UIDs.
2. You have to give MQ permissions to write to the files. This is typically done using a supplementalGroup or an fsGroup setting in the Kubernetes securityContext. You need to find out which permissions your filesystem has, and add the appropriate group.
   • Rarely, a poorly secured file server will only give access to the root user, in which case you use an initContainer as root, which runs runmqserver -i to just change the permissions on the volume then exit. Hopefully this shouldn't be needed though
3. Make sure you're not using Azure Files for storage, as it doesn't support file locks well, and MQ needs robust file locks to operate. Azure block storage is fine though.

This is an old issue, and things have changed over time, and it can be a bit confusing. I'd like to close this issue, but please feel free to raise a new one if you're still having problems.