mkdir: cannot create directory '/mnt/app-root.setup-volume': Permission denied

[jichenjc]

/kind bug

What versions of software are you using?

Operating System:

Output of odo version:
latest (build from master from last week)

How did you run odo exactly?

this is similar to following
#1623

but I used hostPath and NFS both got same error and I think it's not related to anyuid...
I added logs and showed the pv is mount as root:root and as normal user (1000580000), I can't create things there,

from https://kubernetes.io/docs/concepts/storage/volumes/#hostpath

the files or directories created on the underlying hosts are only writable by root. You either need to run your process as root in a privileged Container or modify the file permissions on the host to be able to write to a hostPath volume

so at least hostPath doesn't work, NFS from my test also doesn't work
so the question is what we did on odo CI , which things are changed or PV used?

[root@infnod-1 ~]# crictl logs 15d0e7b81001e
2020/02/27 06:47:57 [go-init] Pre-start command launched : /opt/odo/bin/s2i-setup
1000580000

• set -eo pipefail
• whoami
• id
  uid=1000580000(1000580000) gid=0(root) groups=0(root),1000580000
• ls -l /
  total 8
  lrwxrwxrwx. 1 root root 7 Jan 28 10:42 bin -> usr/bin
  dr-xr-xr-x. 2 root root 6 Jan 28 10:42 boot
  drwxr-xr-x. 5 root root 360 Feb 27 06:47 dev
  drwxr-xr-x. 1 root root 44 Jan 29 14:29 etc
  -rw-r--r--. 1 root root 7753 Jan 29 14:29 help.1
  drwxr-xr-x. 2 root root 6 Jan 28 10:42 home
  lrwxrwxrwx. 1 root root 7 Jan 28 10:42 lib -> usr/lib
  lrwxrwxrwx. 1 root root 9 Jan 28 10:42 lib64 -> usr/lib64
  drwxr-xr-x. 2 root root 6 Dec 14 2017 media
  drwxr-xr-x. 2 root root 6 Dec 14 2017 mnt
  drwxr-xr-x. 1 root root 17 Feb 27 06:47 opt
  dr-xr-xr-x. 190 root root 0 Feb 27 06:47 proc
  dr-xr-x---. 1 root root 23 Jan 28 10:54 root
  drwxr-xr-x. 1 root root 18 Jan 29 14:29 run
  lrwxrwxrwx. 1 root root 8 Jan 28 10:42 sbin -> usr/sbin
  drwxr-xr-x. 2 root root 6 Dec 14 2017 srv
  dr-xr-xr-x. 13 root root 0 Feb 26 06:52 sys
  drwxrwxrwt. 1 root root 6 Jan 29 14:29 tmp
  drwxr-xr-x. 1 root root 69 Jan 29 14:13 usr
  drwxr-xr-x. 1 root root 52 Jan 28 10:42 var
• ls -lh /tmp
  total 4.0K
  -rwx------. 1 root root 836 Jan 28 10:42 ks-script-bn8Lhj
  -rw-------. 1 root root 0 Jan 28 10:42 yum.log
• ls -lh /opt/app-root
  ls: cannot open directory /opt/app-root: Permission denied
  2020/02/27 06:47:57 [go-init] Pre-start command failed
  2020/02/27 06:47:57 [go-init] exit status 2

Actual behavior

Expected behavior

Any logs, error output, etc?

[zhengxiaomei123]

The PVC is mounted at the /mnt mount point inside the Init container and The PV is mounted at the /opt/app-root point inside the application container. Can I think the two "Permission denied" error all means there is no permission to write the PersistentVolume?

[zhengxiaomei123]

Have you check that the HostPath configured on your PV is a valid dictorary?

[jichenjc]

yes, it's a valid path and I even can see that folder
I believe it is the permission issue

[zhengxiaomei123]

yes, it's a valid path and I even can see that folder
I believe it is the permission issue

The folder means "app-root"?

[jichenjc]

no, the /mnt/xx on my host, the app-root as you can see my log
you can see it but you can't write to it, see the links from k8s community above

[zhengxiaomei123]

the files or directories created on the underlying hosts are only writable by root. You either need to run your process as root in a privileged Container or modify the file permissions on the host to be able to write to a hostPath volume
Could you confirm that the file permissions on the host is to be able to write to a hostPath volume?

[jichenjc]

yes, I confirm, I modify the permission to 777 :)

[girishramnani]

@jichenjc are you working this issue?

[jichenjc]

no, this is something I need some help or want to know whether someone has simliar output to me by using hostPath..., maybe it's invalid due to hostPath for PV/PVC，but need some guide here.. thanks

[adisky]

@jichenjc could you please mention exact odosteps how you are trying to run.

[adisky]

I tried to reproduce this, java component gets deployed successfully on my cluster.

[adisky@localhost java-test]$ oc version
Client Version: unknown
Server Version: 4.2.13
Kubernetes Version: v1.14.6+a8d983c
[adisky@localhost java-test]$ odo version
odo v1.1.1 (1eb878ea2)

Server: https://api.crc.testing:6443
Kubernetes: v1.14.6+a8d983c

[adisky@localhost java-test]$ odo create java
 ✓  Validating component [12ms]

Please use `odo push` command to create the component with source deployed
[adisky@localhost java-test]$ odo push
Validation
 ✓  Checking component [9ms]

Configuration changes
 ✓  Initializing component
 ✓  Creating component [89ms]

Pushing to component java-java-test-hpth of type local
 ✓  Checking files for pushing [379414ns]
 ✓  Waiting for component to start [21s]
 ✓  Syncing files to the component [1s]
 ✓  Building component [3s]
 ✓  Changes successfully pushed to component

[jichenjc]

As discussed with @adisky in slack additional info :

0. the odo is build from master around 1 months ago, but I think the problem should still exist
1. I only saw this error in hostPath PV/PVC, switch to NFS solved the problem
2. the test is related to nodejs
3. https://docs.openshift.com/container-platform/4.2/cli_reference/openshift_developer_cli/creating-a-single-component-application-with-odo.html is the doc I used I think it's pretty simple..
4. I can't reproduce this now due to switch to NFS, every record is recorded there and odo component fails when anyuid is set #1623 as well

[adisky]

@jichenjc as you mentioned this issue is same as #1623, Please try using the project other than default, I have also faced similar error recently, I changed the project it is solved.

As per the comments here #1623 (comment), Its mostly related to some security context related settings with the default project.

[jichenjc]

um... I think I switched to default project
anyway , maybe I remembered a wrong thing, thanks for the reminder