Failing to create test-master-0-commoninit.iso

[3rd-st-ninja]

When trying to provision a small cluster on a remote server, it fails when trying to create test-master-0-commoninit.iso. It successfully creates the first two image files in the location. I've confirmed multiple times that there are no permissions issues on that path (it's an nfs path). I'm not an ansible or terraform ninja and haven't had luck googling for information. There's a bunch of info below but I'm happy to provide any additional info anyone may need to help me solve this.

/data/admin/vm/vm_storage/test$ ls
test-master-os_image-0  test-master-os_image_resized-0

TASK [community.general.terraform] ********************************************************************************************************************************************************************************************************************************************
    "msg": "\nError: Error creating libvirt volume for cloudinit device test-master-0-commoninit.iso: Failed to create file '/data/admin/vm/vm_storage/test/test-master-0-commoninit.iso': Operation not permitted\n\n  with libvirt_cloudinit_disk.commoninit[0],\n  on k8s-master.tf line 33, in resource \"libvirt_cloudinit_disk\" \"commoninit\":\n  33: resource \"libvirt_cloudinit_disk\" \"commoninit\" {",
    "rc": 1,
    "stderr": "\nError: Error creating libvirt volume for cloudinit device test-master-0-commoninit.iso: Failed to create file '/data/admin/vm/vm_storage/test/test-master-0-commoninit.iso': Operation not permitted\n\n  with libvirt_cloudinit_disk.commoninit[0],\n  on k8s-master.tf line 33, in resource \"libvirt_cloudinit_disk\" \"commoninit\":\n  33: resource \"libvirt_cloudinit_disk\" \"commoninit\" {\n\n",
    "stderr_lines": [
        "",
        "Error: Error creating libvirt volume for cloudinit device test-master-0-commoninit.iso: Failed to create file '/data/admin/vm/vm_storage/test/test-master-0-commoninit.iso': Operation not permitted",
        "",
        "  with libvirt_cloudinit_disk.commoninit[0],",
        "  on k8s-master.tf line 33, in resource \"libvirt_cloudinit_disk\" \"commoninit\":",
        "  33: resource \"libvirt_cloudinit_disk\" \"commoninit\" {",
        ""
    ],
    "stdout": "libvirt_cloudinit_disk.commoninit[0]: Creating...\n",
    "stdout_lines": [
        "libvirt_cloudinit_disk.commoninit[0]: Creating..."
    ]

Ansible version

$ ansible --version
[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting with Ansible 2.12. Current version: 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0]. This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be
 disabled by setting deprecation_warnings=False in ansible.cfg.
ansible [core 2.11.3] 

Terraform version

$ terraform --version
Terraform v1.0.5

[kubealex]

Thank you @3rd-st-ninja for reporting this.
Could I ask you to please fill in the result for:
cat /etc/os-release

It would be useful to understand also the level of permissions in the folder, I'd ask you to also check that with an:

ls -latrZ /data/admin/vm/vm_storage/test
and
ls -latrdZ /data/admin/vm/vm_storage/test

It really seems like a possible issue with permissions: https://listman.redhat.com/archives/libvirt-users/2016-February/msg00032.html
Thank you!

[3rd-st-ninja]

Ah, it was kind of late when I was messing with this and it didn't occur to me at the time that it could be a libvirt operation causing the problem. Thanks for the link to that issue. Requested info is below as well as the libvirt log lines from syslog. If you know off the top of your head where to fix/work around this, that'd be great. Otherwise, I'll dig into it more and reply back here with a work around for posterity.

Aug 31 01:59:17 r610a libvirtd[102849]: cannot chown '/data/admin/vm/vm_storage/test/test-master-0-commoninit.iso' to (0, 0): Operation not permitted
Aug 31 01:59:17 r610a libvirtd[102849]: Failed to create file '/data/admin/vm/vm_storage/test/test-master-0-commoninit.iso': Operation not permitted

Remote host:

$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

$ ls -latrZ /data/admin/vm/vm_storage/test/
total 550148
drwxrwxr-x 3 nobody nogroup ?        26 Aug 31 01:58 ..
-rw-r--r-- 1 nobody nogroup ? 563150848 Aug 31 01:59 test-master-os_image-0
-rw-r--r-- 1 nobody nogroup ?    197088 Aug 31 01:59 test-master-os_image_resized-0
drwx--x--x 2 nobody nogroup ?        86 Aug 31 02:17 .

$  ls -latrdZ /data/admin/vm/vm_storage/test
drwx--x--x 2 nobody nogroup ? 86 Aug 31 02:17 /data/admin/vm/vm_storage/test

[3rd-st-ninja]

So, ultimately, it came down to the way I'd configured that nfs share it was writing to. I removed the options all_squash and all_root_squash and it seems to work. Have another issue now but it's entirely a libvirt thing, so I'll close this out.

Thanks for your help!!