bug: support jail sub-mounting single file

[CrackedPoly]

Bug

Support mounting a single file to jail. For example, if we define volumes and mounts as follows, codes won't work before this PR.

...
volumeSources:
  - name: controller-spool
    persistentVolumeClaim:
      claimName: "controller-spool-pvc"
      readOnly: false
  - name: jail
    persistentVolumeClaim:
      claimName: "jail-pvc"
      readOnly: false
  - name: gpfs-dataset
    hostPath:
      path: /gpfs/benchmark
      type: Directory
  - name: gpfs-image
    hostPath:
      path: /gpfs/nemofw-training-2403.tar
      type: File
...
...
    volumes:
      jail:
        volumeSourceName: "jail"
      jailSubMounts:
        - name: "gpfs-bench-dataset"
          mountPath: "/benchmark"
          volumeSourceName: "gpfs-dataset"
        - name: "gpfs-bench-image"
          mountPath: "/images/nemofw-training-2403.tar"
          volumeSourceName: "gpfs-image"
...

Starting slurmd entrypoint script
Link users from jail
Link home from jail because slurmd uses it
Bind-mount slurm configs from K8S config map
Make ulimits as big as possible
Apply sysctl limits from /etc/sysctl.conf
vm.max_map_count = 655300
Update linker cache
Complement jail rootfs
/mnt/jail /
Bind-mount virtual filesystems
Bind-mount cgroup filesystem
Remount /tmp
Bind-mount /var/log because it should be node-local
Bind-mount DNS configuration
Bind-mount /etc/hosts
Bind-mount jail submounts from upper /mnt/jail.upper into the actual /mnt/jail
Bind-mount jail submount benchmark
Bind-mount jail submount images/nemofw-training-2403.tar
mount: /mnt/jail/images/nemofw-training-2403.tar: mount(2) system call failed: Not a directory.

Another thing

I think the feature of configurable permission should be set at K8S VolumeMount (field readOnly) and I can implement this feature. Any suggestions of opinions?

[Uburro]

LGTM. @CrackedPoly thanks. I need to make some test before merging it

[Uburro]

it's ok

[rdjjke]

By "configurable permissions" I meant user/group ownerships and permission bits, which can't be done via K8s spec.

But supporting K8s' volumeMount readOnly option will also be useful!

Thanks for the PR @CrackedPoly

[rdjjke]

We actually have an idea to allow the execution of custom scripts in container entrypoints (by replacing the currently unused security limits ConfigMap). Plus, one more script that is executed once after the cluster is created.

So such things can be done there, though not in a very user-friendly manner.

[CrackedPoly]

We actually have an idea to allow the execution of custom scripts in container entrypoints (by replacing the currently unused security limits ConfigMap). Plus, one more script that is executed once after the cluster is created.

So such things can be done there, though not in a very user-friendly manner.

I like this "hook" idea, and I vote for it.