Allow creation of immutable directories below root folder

[HeikoOnnebrink]

When trying to create a folder /dockerdata below root dir I get

ignition[696]: }CRITICAL : Ignition failed: mkdir /sysroot/dockerdata: operation not permitted

We plan to switch over to Fedora Container Linux in the future next to existing CoreOS systems we have in place.
We run hundreds of CoreOS instances with a /dockerdata folder to which we mount local SSD or network disk to persist state of database container (that we mount inside docker container)
In order to stay backward compatible (and avoid hacky symlinking) we need to create directory /dockerdata also from Ignition 3 configs.

As we understand the OStree and its immutable policies do not allow folder creation in root and restrict it to /var and /etc.

Proposal would be to allow a syntax in Ignition 3 that allows folder creation below root as long as they are strictly immutable

directories:
  -- path: /dockerdata
      Immutable: true
      ..

These folder then can be used exclusively as a mount dir ..

[jlebon]

Cross-linking: coreos/rpm-ostree#337

[dustymabe]

hey @HeikoOnnebrink, thanks for bringing this up!

We discussed this in the Fedora CoreOS meeting today. Unfortunately in addition to OSTree having a root directory that is marked as immutable there is another technical reason why modifying the toplevel mount point will be problematic. The Live ISO and Live PXE support where we'll be mounting a squashfs read-only won't support mounting toplevel directories either. Rather than supporting it in one case and not in another we think it would be a more consistent user experince if we do the same in all cases. We will add this to the migration docs and track the documentation effort over in #159.

I'm going to close this out but we can continue the discussion here if there's anything else to discuss.

[cgwalters]

I don't think it'd be too hard to have / be a tmpfs mount in both cases that has basically /usr mounted in; that's what systemd.volatile wants to do. However, it's not trivial either.

[HeikoOnnebrink]

thanks for discussing the topic.. but for sure I am not glad with result to move it to the doc section. its a real pain point for us .. what about comment from @cgwalters .. sounds like there might be a way :-)

[cgwalters]

For mount points, using /var/mnt makes total sense, e.g. /var/mnt/mydata. It can also make sense (particularly for networked mounts) to use /run/mnt (which doesn't exist by default but we could reserve it) - you'd just need a process to create the directory when it's mounted.