Cannot modify host files via bootstrap-container (Permission denied)

[effy-coding]

I am running Bottlerocket OS on EC2 instance and i'm trying to configure containerd settings.

My user-data here

cat<<EOF >> /.bottlerocket/rootfs/etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".registry]
   config_path = "/etc/containerd/certs.d"
EOF

mkdir /.bottlerocket/rootfs/etc/containerd/certs.d
mkdir /.bottlerocket/rootfs/etc/containerd/certs.d/ghcr.io
touch /.bottlerocket/rootfs/etc/containerd/certs.d/ghcr.io/hosts.toml

cat<<EOF > /.bottlerocket/rootfs/etc/containerd/certs.d/ghcr.io/hosts.toml
server = "https://ghcr.io"

[host."https://ghcr.io"]
  capabilities = ["pull", "resolve"]
  [host."https://ghcr.io".header]
    authorization = "Bearer TOKEN"
EOF

My bootstrap-container Dockerfile here

#! /usr/bin/env sh
USER_DATA=/.bottlerocket/bootstrap-containers/current/user-data
chmod +x $USER_DATA

sh $USER_DATA

Problem is here

When ec2 execute user-data, permission denied error occurs creating folders and files.

[  OK  ] Started containerd runtime for host containers.
         Starting bootstrap container ctr-registry-configuration...
[   31.539807] cgroup: cgroup: disabling cgroup2 socket matching due to net_prio or net_cls activation
[   31.594645] audit: type=1400 audit(1644911043.667:3): avc:  denied  { append } for  pid=1622 comm="sh" name="config.toml" dev="tmpfs" ino=138 scontext=system_u:system_r:control_t:s0-s0:c0.c1023 tcontext=system_u:object_r:etc_t:s0 tclass=file permissive=0
[   31.610754] audit: type=1400 audit(1644911043.687:4): avc:  denied  { write } for  pid=1623 comm="mkdir" name="containerd" dev="tmpfs" ino=137 scontext=system_u:system_r:control_t:s0-s0:c0.c1023 tcontext=system_u:object_r:etc_t:s0 tclass=dir permissive=0
[   31.241613] host-ctr[1571]: /.bottlerocket/bootstrap-containers/current/user-data: line 1: can't create /.bottlerocket/rootfs/etc/containerd/config.toml: Permission denied
[   31.262361] host-ctr[1571]: mkdir: can't create directory '/.bottlerocket/rootfs/etc/containerd/certs.d': Permission denied
[   31.267299] host-ctr[1571]: mkdir: can't create directory '/.bottlerocket/rootfs/etc/containerd/certs.d/ghcr.io': No such file or directory
[   31.272294] host-ctr[1571]: touch: /.bottlerocket/rootfs/etc/containerd/certs.d/ghcr.io/hosts.toml: No such file or directory
[   31.276691] host-ctr[1571]: /.bottlerocket/bootstrap-containers/current/user-data: line 10: can't create /.bottlerocket/rootfs/etc/containerd/certs.d/ghcr.io/hosts.toml: nonexistent directory
[   31.312785] host-ctr[1571]: time="2022-02-15T07:44:03Z" level=fatal msg="Container boot.ctr-registry-configuration exited with non-zero status"

👉 Question: How can i create files under /.bottlerocket/rootfs/etc via bootstrap-container? How can i grant it?

[etungsten]

Hi @dangen-effy, those permission errors are due to SElinux. Bootstrap container processes do not have the necessary SElinux label to make changes to configuration files in /etc. /etc is a tmpfs mount intended for certain host processes to write generated configuration files and nothing else. Currently bootstrap container processes are labelled with control_t. You can read more about the labels given to containers here.

Since the end goal is to configure containerd, this should be done through Bottlerocket’s API system. /etc/containerd/config.toml is rendered by Bottlerocket every time there’s a change to any containerd-related Bottlerocket setting. Bottlerocket would overwrite anything the bootstrap container would have written out to that file.

To best support this use-case, we would need to create new settings for these containerd configuration items. What would you like to see exposed via Bottlerocket settings?

[chiragjn]

I too have a requirement to edit config for nvidia-container-toolkit and nvidia-device-plugin systemd service unit.
I am curious can a host container with superpower flag edit these files?

To add more context, I have created this PR: #3718
But maybe these can be exposed as settings?

[arnaldo2792]

Hey @effy-coding , sorry for the rather late response. There are a few Bottlerocket APIs that allow to configure containerd. If you have the need to set a specific configuration, please let us know by cutting an issue so that we can add it to the API. We could also guide you on how to add these configurations if you are interested in contributing 🎉 !

[arnaldo2792]

@chiragjn, I have thought about providing an API to configure the k8s device plugin (see #2347). But that's still on the backlog. For the change you proposed (thanks!), we don't want to make you implement the API for it since there is a learning curve to understanding how the API system works. Thus, we are still evaluating how to take in your change, since as you mentioned in your PR (again, thanks for the great catch!), Bottlerocket isn't just for k8s and the change as it is could break the ECS variant.

[chiragjn]

We'll wait for API settings, but in the meantime, is there any possible way to modify these config files and have them persist
(except building custom AMIs) ?

[arnaldo2792]

@chiragjn , there is not yet. We have think about how to solve this problem but it isn't trivial. In Bottlerocket, the configuration files are /etc are ephemeral and regenerated when the values in an API that affects them changes. Thus, any change on those files has to occur after the API server is done rendering the files. Only after this can custom modifications be applied to the files. In other words:

• API server renders the initial file with values from the API
• Custom configs are applied
• Services that depend on the configuration files are restarted

[mtprovenir]

For anyone finding this while searching, it appears this is now possible via the API.