Extending read-only state to /sysroot and /boot #1265

cgwalters · 2017-10-12T13:37:48Z

Today ostree has a ro bind mount over /usr, but one can just do e.g. rm -rf /sysroot/* and the whole system is destroyed.

Similarly, we tried to have support for /boot as a ro mount but it needs work.

What we should do is have these be ro by default, and then have apps using libostree create a new mount namespace, and make manipulations there. That way the system stays read-only to everything else. Alternatively, we could teach libostree how to do that internally, but it'd get ugly fast...we'd need to fork off a subprocess basically.

For example rpm-ostreed and eos-updater could change their systemd unit files to create a new mount namespace, and tell libostree that one is set up.

(Conceptually this overlaps a ton with systemd's ProtectSystem=strict...we're basically imposing that on the whole system by default)

The text was updated successfully, but these errors were encountered:

cgwalters · 2017-12-23T09:26:05Z

See also for /tmp:
https://pagure.io/fedora-atomic/pull-request/62
coreos/rpm-ostree#820

giuseppe · 2017-12-23T10:29:44Z

would this be an issue for system containers that use /ostree/deploy/$OS_NAME/$DEST_FILE for creating hard links?

cgwalters · 2017-12-27T14:04:14Z

@giuseppe an important bit from the above is:

...and then have apps using libostree create a new mount namespace, and make manipulations there.

The goal here isn't to make things truly immutable, it's to prevent things like restorecon -R / from destroying the system. And similar non-ostree-aware tools.

See also on-list discussion:

https://mail.gnome.org/archives/ostree-list/2017-December/msg00009.html

giuseppe · 2017-12-27T20:25:51Z

I see, thanks. Could the new API allow also the current mount namespace to be used? It will be the user responsibility to ensure a new mount namespace is created and /sysroot can be remounted as writeable (and back to ro if needed).

cgwalters · 2018-10-29T19:01:20Z

PR in #1767

We want to support extending the read-only state to cover `/sysroot` and `/boot`, since conceptually all of the data there should only be written via libostree. This change needs to be opt-in though to avoid breaking anyone. Add a `sysroot/readonly` key to the repository config which instructs `ostree-remount.service` to ensure `/sysroot` is read-only. This requires a bit of a dance because `/sysroot` is actually the same filesystem as `/`, which we can't mount readonly as we want `/etc` to be writable (and `/var` if it's there too). So we actually make it into a bind mount. Also add an API to instruct the libostree shared library that the caller has created a new mount namespace. This way we can freely remount read-write. This approach extends upon in a much better way previous work we did to support remounting `/boot` read-write. Closes: ostreedev#1265

Part of the implementation of ostreedev/ostree#1265 Convert to systemd mount units so we can easily specify the `ro` flag, and also to use the `LABEL={root,var}` rather than UUIDs. Enable the sysroot/readonly flag.

Prep for ostreedev/ostree#1265

cgwalters · 2018-12-26T18:47:04Z

Requirements to implement this for Fedora CoreOS:

coreos/fedora-coreos-config#39
coreos/ignition-dracut#36

We want to support extending the read-only state to cover `/sysroot` and `/boot`, since conceptually all of the data there should only be written via libostree. This change needs to be opt-in though to avoid breaking anyone. Add a `sysroot/readonly` key to the repository config which instructs `ostree-remount.service` to ensure `/sysroot` is read-only. This requires a bit of a dance because `/sysroot` is actually the same filesystem as `/`, which we can't mount readonly as we want `/etc` to be writable (and `/var` if it's there too). So we actually make it into a bind mount. Also add an API to instruct the libostree shared library that the caller has created a new mount namespace. This way we can freely remount read-write. This approach extends upon in a much better way previous work we did to support remounting `/boot` read-write. Closes: ostreedev#1265

Prep for ostreedev/ostree#1265

We want to support extending the read-only state to cover `/sysroot` and `/boot`, since conceptually all of the data there should only be written via libostree. This change needs to be opt-in though to avoid breaking anyone. Add a `sysroot/readonly` key to the repository config which instructs `ostree-remount.service` to ensure `/sysroot` is read-only. This requires a bit of a dance because `/sysroot` is actually the same filesystem as `/`, which we can't mount readonly as we want `/etc` to be writable (and `/var` if it's there too). So we actually make it into a bind mount. Also add an API to instruct the libostree shared library that the caller has created a new mount namespace. This way we can freely remount read-write. This approach extends upon in a much better way previous work we did to support remounting `/boot` read-write. Closes: ostreedev#1265

jlebon · 2019-04-18T15:17:19Z

Yeah, we really should fix this. Right now, restorecon -R / and autorelabel=1 are super dangerous (see e.g. user report at https://twitter.com/goozbach/status/1118697493657681920?s=19).

To anyone in that situation, you can recover from this by booting with enforcing=0, then:

# rpm-ostree status -v # copy "BaseCommit" or if missing, "Commit"
# ostree checkout -H $checksum /ostree/repo/tmp/selinux-fix
# ostree fsck --delete
# ostree commit --consume --link-checkout-speedup --orphan --selinux-policy=/ /ostree/repo/tmp/selinux-fix
# restorecon -Rv /var
# restorecon -Rv /etc
# ostree admin deploy fedora-atomic:fedora/29/x86_64/atomic-host # or whatever refspec you're on
# reboot

At this point, you should be able to boot in enforcing mode again. Final cleanup:

# rpm-ostree cleanup --rollback

Note this will blow away all your layered packages; you'll have to relayer those again.

goozbach · 2019-04-18T18:27:27Z

should that be BaseCommit or Commit instead of checksum?

jlebon · 2019-04-18T18:33:50Z

Thanks! I edited the original comment.

goozbach · 2019-04-19T04:31:22Z

For future reference I had to do an rpm-ostree upgrade prior to the above steps for them to work.

Other than that, this workaround is good!

We want to support extending the read-only state to cover `/sysroot` and `/boot`, since conceptually all of the data there should only be written via libostree. This change needs to be opt-in though to avoid breaking anyone. Add a `sysroot/readonly` key to the repository config which instructs `ostree-remount.service` to ensure `/sysroot` is read-only. This requires a bit of a dance because `/sysroot` is actually the same filesystem as `/`, which we can't mount readonly as we want `/etc` to be writable (and `/var` if it's there too). So we actually make it into a bind mount. Also add an API to instruct the libostree shared library that the caller has created a new mount namespace. This way we can freely remount read-write. This approach extends upon in a much better way previous work we did to support remounting `/boot` read-write. Closes: ostreedev#1265

We want to support extending the read-only state to cover `/sysroot` and `/boot`, since conceptually all of the data there should only be written via libostree. This change needs to be opt-in though to avoid breaking anyone. Add a `sysroot/readonly` key to the repository config which instructs `ostree-remount.service` to ensure `/sysroot` is read-only. This requires a bit of a dance because `/sysroot` is actually the same filesystem as `/`; so we make `/etc` a writable bind mount in this case. We also need to handle `/var` in the "OSTree default" case of a bind mount; the systemd generator now looks at the writability state of `/sysroot` and uses that to determine whether it should have the `var.mount` unit happen before or after `ostree-remount.service.` Also add an API to instruct the libostree shared library that the caller has created a new mount namespace. This way we can freely remount read-write. This approach extends upon in a much better way previous work we did to support remounting `/boot` read-write. Closes: ostreedev#1265

Prep for using the default mount namespace handling there that will land as part of the read-only `/sysroot` and `/boot` work. See ostreedev#1265

We want to support extending the read-only state to cover `/sysroot` and `/boot`, since conceptually all of the data there should only be written via libostree. This change needs to be opt-in though to avoid breaking anyone. Add a `sysroot/readonly` key to the repository config which instructs `ostree-remount.service` to ensure `/sysroot` is read-only. This requires a bit of a dance because `/sysroot` is actually the same filesystem as `/`; so we make `/etc` a writable bind mount in this case. We also need to handle `/var` in the "OSTree default" case of a bind mount; the systemd generator now looks at the writability state of `/sysroot` and uses that to determine whether it should have the `var.mount` unit happen before or after `ostree-remount.service.` Also add an API to instruct the libostree shared library that the caller has created a new mount namespace. This way we can freely remount read-write. This approach extends upon in a much better way previous work we did to support remounting `/boot` read-write. Closes: ostreedev#1265

Prep for using the default mount namespace handling there that will land as part of the read-only `/sysroot` and `/boot` work. See ostreedev#1265

We want to support extending the read-only state to cover `/sysroot` and `/boot`, since conceptually all of the data there should only be written via libostree. This change needs to be opt-in though to avoid breaking anyone. Add a `sysroot/readonly` key to the repository config which instructs `ostree-remount.service` to ensure `/sysroot` is read-only. This requires a bit of a dance because `/sysroot` is actually the same filesystem as `/`; so we make `/etc` a writable bind mount in this case. We also need to handle `/var` in the "OSTree default" case of a bind mount; the systemd generator now looks at the writability state of `/sysroot` and uses that to determine whether it should have the `var.mount` unit happen before or after `ostree-remount.service.` Also add an API to instruct the libostree shared library that the caller has created a new mount namespace. This way we can freely remount read-write. This approach extends upon in a much better way previous work we did to support remounting `/boot` read-write. Closes: ostreedev#1265

Let's opt-in to this by default. See ostreedev/ostree#1265 This currently is a no-op if the required ostree support hasn't landed yet, so I think we can safely merge this PR first.

We want to support extending the read-only state to cover `/sysroot` and `/boot`, since conceptually all of the data there should only be written via libostree. Or at least for `/boot` should *mostly* just be written by ostree. This change needs to be opt-in though to avoid breaking anyone. Add a `sysroot/readonly` key to the repository config which instructs `ostree-remount.service` to ensure `/sysroot` is read-only. This requires a bit of a dance because `/sysroot` is actually the same filesystem as `/`; so we make `/etc` a writable bind mount in this case. We also need to handle `/var` in the "OSTree default" case of a bind mount; the systemd generator now looks at the writability state of `/sysroot` and uses that to determine whether it should have the `var.mount` unit happen before or after `ostree-remount.service.` Also add an API to instruct the libostree shared library that the caller has created a new mount namespace. This way we can freely remount read-write. This approach extends upon in a much better way previous work we did to support remounting `/boot` read-write. Closes: ostreedev#1265

Prep for using the default mount namespace handling there that will land as part of the read-only `/sysroot` and `/boot` work. See ostreedev#1265

Let's opt-in to this by default. See ostreedev/ostree#1265 This currently is a no-op if the required ostree support hasn't landed yet, so I think we can safely merge this PR first.

This is all we need to tell libostree that we support a read-only `/sysroot` and `/boot`. See ostreedev/ostree#1265 PR in ostreedev/ostree#1767

Let's opt-in to this by default. See ostreedev/ostree#1265 This currently is a no-op if the required ostree support hasn't landed yet, so I think we can safely merge this PR first.

This is all we need to tell libostree that we support a read-only `/sysroot` and `/boot`. See ostreedev/ostree#1265 PR in ostreedev/ostree#1767

We want to use the new read-only `/sysroot` feature of libostree. Opt-in to that to tell cosa we support it and want it. For more details, see: ostreedev/ostree#1265 coreos/coreos-assembler#1235

bam80 · 2020-04-21T10:24:33Z

Yeah, we really should fix this. Right now, restorecon -R / and autorelabel=1 are super dangerous

I still broke my Fedora Silverblue 31 system with restorecon -R / command. Isn't it fixed yet?

jlebon · 2020-04-21T13:35:32Z

I still broke my Fedora Silverblue 31 system with restorecon -R / command. Isn't it fixed yet?

OSTree knows how to handle read-only /sysroot, but it doesn't automatically apply to existing nodes, because it requires modifying files not maintained by updates. We'll probably need a systemd service in fedora-release-silverblue or something to transition over existing systems.

bam80 · 2020-04-21T14:38:18Z

Thanks @jlebon. Could we dedicate a separate issue for that part of the task then?

We want to use the new read-only `/sysroot` feature of libostree. Opt-in to that to tell cosa we support it and want it. For more details, see: ostreedev/ostree#1265 coreos/coreos-assembler#1235

cgwalters mentioned this issue Oct 12, 2017

syscontainers: Fall back to destination projectatomic/atomic#1114

Closed

cgwalters added the enhancement label Oct 12, 2017

This was referenced Jan 4, 2018

Admin command somtimes becomes useless when more than two deployments exist #1394

Open

sysroot: Support /boot on / for syslinux and uboot #1404

Closed

cgwalters mentioned this issue Jan 16, 2018

ostree admin deploy: Add --no-prune option #1418

Closed

1 task

cgwalters mentioned this issue Oct 29, 2018

Support mounting /sysroot (and /boot) read-only #1767

Merged

cgwalters mentioned this issue Dec 26, 2018

WIP: image: Enable read-only /boot and /sysroot coreos/fedora-coreos-config#39

Closed

cgwalters added a commit to cgwalters/ignition-dracut that referenced this issue Dec 26, 2018

ignition-firstboot-complete.service: Remount /boot rw

a3922f9

Prep for ostreedev/ostree#1265

cgwalters mentioned this issue Dec 26, 2018

ignition-firstboot-complete.service: Remount /boot rw coreos/ignition-dracut#36

Merged

cgwalters added a commit to cgwalters/ignition-dracut that referenced this issue Jan 3, 2019

ignition-firstboot-complete.service: Remount /boot rw

650d58c

Prep for ostreedev/ostree#1265

cgwalters mentioned this issue Sep 5, 2019

create_disk: Enable read-only /sysroot coreos/coreos-assembler#736

Merged

openshift-merge-robot closed this as completed in 5af403b Dec 11, 2019

jlebon mentioned this issue Mar 16, 2020

image.yaml: enable sysroot-ro coreos/fedora-coreos-config#304

Merged

bam80 mentioned this issue Apr 20, 2020

Add initial troubleshooting guide fedora-silverblue/silverblue-docs#30

Merged

travier mentioned this issue Jan 21, 2022

Make Silverblue more robust against SELinux labels change breakage fedora-silverblue/issue-tracker#232

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extending read-only state to /sysroot and /boot #1265

Extending read-only state to /sysroot and /boot #1265

cgwalters commented Oct 12, 2017 •

edited

cgwalters commented Dec 23, 2017

giuseppe commented Dec 23, 2017

cgwalters commented Dec 27, 2017

giuseppe commented Dec 27, 2017

cgwalters commented Oct 29, 2018

cgwalters commented Dec 26, 2018

jlebon commented Apr 18, 2019 •

edited

goozbach commented Apr 18, 2019

jlebon commented Apr 18, 2019

goozbach commented Apr 19, 2019

bam80 commented Apr 21, 2020

jlebon commented Apr 21, 2020

bam80 commented Apr 21, 2020

Extending read-only state to /sysroot and /boot #1265

Extending read-only state to /sysroot and /boot #1265

Comments

cgwalters commented Oct 12, 2017 • edited

cgwalters commented Dec 23, 2017

giuseppe commented Dec 23, 2017

cgwalters commented Dec 27, 2017

giuseppe commented Dec 27, 2017

cgwalters commented Oct 29, 2018

cgwalters commented Dec 26, 2018

jlebon commented Apr 18, 2019 • edited

goozbach commented Apr 18, 2019

jlebon commented Apr 18, 2019

goozbach commented Apr 19, 2019

bam80 commented Apr 21, 2020

jlebon commented Apr 21, 2020

bam80 commented Apr 21, 2020

cgwalters commented Oct 12, 2017 •

edited

jlebon commented Apr 18, 2019 •

edited