How to fix ext4 root filesystem corruption in container machines?

[harry-xm]

I got the following error in a container machine, which seems to be filesystem corruption. How can I fix it?

[<user>@almalinux-machine-10 ~]$ rm /home/<user>/.codex/memories_1.sqlite-*
rm: cannot remove '/home/<user>/.codex/memories_1.sqlite-shm': Structure needs cleaning
rm: cannot remove '/home/<user>/.codex/memories_1.sqlite-wal': Structure needs cleaning

[JehadurRE]

Structure needs cleaning means ext4 metadata corruption in the container filesystem. Since this is a container machine (not a VM with a block device you can fsck directly), a few options:

1. If the container has a persistent volume, try running fsck from inside the container if e2fsprogs is available: fsck -y /path/to/mounted/fs. You may need to unmount first or boot into a recovery container.

2. If the corruption is only in that specific sqlite files directory and you do not care about the data, you can try removing the entire .codex directory rather than individual files: rm -rf /home/user/.codex

3. If rm still fails with the same error, the corruption may be in the directory inode itself. In that case the cleanest fix is to recreate the container machine from scratch, since container filesystems are meant to be disposable.

4. If this keeps happening, check whether the host filesystem (where the container overlay lives) has errors - container fs corruption often points to underlying host disk issues.

[harry-xm]

rm -rf still fails, but not on the directory. Do you know how to boot into a recovery container?

"Meant to be disposable" doesn't apply to container machines, which purport to be long-lived Linux environments.

[JehadurRE]

you are right, my bad on the disposable comment. container machines are persistent Linux VMs, not throwaway containers.

on the recovery question - there is no built-in recovery mode documented. but here is what you can try based on how the filesystem works:

the container machine root filesystem is an ext4 image stored on your Mac, likely under ~/Library/Application Support/com.apple.container/. you can find the exact path with:

container machine inspect

look for a source or storage path in the JSON output.

once you have the image path, options are:

1. stop the machine, then run fsck on the image from macOS using a Linux VM or docker container that has e2fsprogs. you cannot fsck a mounted filesystem from inside the running machine.

2. try root access first - container machine run -n --root - then attempt rm -rf /home/user/.codex from there. root may have better luck with corrupted inodes than your user account.

3. if that fails, export what you can with container export, then recreate the machine and copy the good files back. this is the nuclear option but preserves your data.

4. if the corruption keeps recurring, that points to an underlying issue with how the ext4 image is being written - worth filing an issue on the repo with your machine inspect output.

the sqlite shm/wal files being corrupted suggests the database was mid-write when something went wrong. if you do not need the codex data, option 2 is the fastest path.

[harry-xm]

Could you reproduce the issue? If so, file an issue on the repo.