fix: properly reload persistent snapshotter data and restart services

[pendo324]

Issue #, if available: re-verified #412

• Through extensive e2e test debugging, I noticed that soci and stargz snapshotters weren't persisting data as expected. After debugging, I found some context in these two PRs:
  • Keep directories when SIGINT sent to daemon awslabs/soci-snapshotter#881
  • Keep directories when SIGINT sent to daemon containerd/stargz-snapshotter#1526
    Unfortunately, neither of them are deployed yet, so I've implemented a hacky workaround for now. After this change, an image/container can be pull/run, the VM can be restarted, and then the container can be re-started again.

Description of changes:

• Redo how BuildKit/Stargz/SOCI are related to containerd using systemd's PartOf
  
  • this ensures that all of these services are restarted when containerd is restarted, which the lack of has caused errors in the past
• Create some missing directories that might throw errors in cloud-init
• Ensure that SIGTERM is used to kill the snapshotter services for now

Testing done:

• manual testing

• I've reviewed the guidance in CONTRIBUTING.md

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[weikequ]

Redo how BuildKit/Stargz/SOCI are related to containerd using systemd's PartOf

Nice work! Thanks for the PR

[pendo324]

Looks like the persistence fix had some unintended consequences :(. I'll keep looking into it

[pendo324]

Interesting, seems like the e2e test is failing spuriously? I tried to repro the basic scenario on my local machine, and it only occurs when I stop my VM before stopping the container. There may be some metadata we can clean up to handle this, but for now, starting it 3 times has consistently fixed it for me.

Wondering if #759 fixes the e2e test failure.

[pendo324]

Although I'm not yet convinced that this is a real issue (probably just a bad test?) I did some more digging. This is also relevant for #219.

After reading through the source code of the cni plugins (specifically, looking through every instance where the state is cleaned) I found what the lingering state is when you restart your machine while a container is running.

We can add (maybe a more careful version) of this cleanup code:

sudo rm /var/lib/cni/networks/bridge/**
sudo rm /var/lib/cni/results/bridge-finch-*

the bridge dir holds information like:

$ sudo ls /var/lib/cni/networks/bridge/
10.4.0.2            last_reserved_ip.0  lock

where each file is empty and 10.4.0.2 represents the address of the container on the (default) nerdctl0 bridge network.

This workaround completely fixes the network issue, and we can add this command on boot/reboot (need to think further on a fix for lima/nerdctl in general).

However, even after fixing that, there's an other issue where the filesystem of the container will be in a weird limbo state where the files "exist" but they have no content if the system is rebooted while the container is running.

Luckily, this recent PR (containerd/containerd#9401) for containerd fixes this issue by forcing Linux to flush to disk. We just need to wait for nerdctl 2.0 to be adopted by Lima, or wait for the PR to get backported into containerd 1.7.

[sondavidb]

In case it helps, v0.5.0 of SOCI just released on Friday, which includes this fix. We'd still have to wait for the fix in stargz, though.

[pendo324]

Seems like my workaround of using systemd services that run at boot and on shutdown worked, and this has the added benefit of fixing #219

[weikequ]

Nice job on the investigation! Do you think it could be useful to file this information away somewhere? Would be nice to put it in an internal kb or something.

[pendo324]

Overriding since there are still lingering unrun old test cases on this PR.