[BUG] killing soci-snapshotter-grpc while a container is running requires manual cleanup

[sparr]

Describe the bug
When the soci process is killed while a container is running, some of the mounts and metadata are left in a problematic state.

Steps To Reproduce

1. soci-snapshotter-grpc
2. ctr run --snapshotter soci [...]
3. killall soci-snapshotter-grpc

Expected behavior
All state (running processes, files, container and image metadata in the data store and registry, etc) is left in a state allowing the snapshotter and container to be started again without manual intervention.

Additional context
Problem statement is non-specific because I did not take sufficient notes when encountering this problem while troubleshooting a more pressing issue. Investigating will be a necessary step of resolving this issue.

[rdpsin]

The problem is that FUSE mounts' lifetimes are tied to the snapshotter. If the snapshotter crashes, the FUSE mounts will die too and we have no way to re-mount them. A couple of options are:

1. Separate out the FUSE implementation from the snapshotter, so that the mounts still exist even if the snapshotter crashes.

2. Persist some kind of state on disk that will allow the snapshotter to reconstruct the FUSE mount whenever it comes back online.

[Kern--]

Is this the same problem?
#93

[Kern--]

I think the "solution" to this is the config to ignore the broken data: https://github.com/awslabs/soci-snapshotter/blob/main/config/service.go#L79

Maybe we could also have the SOCI snapshotter call to containerd to remove the broken snapshots? That seems a bit weird, though.