Dangling stargz fuse mounts

[dmosdallas]

Background

I've been playing with the stargz snapshotter over the past few days and, mostly running containers via docker.

Problem

After many containers started/stopped/removed, I've noticed that there are now some dangling stargz fuse mounts that I am unable to clean up.

Things I've tried

• Usually running docker rmi or docker system prune -a causes the mounted up images to be cleaned. This had worked for me until yesterday. Now, even after running the very aggressive system prune, there are still dangling mounts
• I had also been playing with containerd & buildctl directly, so I thought I may have something left over there. Running buildctl prune, ctr i rm, ctr-remote i rm to flush out everything also had no effect.
• I restarted the stargz-snapshotter service, but the mounts remained.
• I rebooted the machine and the mounts were re-created.
• I manually unmounted the dangling mountpoints & they were re-created after a subsequent reboot.

Question

Is there a way to track down what is keeping these dangling mounts alive?

[ktock]

@dmosdallas Does docker buildx prune work?

[dmosdallas]

@ktock, nope, that doesn't clear out the dangling mounts for me :/

[ktock]

@dmosdallas Could you please post small reproducing steps of the issue?

[dmosdallas]

@ktock, sadly, I'm not entirely sure.

I've been doing a bunch of experiments with building & running containers with this snapshotter, so this could be caused by abnormal things I've done. Some notes on what I've done that may help us narrow down the root cause:

• I've been testing this both on my own workstation and on a vm in aws. I've only seen this issue on my workstation.
• On my workstation, I've ctrl+c'd out of builds when I notice it's not doing what I want (e.g., when it starts pulling down the entire base image. Maybe this is causing things to not be cleaned up correctly?
• Looking at the contents of the snapshot directories, it looks like these are leftover layers from when I was doing a docker buildx build. This was before I was using the builder you mentioned in Is it possible to lazy pull eStargz images during (docker buildx) build #1390 (comment)

[dmosdallas]

I've not been able to reproduce this issue in the past week. I think it's likely due to me doing something weird while I was setting up the snapshotter initially. I'll close this issue until I can come up with real reproduction steps.