handle -killed- preload side-car holding on to the "minikube' volume preventing volume from being deleted

[medyagh]

first noticed when I apt-get thought "vim" was installed on a Fresh minikube !!! (however not it was installed in path)
and the reason was it was using the same volume as previous deleted minikube

 $ minikube ssh
Last login: Sun Feb  7 21:58:01 2021 from 192.168.49.1
docker@minikube:~$ vim  
-bash: vim: command not found
docker@minikube:~$ sudo apt-get update && sudo apt-get install -y vim haproxy
...
...
...
vim is already the newest version (2:8.1.2269-1ubuntu5).
haproxy is already the newest version (2.0.13-2ubuntu0.1).

 $ minikube delete --all
🔥  Successfully deleted all profiles

 $ docker volume ls 
DRIVER    VOLUME NAME
local     minikube


$ docker volume rm --force minikube
Error response from daemon: remove minikube: volume is in use - [b4697cfa9aa1c955140909fb0a9cbd6b02291d308a462233fc2d2f9034e6f39b]

further debugging showed that I have a volume attached to a random name container (that seems to be the side-car for preload)

 $ docker ps -a

CONTAINER ID   IMAGE                                 COMMAND                  CREATED      STATUS    PORTS     NAMES
b4697cfa9aa1   gcr.io/k8s-minikube/kicbase:v0.0.17   "/usr/bin/test -d /v…"   2 days ago   Created             wonderful_galileo

I purpose we give a name to the side-car for example "profilename-preload-sidecar" so when we try to delete the cluster using minikube delete or minikube delete --all we can clean up possible left overs of side car.

CC: @priyawadhwa

[medyagh]

here is where in the code that we create side Car
https://github.com/medyagh/minikube/blob/8f04657007a61b32d80b6bcf6670c44784a503d0/pkg/drivers/kic/oci/volumes.go#L112

feel free to rename the func to something more meaningful. maybe prepareVolumeSideCar, we need to add a Container Name to it (so docker doesnt choose random names for us) and then in the clean up process, if user calls minikube delete, it should try to clean up the side-car container tooo profilename-preload-sidecar

// prepareVolume will copy the initial content of the mount point by starting a container to check the expected content
func prepareVolume(ociBin string, imageName string, nodeName string) error {
	cmdArgs := []string{"run", "--rm", "--entrypoint", "/usr/bin/test"}
	cmdArgs = append(cmdArgs, "-v", fmt.Sprintf("%s:/var", nodeName), imageName, "-d", "/var/lib")
	cmd := exec.Command(ociBin, cmdArgs...)
	if _, err := runCmd(cmd); err != nil {
		return err
	}
	return nil
}

[govargo]

/assign

[govargo]

I cannot reproduce this behaviour.
But I agree fixing this sidecar name in order to prevent unexpected behaviour.

My reproduce step

1. Delete minikube home directory
$ rm -rf ~/.minikube

2. Start Minikube with Docker driver
$ minikube start --driver docker

3. SSH and install vim at the first time
$ minikube ssh
docker@minikube:~$ sudo apt-get update && sudo apt-get install -y vim haproxy
(new install start)

4. Delete minikube cluster
$ minikube delete --all

5. Check if the docker volume remains
$ docker volume inspect minikube
[]
Error: No such volume: minikube

6. Restart Minikube cluster
$ minikube start --driver docker

7. SSH and vim install at the second time
$ sudo apt-get update && sudo apt-get install -y vim haproxy
(new install start)

If it's possible, please share the reproduce step with me.
Anyway, I'll work on this.

[medyagh]

@govargo I think one way to reproduce this is, to remove the -rm from

cmdArgs := []string{"run", "--rm", "--entrypoint", "/usr/bin/test"}

so the Side container does not get deleted. (this is reproducing when the container is created and while it is doing extracting it the Docker Service is Quited or killed so it doesnt get chance to remove it self) and it is get stuck in "Created" phase