custom addon images get ignored on minikube restart

[clamprecht01]

**Note this environment is entirely air-gapped.
minikube start --driver=docker --base-image="<local_IP>:5000/gcr.io/k8s-minikube/kicbase:v0.0.18" --insecure-registry="<local_IP>:5000" --image-repository="<local_IP>:5000" --alsologtostderr 2>&1 >> minikube_clean_startup_logs.log

Note this capability added with PR-10111
Steps to reproduce the issue:

1. Start minikube for offline environment (above)

2. Enable ingress addones (with --image flag)
   minikbe_ingress_addon_enabled.log

3. Stop minikube

4. Start Minikube (with or without flags, same result) minikube_bad_startup_logs.log

5. Ingress container fails to start (wrong image)
   Minikube_ingress_failed.txt

Full output of failed command:
Alsologtostderr is already included.

Full output of minikube start command used, if not already included:

Optional: Full output of minikube logs command:

Workaround: (disabling and re-adding ingress fixes it) 1. minikube addons disable ingress 2. minikube addons enable ingress --images=

[sharifelgamal]

I can verify this is broken at HEAD. We should fix this ASAP. Help wanted!

[andriyDev]

/assign andriyDev

[r4j4h]

Further verifying, even in a completely non-air gapped environment.

Also, this may be a separate bug but a similar issue affects running configure after using custom images.

$ minikube image load metallb/controller:v0.9.6
$ minikube image load metallb/speaker:v0.9.6
$ minikube addons enable metallb --images="Controller=metallb/controller:v0.9.6,Speaker=metallb/speaker:v0.9.6"
    ▪ Using image metallb/controller:v0.9.6
    ▪ Using image metallb/speaker:v0.9.6
🌟  The 'metallb' addon is enabled
$ minikube addons configure metallb --profile=${PROFILE}
    ▪ Using image metallb/speaker:v0.8.2
    ▪ Using image metallb/controller:v0.8.2
✅  metallb was successfully configured

After resetting the image by re-enabling, identically to this bug after minikube restart they are also set back :(

$ minikube addons enable metallb --images="Controller=metallb/controller:v0.9.6,Speaker=metallb/speaker:v0.9.6"
  ▪ Using image metallb/speaker:v0.9.6
  ▪ Using image metallb/controller:v0.9.6
🌟  The 'metallb' addon is enabled
$ minikube stop
$ minikube start
🔎  Verifying Kubernetes components...
     .....
    ▪ Using image metallb/speaker:v0.8.2
    ▪ Using image metallb/controller:v0.8.2

[clamprecht01]

@r4j4h That seems like the same bug. Appears to not apply to just air-gapped environments.

[andriyDev]

Simplified reproduction steps:

$ minikube addons enable dashboard --images="Dashboard=k8s.gcr.io/echoserver:1.4"
    ▪ Using image k8s.gcr.io/echoserver:1.4
    ▪ Using image kubernetesui/metrics-scraper:v1.0.4
💡  Some dashboard features require the metrics-server addon. To enable all features please run:

	minikube addons enable metrics-server	


🌟  The 'dashboard' addon is enabled
$ kubectl describe deploy/kubernetes-dashboard -nkubernetes-dashboard
<stuff>
    Image:      k8s.gcr.io/echoserver:1.4
<stuff>
$ minikube stop
✋  Stopping node "minikube"  ...
🛑  Powering off "minikube" via SSH ...
🛑  1 nodes stopped.
$ minikube start
😄  minikube v1.20.0 on Debian rodete
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.6 ...
🔎  Verifying Kubernetes components...
    ▪ Using image kubernetesui/dashboard:v2.1.0
    ▪ Using image kubernetesui/metrics-scraper:v1.0.4
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass, dashboard
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
$ kubectl describe deploy/kubernetes-dashboard -nkubernetes-dashboard
<stuff>
    Image:      kubernetesui/dashboard:v2.1.0@sha256:7f80b5ba141bead69c4fee8661464857af300d7d7ed0274cf7beecedc00322e6
<stuff>