gcp-auth leading to deployment failures from duplicate volumeMounts

[briandealwis]

@daveswersky reported a problem using minikube, gcp-auth and skaffold dev where Skaffold's redeploying of a changed image lead to a deployment failure due to a pod having a duplicate volumemount. I'm unable to reproduce, but it seems the admission controller should check that a container doesn't already have a necessary volumemount:

gcp-auth-webhook/server.go

Lines 157 to 171 in 1b44e85

	if needsCreds {
	if len(c.VolumeMounts) == 0 {
	patch = append(patch, patchOperation{
	Op: "add",
	Path: fmt.Sprintf("/spec/containers/%d/volumeMounts", i),
	Value: []corev1.VolumeMount{mount},
	})
	} else {
	patch = append(patch, patchOperation{
	Op: "add",
	Path: fmt.Sprintf("/spec/containers/%d/volumeMounts", i),
	Value: append(c.VolumeMounts, mount),
	})
	}
	}

$ skaffold dev
Listing files to watch...
 - skaffold-example
Generating tags...
 - skaffold-example -> skaffold-example:9e8762f
Checking cache...
 - skaffold-example: Found Locally
Tags used in deployment:
 - skaffold-example -> skaffold-example:34fabbca4745e9794878944aff798e65a07173516065d9a9164642a67547e894
Starting deploy...
 - pod/getting-started created
Waiting for deployments to stabilize...
 - pods is ready.
Deployments stabilized in 2.112 seconds
Press Ctrl+C to exit
Watching for changes...
[getting-started] Hello world!
[getting-started] Hello world!
[getting-started] Hello world!
[getting-started] Hello world!
[getting-started] Hello world!
[getting-started] Hello world!
[getting-started] Hello world!
[getting-started] Hello world!
<<<made a change, triggering a rebuild>>>
Generating tags...
 - skaffold-example -> skaffold-example:9e8762f-dirty
Checking cache...
 - skaffold-example: Found. Tagging
Tags used in deployment:
 - skaffold-example -> skaffold-example:023c1c3c47c45f1515a47833d8087a33265b392ac3e87b657ede9bffc2da461b
Starting deploy...
 - The Pod "getting-started" is invalid: 
 - * spec.volumes[2].name: Duplicate value: "gcp-creds"
 - * spec.containers[0].volumeMounts[2].mountPath: Invalid value: "/google-app-creds.json": must be unique
 - * spec: Forbidden: pod updates may not change fields other than spec.containers[*].image, spec.initContainers[*].image, spec.activeDeadlineSeconds or spec.tolerations (only additions to existing tolerations)
 -   core.PodSpec{
 -      Volumes: []core.Volume{
 -              {Name: "default-token-5bm8g", VolumeSource: core.VolumeSource{Secret: &core.SecretVolumeSource{SecretName: "default-token-5bm8g", DefaultMode: &420}}},
 -              {Name: "gcp-creds", VolumeSource: core.VolumeSource{HostPath: &core.HostPathVolumeSource{Path: "/var/lib/minikube/google_application_credentials.json", Type: &"File"}}},
 - -            {
 - -                    Name: "gcp-creds",
 - -                    VolumeSource: core.VolumeSource{
 - -                            HostPath: &core.HostPathVolumeSource{Path: "/var/lib/minikube/google_application_credentials.json", Type: &"File"},
 - -                    },
 - -            },
 -      },
 -      InitContainers: nil,
 -      Containers: []core.Container{
 -              {
 -                      ... // 5 identical fields
 -                      Ports:   nil,
 -                      EnvFrom: nil,
 -                      Env: []core.EnvVar{
 -                              ... // 4 identical elements
 -                              {Name: "GOOGLE_CLOUD_PROJECT"},
 -                              {Name: "CLOUDSDK_CORE_PROJECT"},
 - -                            {Name: "GOOGLE_APPLICATION_CREDENTIALS", Value: "/google-app-creds.json"},
 - -                            {Name: "PROJECT_ID"},
 - -                            {Name: "GCP_PROJECT"},
 - -                            {Name: "GCLOUD_PROJECT"},
 - -                            {Name: "GOOGLE_CLOUD_PROJECT"},
 - -                            {Name: "CLOUDSDK_CORE_PROJECT"},
 -                      },
 -                      Resources: core.ResourceRequirements{},
 -                      VolumeMounts: []core.VolumeMount{
 -                              {Name: "default-token-5bm8g", ReadOnly: true, MountPath: "/var/run/secrets/kubernetes.io/serviceaccount"},
 -                              {Name: "gcp-creds", ReadOnly: true, MountPath: "/google-app-creds.json"},
 - -                            {Name: "gcp-creds", ReadOnly: true, MountPath: "/google-app-creds.json"},
 -                      },
 -                      VolumeDevices: nil,
 -                      LivenessProbe: nil,
 -                      ... // 10 identical fields
 -              },
 -      },
 -      EphemeralContainers: nil,
 -      RestartPolicy:       "Always",
 -      ... // 25 identical fields
 -   }
 - 
WARN[0013] Skipping deploy due to error:kubectl apply: exit status 1  subtask=-1 task=DevLoop
Watching for changes...

[daveswersky]

• Brand-new MBP laptop (replacement)
• Just installed Docker, minikube, Skaffold, etc
• Working with Skaffold example app: https://github.com/GoogleContainerTools/skaffold
• First deploy always works, any change to example main.go (changing the text of the println) triggers a rebuild but the deploy fails (see error above)
• Shut down Docker & minikube, started over after making a change to main.go: same pattern, first deploy succeeds, subsequent changes fail
• Disabling gcp-auth addon resolved the issue, changes successfully trigger build and deploy
• minikube v1.24.0
• skaffold v1.34.0

[sharifelgamal]

Yeah, this is definitely a legitimate issue. I'll see if I can fix it asap.