skaffold verify ignores container environment variables on local minikube cluster

[ritsuki1227]

When I run skaffold verify with the following manifests, an environment variable will be ignored on my local minikube cluster.

Information

skaffold.yaml:

apiVersion: skaffold/v4beta6
kind: Config
metadata:
  name: foo
verify:
  - name: foo
    container: 
      name: foo
      image: alpine:latest
      command: ["sh", "-c", "echo env var is: $FOO"]
    executionMode:
      kubernetesCluster:
        jobManifestPath: job.yaml

job.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: foo
spec:
  template:
    spec:
      containers:
      - name: foo
        image: alpine:latest
        env:
        - name: FOO
          value: foo
      restartPolicy: Never

• Skaffold version: v2.6.2
• minikube version: v1.31.1
• Operating system: macOS 11.5.1
• Installed via: Homebrew

Expected behavior

Result of the echo: env var is: foo

Actual behavior

Result of the echo: env var is:

Steps to reproduce the behavior

$ skaffold verify -v debug
DEBU[0000] skaffold API not starting as it's not requested  subtask=-1 task=DevLoop
INFO[0000] Skaffold &{Version:v2.6.2 ConfigVersion:skaffold/v4beta6 GitVersion: GitCommit:0d544d26766058c5a49ba80907edf438a01c90d4 BuildDate:2023-07-18T19:56:24Z GoVersion:go1.20.6 Compiler:gc Platform:darwin/amd64 User:}  subtask=-1 task=DevLoop
INFO[0000] Loaded Skaffold defaults from "/my/home/path/.skaffold/config"  subtask=-1 task=DevLoop
DEBU[0000] parsed 1 configs from configuration file /my/example/path/skaffold.yaml  subtask=-1 task=DevLoop
DEBU[0000] Defaulting build type to local build          subtask=-1 task=DevLoop
DEBU[0000] Defaulting deploy type to kubectl             subtask=-1 task=DevLoop
INFO[0000] Using kubectl context: minikube               subtask=-1 task=DevLoop
DEBU[0000] getting client config for kubeContext: `minikube`  subtask=-1 task=DevLoop
INFO[0000] Using local-cluster=true from config          subtask=-1 task=DevLoop
DEBU[0000] getting client config for kubeContext: `minikube`  subtask=-1 task=DevLoop
DEBU[0000] Running command: [minikube version --output=json]  subtask=-1 task=DevLoop
DEBU[0000] Command output: [{"commit":"fd3f3801765d093a485d255043149f92ec0a695f","minikubeVersion":"v1.31.1"}
]  subtask=-1 task=DevLoop
DEBU[0000] Running command: [/usr/local/bin/minikube docker-env --shell none -p minikube --user=skaffold]  subtask=-1 task=DevLoop
DEBU[0001] Command output: [DOCKER_TLS_VERIFY=1
DOCKER_HOST=tcp://127.0.0.1:58724
DOCKER_CERT_PATH=/my/home/path/.minikube/certs
MINIKUBE_ACTIVE_DOCKERD=minikube
SSH_AUTH_SOCK=
SSH_AGENT_PID=
], stderr: Host added: /my/home/path/.ssh/known_hosts ([127.0.0.1]:58723)  subtask=-1 task=DevLoop
DEBU[0001] setting Docker user agent to skaffold-v2.6.2  subtask=-1 task=DevLoop
INFO[0001] Using minikube docker daemon at tcp://127.0.0.1:58724  subtask=-1 task=DevLoop
INFO[0001] no kpt renderer or deployer found, skipping hydrated-dir creation  subtask=-1 task=DevLoop
DEBU[0001] Running command: [kubectl config view --minify -o jsonpath='{..namespace}']  subtask=-1 task=DevLoop
DEBU[0001] Command output: ['default']          subtask=-1 task=DevLoop
DEBU[0001] CLI platforms provided: ""                    subtask=-1 task=DevLoop
DEBU[0001] platform detection from active kubernetes cluster is not enabled  subtask=-1 task=DevLoop
DEBU[0001] Using builder: local                          subtask=-1 task=DevLoop
DEBU[0001] push value not present in NewBuilder, defaulting to false because cluster.PushImages is false  subtask=-1 task=DevLoop
INFO[0001] build concurrency set to default value of -1  subtask=-1 task=DevLoop
DEBU[0001] getting client config for kubeContext: `minikube`  subtask=-1 task=DevLoop
DEBU[0001] getting client config for kubeContext: `minikube`  subtask=-1 task=DevLoop
DEBU[0001] getting client config for kubeContext: `minikube`  subtask=-1 task=DevLoop
[foo] env var is:
DEBU[0006] getting client config for kubeContext: `minikube`  subtask=-1 task=DevLoop
DEBU[0006] Running command: [kubectl --context minikube --namespace default delete pod --force --grace-period 0 --wait=true --selector job-name=foo]  subtask=-1 task=DevLoop
DEBU[0006] Command output: [No resources found
], stderr: Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.  subtask=-1 task=DevLoop

DEBU[0006] exporting metrics disabled                    subtask=-1 task=DevLoop

[ericzzzzzzz]

Hi @ritsuki1227 , to use verify with environment variable, you need to put your environment into a file. Then use --env-file to pass that file.
For example, your create a test_env file that contains:

FOO=123

run skaffold verify --env-file test_env , the value you set for FOO will be available in your container.
You can find more info with command skaffold verify --help

[ritsuki1227]

@ericzzzzzzz Thank you for sharing, it helps me a lot. Does it mean that if I deploy a Configmap or Secret by skaffold and set these values as environment variables in a container in the Kubertes Job manafest, they will be ignored by skaffold verify? That's not expected behavior for me.

[ericzzzzzzz]

Yes! Skaffold currently completely overrides job.Spec.Template.Spec.Containers defined in Job manifest with the one you defined in skaffold verify container stanza, which also means that you can set up environment variable from skaffold verify container stanza for you job container like,

apiVersion: skaffold/v4beta6
kind: Config
metadata:
  name: foo
verify:
  - name: foo
    container:
      name: foo
      image: alpine:latest
      command: ["sh", "-c", "echo env var is: $FOO"]
      env:
         - name: FOO
            value: foo
    executionMode:
      kubernetesCluster:
        jobManifestPath: job.yaml

related code.

skaffold/pkg/skaffold/verify/k8sjob/verify.go

Line 378 in 3b1aabc

job.Spec.Template.Spec.Containers = []corev1.Container{verifyContainerToK8sContainer(container)}

it seems a little strange to me, but I don't have any more context. Sorry.

[ericzzzzzzz]

Updated, we're currently discussing about how we want to handle this case. In the meantime, you can use configure overrides under executionMode.kubernetesCluster with inline-json to set up env with secret

something like
executionMode: kubernetesCluster: overrides: '{"spec": {"template" : {"spec": {"containers": ... }}}}'

related schema

https://skaffold.dev/docs/references/yaml/#customActions-executionMode-kubernetesCluster-overrides