Are modules broken on 2.9.0, 2.8.0? skaffold.yaml deploys, but fails to deploy when require:d as a module

[tarasglek]

Expected behavior

skaffold run -f skaffold.yaml and skaffold run -f skaffold-namespace.yaml both create cert-manager namespace.

I'm new to skaffold, confused that currently released version doesn't behave as I would expect from docs and integration test

skaffold/integration/testdata/diagnose/multi-config/skaffold.yaml

Line 4 in 6632e12

- path: ./skaffold2.yaml

Actual behavior

skaffold renders correct yaml in both cases but deploy of outer skaffold.yaml doesn't deploy anything

Information

• Skaffold version: 2.8.0, 2.9.0
• Operating system: Linux and MacOS arm64
• Installed via: skaffold.dev
• Contents of skaffold.yaml:

apiVersion: skaffold/v4beta8
kind: Config
requires:
  - path: skaffold-namespace.yaml

cat skaffold-namespace.yaml

apiVersion: skaffold/v4beta8
kind: Config
metadata:
    name: misc-helms
manifests:
    rawYaml:
        - namespace.yaml

cat namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
    name: cert-manager

Steps to reproduce the behavior

Failing to deploy as expected:
skaffold run -f skaffold.yaml -v debug

DEBU[0000] skaffold API not starting as it's not requested  subtask=-1 task=DevLoop
INFO[0000] Skaffold &{Version:v2.9.0 ConfigVersion:skaffold/v4beta8 GitVersion: GitCommit:6071a3f7574702c8666a243d89254e9b0d8ff4d7 BuildDate:2023-11-07T13:20:11Z GoVersion:go1.21.0 Compiler:gc Platform:darwin/arm64 User:}  subtask=-1 task=DevLoop
INFO[0000] Loaded Skaffold defaults from "/Users/taras/.skaffold/config"  subtask=-1 task=DevLoop
DEBU[0000] parsed 1 configs from configuration file /Users/taras/Downloads/skaffold/skaffold.yaml  subtask=-1 task=DevLoop
DEBU[0000] Defaulting build type to local build          subtask=-1 task=DevLoop
DEBU[0000] parsed 1 configs from configuration file /Users/taras/Downloads/skaffold/skaffold-namespace.yaml  subtask=-1 task=DevLoop
DEBU[0000] Defaulting build type to local build          subtask=-1 task=DevLoop
INFO[0000] Using kubectl context: default                subtask=-1 task=DevLoop
DEBU[0000] getting client config for kubeContext: `default`  subtask=-1 task=DevLoop
DEBU[0000] Running command: [minikube version --output=json]  subtask=-1 task=DevLoop
DEBU[0000] setting Docker user agent to skaffold-v2.9.0  subtask=-1 task=DevLoop
INFO[0000] DOCKER_HOST env is not set, using the host from docker context.  subtask=-1 task=DevLoop
DEBU[0000] Running command: [docker context inspect --format {{.Endpoints.docker.Host}}]  subtask=-1 task=DevLoop
WARN[0000] Could not get docker context: starting command docker context inspect --format {{.Endpoints.docker.Host}}: exec: "docker": executable file not found in $PATH, falling back to the default docker host  subtask=-1 task=DevLoop
INFO[0000] no kpt renderer or deployer found, skipping hydrated-dir creation  subtask=-1 task=DevLoop
DEBU[0000] Running command: [kubectl config view --minify -o jsonpath='{..namespace}']  subtask=-1 task=DevLoop
DEBU[0000] CLI platforms provided: ""                    subtask=-1 task=DevLoop
DEBU[0000] getting client config for kubeContext: `default`  subtask=-1 task=DevLoop
DEBU[0000] platforms detected from active kubernetes cluster nodes: "linux/amd64"  subtask=-1 task=DevLoop
DEBU[0000] Using builder: local                          subtask=-1 task=DevLoop
DEBU[0000] push value not present in NewBuilder, defaulting to true because cluster.PushImages is true  subtask=-1 task=DevLoop
DEBU[0000] Using builder: local                          subtask=-1 task=DevLoop
DEBU[0000] push value not present in NewBuilder, defaulting to true because cluster.PushImages is true  subtask=-1 task=DevLoop
INFO[0000] build concurrency set to default value of -1  subtask=-1 task=DevLoop
No tags generated
INFO[0000] Tags generated in 14.833µs                    subtask=-1 task=Build
INFO[0000] Cache check completed in 15.667µs             subtask=-1 task=Build
Starting test...
INFO[0000] Test completed in 9.292µs                     subtask=-1 task=Test
INFO[0000] Starting render...                            subtask=-1 task=DevLoop
INFO[0000] starting render process                       subtask=0 task=Render
DEBU[0000] Executing template &{envTemplate 0x14000bac7e0 0x14000489d10  } with environment map[HOME:/Users/taras PWD:/Users/taras/Downloads/skaffold SHLVL:1 _:/usr/local/bin/skaffold]  subtask=-1 task=DevLoop
DEBU[0000] manifests with tagged images:apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager  subtask=0 task=Render
DEBU[0000] manifests with labels apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager  subtask=-1 task=DevLoop
INFO[0000] Render completed in 632.292µs                 subtask=-1 task=DevLoop
Starting deploy...
INFO[0000] Deploy completed in 5.875µs                   subtask=-1 task=Deploy
You can also run [skaffold run --tail] to get the logs

DEBU[0000] exporting metrics                             subtask=-1 task=DevLoop
DEBU[0001] metrics uploading complete in 757.461917ms    subtask=-1 task=DevLoop

Working:

skaffold run -f skaffold-namespace.yaml -v debug

DEBU[0000] skaffold API not starting as it's not requested  subtask=-1 task=DevLoop
INFO[0000] Skaffold &{Version:v2.9.0 ConfigVersion:skaffold/v4beta8 GitVersion: GitCommit:6071a3f7574702c8666a243d89254e9b0d8ff4d7 BuildDate:2023-11-07T13:20:11Z GoVersion:go1.21.0 Compiler:gc Platform:darwin/arm64 User:}  subtask=-1 task=DevLoop
INFO[0000] Loaded Skaffold defaults from "/Users/taras/.skaffold/config"  subtask=-1 task=DevLoop
DEBU[0000] parsed 1 configs from configuration file /Users/taras/Downloads/skaffold/skaffold-namespace.yaml  subtask=-1 task=DevLoop
DEBU[0000] Defaulting build type to local build          subtask=-1 task=DevLoop
DEBU[0000] Found raw k8s manifests without cloud run deploy, adding kubectl deployer  subtask=-1 task=DevLoop
INFO[0000] Using kubectl context: default                subtask=-1 task=DevLoop
DEBU[0000] getting client config for kubeContext: `default`  subtask=-1 task=DevLoop
DEBU[0000] Running command: [minikube version --output=json]  subtask=-1 task=DevLoop
DEBU[0000] setting Docker user agent to skaffold-v2.9.0  subtask=-1 task=DevLoop
INFO[0000] DOCKER_HOST env is not set, using the host from docker context.  subtask=-1 task=DevLoop
DEBU[0000] Running command: [docker context inspect --format {{.Endpoints.docker.Host}}]  subtask=-1 task=DevLoop
DEBU[0000] Command output: [unix:///Users/taras/.docker/run/docker.sock
]  subtask=-1 task=DevLoop
INFO[0000] no kpt renderer or deployer found, skipping hydrated-dir creation  subtask=-1 task=DevLoop
DEBU[0000] Running command: [kubectl config view --minify -o jsonpath='{..namespace}']  subtask=-1 task=DevLoop
DEBU[0000] Command output: ['']                          subtask=-1 task=DevLoop
DEBU[0000] CLI platforms provided: ""                    subtask=-1 task=DevLoop
DEBU[0000] getting client config for kubeContext: `default`  subtask=-1 task=DevLoop
DEBU[0000] platforms detected from active kubernetes cluster nodes: "linux/amd64"  subtask=-1 task=DevLoop
DEBU[0000] Using builder: local                          subtask=-1 task=DevLoop
DEBU[0000] push value not present in NewBuilder, defaulting to true because cluster.PushImages is true  subtask=-1 task=DevLoop
INFO[0000] build concurrency set to default value of -1  subtask=-1 task=DevLoop
No tags generated
INFO[0000] Tags generated in 11.792µs                    subtask=-1 task=Build
INFO[0000] Cache check completed in 489.541µs            subtask=-1 task=Build
Starting test...
INFO[0000] Test completed in 11.875µs                    subtask=-1 task=Test
INFO[0000] Starting render...                            subtask=-1 task=DevLoop
INFO[0000] starting render process                       subtask=0 task=Render
DEBU[0000] Executing template &{envTemplate 0x14000c50000 0x140005f3e50  } with environment map[HOME:/Users/taras PATH:/usr/gnu/bin:/usr/local/bin:/bin:/usr/bin:.:/usr/local/bin/kubectl:/usr/local/bin/ PWD:/Users/taras/Downloads/skaffold SHLVL:1 _:/usr/local/bin/skaffold]  subtask=-1 task=DevLoop
DEBU[0000] manifests with tagged images:apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager  subtask=0 task=Render
DEBU[0000] manifests with labels apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager  subtask=-1 task=DevLoop
INFO[0000] Render completed in 1.072ms                   subtask=-1 task=DevLoop
Starting deploy...
DEBU[0000] getting client config for kubeContext: `default`  subtask=-1 task=DevLoop
DEBU[0000] Running command: [kubectl --context default get -f - --ignore-not-found -ojson]  subtask=0 task=Deploy
DEBU[0001] Command output: [{
    "apiVersion": "v1",
    "items": [
        {
            "apiVersion": "v1",
            "kind": "Namespace",
            "metadata": {
                "annotations": {
                    "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"cert-manager\"}}\n"
                },
                "creationTimestamp": "2023-12-23T20:58:55Z",
                "labels": {
                    "kubernetes.io/metadata.name": "cert-manager"
                },
                "name": "cert-manager",
                "resourceVersion": "102066",
                "uid": "c3fd1f04-a4e6-4f2f-981c-16655ebdf30c"
            },
            "spec": {
                "finalizers": [
                    "kubernetes"
                ]
            },
            "status": {
                "phase": "Active"
            }
        }
    ],
    "kind": "List",
    "metadata": {
        "resourceVersion": ""
    }
}
]  subtask=0 task=Deploy
DEBU[0001] 1 manifests to deploy. 1 are updated or new   subtask=0 task=Deploy
DEBU[0001] Running command: [kubectl --context default apply -f -]  subtask=0 task=Deploy
 - namespace/cert-manager unchanged
Waiting for deployments to stabilize...
DEBU[0001] getting client config for kubeContext: `default`  subtask=-1 task=DevLoop
DEBU[0001] getting client config for kubeContext: `default`  subtask=-1 task=DevLoop
Deployments stabilized in 251.710042ms
INFO[0002] Deploy completed in 1.28 second               subtask=-1 task=Deploy
DEBU[0002] getting client config for kubeContext: `default`  subtask=-1 task=DevLoop
You can also run [skaffold run --tail] to get the logs

DEBU[0002] exporting metrics                             subtask=-1 task=DevLoop
DEBU[0003] metrics uploading complete in 817.253541ms    subtask=-1 task=DevLo

[hgaron-blockbar]

I am having this same issue on 2.9.0; helm deployments still work in modules but rawYaml does not. If I move rawYaml to the root skaffold.yaml it will deploy correctly, but if it's inside a module, no such luck.

I have also verified skaffold correctly renders the yaml by running skaffold render > rendered.yaml and verifying the output. But any raw yaml just isn't deployed.

EDIT:
After a good deal of debugging, it looks like the rawYaml only works on the root skaffold.yaml if the requires key is omitted entirely. Even having a requires key with profiles that are not invoked breaks the rawYaml property, regardless of whether it is defined in a module or at the root skaffold.yaml.

For reference, I am also using apiVersion: skaffold/v4beta8.

[hgaron-blockbar]

I downgraded to skaffold v2.8.0 and the issue persists there as well.

[renzodavid9]

Hello there! This is an intended behaviour: when using Skaffold modules, Skaffold itself is not able to add a default deployer. To make it work you'll need to add a deployer explicitly in the modules (deploy.*). For more context please refer to #9215 and #9138 . Thanks! 😄

[hgaron-blockbar]

@renzodavid9 Forgive me but I think I misunderstand. I do have a Helm deployer present in my modules deploy.* and that works fine, but for the raw yaml manifests I'm not sure how to add a deployer?

In the v4beta8 schema, if I add a kubectl deployer there's no explicit way to define manifests (only a remoteManifests key exists). Should I just define it with an empty object?

ie.

manifests:
  rawYaml:
    - ../../manifests/*
deploy:
  kubectl: {}

Thanks for your help; cheers!

[renzodavid9]

Hey @hgaron-blockbar! Yep, the example you added is exactly the way to configure the kubectl deployer for your use case. Behind the scene, Skaffold will first, render the manifests defined in manifests.rawYaml, and then will send them to the deploy.kubectl deployer. Hope that solves your use case. Thanks!

[renzodavid9]

I'll proceed to close this issue; we have #9138 to improve the messages for this case. Please feel free to re-open or create a new issue if needed. Thanks!