garden-util pods not found

[to266]

Bug

Current Behavior

On a fresh environment, nearly always the first run fails with this:

ℹ garden-util               → Resources ready
-> Deploying garden-util service in ci-116104942-eee536a0 namespace (was missing) → Done!
Failed building rust-workspace. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Could not find a running Pod in Deployment garden-util
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Comparing expected and deployed resources...
All resources match.
Failed building python-poetry. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Could not find a running Pod in Deployment garden-util
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Your environment

• OS: linux, gitlab runners
• How I'm running Kubernetes: AKS

Using docker image sha256:18a78542503a222ff1d128fd7abdf9fdae5e1f11dcfa9e17381da7ab4480776a for gardendev/garden-azure:0.12.25

[eysi09]

Hi @to266

Hmm, that is odd. Would you mind sharing your build mode and other relevant project configuration?

It would also be useful to see the output from kubectl get pods in the namespace and the state of the garden-util pod.

[to266]

Actually I don't see this behavior happening any more, which is a bit strange - but I'm not complaining. I would guess it's something peculiar with the setup we have, however even that did not change much. The biggest difference between now and the (occasionally) failing state is that back then we would tend to use many low-resource nodes (that had the strange drawback that pod startups were extra slow), whereas now we have fewer more powerful nodes.

I think the most special thing about the setup, is that garden-util seem to completely inherit both the nodeSelector as well as tolerations from the kaniko part of the setup, which in our case means they always try (and pretty quickly do) schedule on the spot instances.

As we don't see the issue any more, feel free to close this, if no additional investigation is required. I'll open and try to get more info if this repeats

[eysi09]

Thanks for the update. That might indeed be the case. I'll close for now but please re-open if you bump into this again.