Crash: getaddrinfo ENOTFOUND 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com

[jghowe]

Crash report

Error message

Payload:
  {"anonymousId":"d85a6ca6-c782-4f6e-938f-20ffab2101c9","event":"Command Result","properties":{"projectId":"67663e0418ebc48269522093da45c634c181ceff68c51f0362e982d69de2a5d9d41fb8067d4ef8f4707d4dd9168dd394e9b57a3de5adf2b00ebdd117e47db13f","projectIdV2":"unsuitable-belligerent-brother_67663e0418ebc48269522093da45c634","projectName":"998ea67331e2c435208a6fe3b8f73565ab0574993e65961cf391a343c163ecae088a0c746a95207189e4fd434771ab9e91dc9c647f11dce3b25dd18c4d041505","projectNameV2":"ruddy-curved-fortune_998ea67331e2c435208a6fe3b8f73565","enterpriseDomain":"2769c2abae62151b2ebb8658628f7c5f5d0dc0c29fdefdd19a23dec9cb0a7b96d74d82512d1f6906bef65b24a29d84685dfd2fd66964a56fbdaff39fabd69206","enterpriseDomainV2":"ratty-willing-science_2769c2abae62151b2ebb8658628f7c5f","isLoggedIn":false,"ciName":null,"system":{"platform":"darwin","platformVersion":"22.6.0","gardenVersion":"0.13.18"},"isCI":false,"sessionId":"3de417b0-8583-47da-9f7d-21c2aa781dc5","parentSessionId":"3de417b0-8583-47da-9f7d-21c2aa781dc5","projectMetadata":{"modulesCount":42,"moduleTypes":["container"],"tasksCount":46,"servicesCount":49,"testsCount":36,"actionsCount":0,"buildActionCount":0,"testActionCount":0,"deployActionCount":0,"runActionCount":0},"firstRunAt":"2023-09-30T00:10:04.238Z","latestRunAt":"2023-10-20T22:23:00.459Z","isRecurringUser":true,"errors":["runtime"],"lastError":{"error":{"errorType":"runtime","stackTrace":{"functionName":"handleProcessResults","relativeFileName":"commands/base.js","lineNumber":730}},"wrapped":{"errorType":"graph","taskType":"deploy","stackTrace":{"functionName":"GetAddrInfoReqWrap.onlookup [as oncomplete]","lineNumber":84}},"leaf":{"errorType":"crash","stackTrace":{"functionName":"GetAddrInfoReqWrap.onlookup [as oncomplete]","lineNumber":84}}},"result":"failure","name":"deploy","durationMsec":16347}}
1 deploy action(s) failed!
[silly] Error: 1 deploy action(s) failed!
    at handleProcessResults (/snapshot/pkg/core/build/src/commands/base.js:730:23)
    at DeployCommand.action (/snapshot/pkg/core/build/src/commands/deploy.js:291:48)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /snapshot/pkg/core/build/src/commands/base.js:214:30
    at async /snapshot/pkg/core/build/src/util/open-telemetry/spans.js:46:20
    at async /snapshot/pkg/core/build/src/cli/cli.js:258:26
    at async /snapshot/pkg/core/build/src/cli/cli.js:433:33
    at async /snapshot/pkg/core/build/src/util/open-telemetry/spans.js:46:20
    at async GardenCli.run (/snapshot/pkg/core/build/src/cli/cli.js:431:32)
    at async /snapshot/pkg/cli/build/src/cli.js:100:29
    at async /snapshot/pkg/core/build/src/util/open-telemetry/spans.js:46:20
    at async Object.runCli (/snapshot/pkg/cli/build/src/cli.js:95:18)

Error type: runtime

Wrapped errors:
⮑ Error: getaddrinfo ENOTFOUND 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
       at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:84:26)
       at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17)

   Error type: graph

   Wrapped errors:
   ⮑ Error: getaddrinfo ENOTFOUND 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
          at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:84:26)
          at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17)

      Error type: graph

      Wrapped errors:
      ⮑ Error: getaddrinfo ENOTFOUND 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
             at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:84:26)
             at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17)

         Error type: graph

         Wrapped errors:
         ⮑ Error: getaddrinfo ENOTFOUND 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
                at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:84:26)
                at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17)

            Error type: graph

            Wrapped errors:
            ⮑ Error: getaddrinfo ENOTFOUND 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
                   at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:84:26)
                   at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17)

               Error type: crash

What did you do?

I'm getting this consistent error when trying to deploy using the latest versions of garden CLI (0.13.18). This issue does not appear when I use an older version (0.13.13).

Your environment

• OS: mac OS Ventura 13.6
• How I'm running Kubernetes: EKS
• Garden version: 0.13.18

Frequency

This crash appears to be happening every time I try to deploy.

Workaround

Downrev garden version to 0.13.13.

Additional context

Other developers on my team are also experiencing this issue. I tried the following command to see if there was some kind of networking issue, the output seems legit.

dig 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com

; <<>> DiG 9.10.6 <<>> 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1743
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com. IN A

;; ANSWER SECTION:
06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com. 31 IN	A 54.241.117.126
06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com. 31 IN	A 54.177.204.177

;; Query time: 23 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Oct 20 15:22:13 PDT 2023
;; MSG SIZE  rcvd: 125

nslookup 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
Server:		127.0.0.1
Address:	127.0.0.1#53

Non-authoritative answer:
Name:	06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
Address: 54.241.117.126
Name:	06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
Address: 54.177.204.177

[jghowe]

After doing a bit more triage, I believe the issue may be a client/server version mismatch.

arden tools kubernetes.kubectl -- version 
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.16-eks-2d98532", GitCommit:"af930c12e26ef9d1e8fac7e3532ff4bcc1b2b509", GitTreeState:"clean", BuildDate:"2023-07-28T16:52:47Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}

The EKS server is using version 1.24 and the client is using 1.23. I updated the project file and added kubectlPath to point to a local install of client version 1.25 and I did not get this error anymore.

[jghowe]

It doesn't appear as though the kubectl version was the culprit as we are still seeing this error. Here is another error which appears to show a little more context, appears to be failing during readNamespacedSecret.

✖ deploy.onboarding-service → Failed (took 0 sec)
✖ deploy.onboarding-service → Failed processing Deploy type=container name=onboarding-service (from module onboarding-service) (took 140.13 sec). This is what happened:

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Encountered an unexpected Garden error. This is likely a bug 🍂

You can help by reporting this on GitHub: https://github.com/garden-io/garden/issues/new?labels=bug,crash&template=CRASH.md&title=Crash%3A%20toKubernetesError%20encountered%20an%20unknown%20error%20unexpectedly%20during%20Kub...

Please attach the following information to the bug report after making sure that the error message does not contain sensitive information:

Failed processing Deploy type=container name=onboarding-service (from module onboarding-service) (took 140.13 sec). This is what happened:
Error: getaddrinfo ENOTFOUND 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:84:26)
    at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17)
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

toKubernetesError encountered an unknown error unexpectedly during Kubernetes API: readNamespacedSecret: toKubernetesError encountered an unknown error unexpectedly during readNamespacedSecret: getaddrinfo ENOTFOUND 06a5b456cad986cb2760fe5296f7995e.gr7.us-west-1.eks.amazonaws.com
1 deploy action(s) failed!

[soren121]

Maybe a duplicate of #5217?

[jghowe]

Yes, I am starting to think that this is indeed a DNS issue. I added an entry to my /etc/hosts file and haven't run into the problem again (hard to tell though as the problem appears intermittent). Still, seems strange that the problem was not present on earlier versions of the garden CLI.

[stefreak]

Thank you for investigating this @jghowe 👍

You are right @soren121, this is a duplicate of #5217

We now have a pull request with a possible fix in #5326

I'm closing this in favor of #5217