"helm test" pod with injected sidecars ends up in NotReady state

[ahlae]

When running helm test --logs with injected sidecars the pod which runs the tests ends up in a NotReady state. All containers inside the pod are running without issues, have logs without errors and the tests finish and are logged inside kubernetes but the helm test command does not return and does not report the logs.

The app is intended to run on the Hashicorp Consul Connect App Mesh (which injects an envoy sidecar proxy) and Hashicorp Vault Secrets Inject Agent (which injects an secrets agent sidecar that puts secrets into the pod filesystem).

This issue also happens when running with only one sidecar (or envoy or vault inject agent). When running without any sidecars helm test runs as expected, reports the logs and finishes. That's why I presume this might be a more general problem tied to how helm runs tests. But perhaps it might be also a consequence of how Consul/Vault handles sidecar injection. That’s why I’m posting this here and also on the Hashicorp discuss page:

https://discuss.hashicorp.com/t/helm-tests-pod-with-injected-consul-vault-sidecars-ends-up-in-notready-state/27741

helm is running inside a GitLab Runner.

Output of helm version:

version.BuildInfo{Version:"v3.5.4", GitCommit:"1b5edb69df3d3a08df77c9902dc17af864ff05d1", GitTreeState:"clean", GoVersion:"go1.15.11"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.12", GitCommit:"e2a822d9f3c2fdb5c9bfbe64313cf9f657f0a725", GitTreeState:"clean", BuildDate:"2020-05-06T05:17:59Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.8-eks-96780e", GitCommit:"96780e1b30acbf0a52c38b6030d7853e575bcdf3", GitTreeState:"clean", BuildDate:"2021-03-10T21:32:29Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.):

EKS

[bacongobbler]

Hi @ahlae.

There's a lot of technologies at play here (Hashicorp Consul Connect App Mesh, Hashicorp Vault Secrets Inject Agent, GitLab Runner), so let's start with a very simple base case and then go from there.

Can you share an example that other community members can use to test and reproduce a similar issue? That way we can determine the root cause and a possible fix. I would try to re-create the environment using a simple test hook with a basic "helloworld" sidecar container, then try to deduce the issue from there.

I'm afraid your current setup may be very difficult to recreate for most users so a very simple how-to guide with vanilla Kubernetes and a simple test chart may be the best course of action here.

Thanks!

[mattfarina]

Can you share what the Helm test command output? The code that handles reading and outputting the logs is here. If the client can't get the logs it should say that. Is it reporting that? The output would help us.

[ahlae]

Hi @bacongobbler,

thank you for your time! :)

I admit that's not an easy setup to debug. I posted it hoping that perhaps someone already knows where to look at specifically. I will try to recreate a simpler setup with minikube and a few commands and try to find out if the problem persists. Then I'll update the issue here.

[ahlae]

@mattfarina The command doesn't return and hangs there without output until the GitLab Runner times out after about 10 minutes.

[mattfarina]

It is likely getting hung up on one of these two lines that are using the k8s packages to interact with the API.

[ahlae]

The problem also appears when not using the --logs flag. This is the state in which the whole helm deployment hangs even if all the pods are running as described above:

NAME-------------------------------------------------- READY STATUS RESTARTS AGE
tests----------------------------------------------------- 1/2 NotReady 0 46s
tests-celery-7568d86b9-vcgkb------------------- 4/4 Running 0 2m47s
tests-celery-beat-666679b6ff-tgtd9-------------- 4/4 Running 0 2m47s
tests-postgresql-postgresql-0--------------------- 3/3 Running 0 3m28s
tests-redis-master-0--------------------------------- 3/3 Running 0 3m53s
tests-vault-0-------------------------------------------- 3/3 Running 0 5m27s
tests-vault-agent-injector-74c644587f-dpppl-- 1/1 Running 0 5m27s

Is GetPodLogs still executed when not using the --logs flag?

[bacongobbler]

Is GetPodLogs still executed when not using the --logs flag?

No:

helm/cmd/helm/release_testing.go

Lines 79 to 82 in a499b4b

	if outputLogs {
	// Print a newline to stdout to separate the output
	fmt.Fprintln(out)
	if err := client.GetPodLogs(out, rel); err != nil {

[ahlae]

If there are multiple containers in the pod how would you pass "-c <container_name>" to helm test --logs? That's what would be necessary when using kubectl logs.

[bacongobbler]

That functionality is currently unavailable. #9892 and #6902 requested similar functionality. PRs to introduce that functionality would be most welcome.

[github-actions]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.