Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Potential resource leak #759

Closed
howardjohn opened this issue Aug 7, 2019 · 8 comments

Comments

@howardjohn
Copy link

commented Aug 7, 2019

What happened:
On July 10th, we started using kind in our prow cluster for Istio (see istio/test-infra#1455). We have about 6 tests or so that use kind, and maybe run 40 tests a day, so I'd estimate we are creating roughly 250 kind clusters a day across 60 nodes.

Ever since July 10th, we have run into problems on our cluster. Almost immediately, we ran out of inotify watches, which we fixed by just increasing the limit -- see #717.

Since then, the CPU of our nodes has slowly increased, like a memory leak but CPU. This can be attributed to the Kubelet process, which a profile indicates most of the time spent by cAdvisor.

Here is a graph of the minimum CPU of our nodes for a 4 hour interval. During a 4 hour time, it is almost certain we will have no test pods scheduled on the node, so this essentially shows the base overhead CPU of the nodes.

2019-07-08_08-23-40

The big drops are times when we got rid of nodes. You can see the problems seem to start right around when we started using kind (note - it is possible it is a coincidence. My only evidence it is related to kind is the timing).

Within two weeks we see some nodes using 90% of CPU just idling.

What you expected to happen:
kind does not cause cluster-wide degradation.

How to reproduce it (as minimally and precisely as possible):
I am not sure, but we consistently run into this problem so I can run some diagnostics on the node.

Environment:

  • kind version: (use kind version): v0.3, v0.4
  • Kubernetes version: (use kubectl version): GKE versions 1.11, 1.12, 1.13 (tried upgrading twice to attempt to resolve this)
  • OS (e.g. from /etc/os-release): COS

@howardjohn howardjohn added the kind/bug label Aug 7, 2019

@aojea

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2019

this issue remembers me this other one #421

@howardjohn

This comment has been minimized.

Copy link
Author

commented Aug 7, 2019

I did run into that while investigating, but it seems the conclusion was

I can confirm that repeated cluster creation / deletion does NOT leak cgroups. The output of lscgroup | grep -c memory is the same after each creation / deletion cycle.

By the way, if it is relevant, we never do kind cluster delete, we just do kind create cluster then run some test and exit.

@aojea

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2019

By the way, if it is relevant, we never do kind cluster delete, we just do kind create cluster then run some test and exit.

but ... then the old cluster containers keep running forever ... is it possible to add the kind cluster delete to your workflow once the tests finish and check if this solves the problem?

@howardjohn

This comment has been minimized.

Copy link
Author

commented Aug 7, 2019

But we run it in a pod, once the pod is removed shouldn't everything be cleaned up? Or maybe because we have

        volumeMounts:
        - mountPath: /lib/modules
          name: modules
          readOnly: true
        - mountPath: /sys/fs/cgroup
          name: cgroup
      volumes:
      - hostPath:
          path: /lib/modules
          type: Directory
        name: modules
      - hostPath:
          path: /sys/fs/cgroup
          type: Directory
        name: cgroup

in our pod spec, so it never gets properly cleaned up.

I'll try adding the delete cluster to the end.

Does Kubernetes prow do this in their tests using kind? I am worried if the test crashes part way through we won't properly clean up

@aojea

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2019

Does Kubernetes prow do this in their tests using kind? I am worried if the test crashes part way through we won't properly clean up

@BenTheElder is the authority on this, but the tests that are running in the CI execute the hack/ci/e2e.sh, and it does have a cleanup function that delete the cluster on EXIT

kind/hack/ci/e2e.sh

Lines 26 to 41 in 991e45e

# our exit handler (trap)
cleanup() {
# always attempt to dump logs
kind "export" logs "${ARTIFACTS}/logs" || true
# KIND_IS_UP is true once we: kind create
if [[ "${KIND_IS_UP:-}" = true ]]; then
kind delete cluster || true
fi
# clean up e2e.test symlink
rm -f _output/bin/e2e.test
# remove our tempdir
# NOTE: this needs to be last, or it will prevent kind delete
if [[ -n "${TMP_DIR:-}" ]]; then
rm -rf "${TMP_DIR}"
fi
}

@howardjohn

This comment has been minimized.

Copy link
Author

commented Aug 7, 2019

Thank you! we will try that out

howardjohn added a commit to howardjohn/istio that referenced this issue Aug 7, 2019

Delete kind cluster after test completion
Because we mount host paths in the pods, resources will not actually be
fully freed once a test is complete. This causes a resource leak that
eventually leads to a complete degradation of the entire node.

See kubernetes-sigs/kind#759 for details.

istio-testing added a commit to istio/istio that referenced this issue Aug 7, 2019

Delete kind cluster after test completion (#16105)
Because we mount host paths in the pods, resources will not actually be
fully freed once a test is complete. This causes a resource leak that
eventually leads to a complete degradation of the entire node.

See kubernetes-sigs/kind#759 for details.
@howardjohn

This comment has been minimized.

Copy link
Author

commented Aug 8, 2019

Well, it seems resolved - hard to be 100% sure since its only been 8 hrs, but seems good. I feel pretty dumb trying to figure this out for a month or so when it was such a simple fix -- thanks for the help!!

@howardjohn howardjohn closed this Aug 8, 2019

@BenTheElder

This comment has been minimized.

Copy link
Member

commented Aug 8, 2019

The docker in docker runner / wrapper script we use in test-infra / prow.k8s.io also terminates all containers in an exit handler, amongst other things, redundantly to the cluster deletion we do in the kind specific scripts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.