Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(kuma-cp): don't leak goroutine on every tick in SimpleWatchdog #7348

Merged
merged 6 commits into from Jul 26, 2023

Conversation

lukidzi
Copy link
Contributor

@lukidzi lukidzi commented Jul 25, 2023

Checklist prior to review

While debugging some performance issue I've noticed that Kuma creates thousands of goroutines that might grow indefinitely. Whenever clients connect to KDS create an instance of SimpleWatchdog which is responsible for configuration generation. It's triggered every second and generates a snapshot which later puts into the cache. Lately, while implementing better context cancelation we introduced a change in a SimpleWatchdog which creates a new goroutine responsible for listing stop events and canceling the context. Goroutine is spawned in a loop and every second there was a new goroutine created which was listening events. In the case of KDS this code was running until the zone disconnects(disconnect cleaned up all the goroutines), and because connections are long-lasting it might create over 200k+ goroutines. It seems to be relevant to all the code using SimpleWatchdog. I am not sure if that fix handling all the cases correctly but while testing I've noticed that there is no more leak.

Before 1 zone connected for over a 1min:
image

After fix:
image

  • Link to relevant issue as well as docs and UI issues --
  • This will not break child repos: it doesn't hardcode values (.e.g "kumahq" as a image registry) and it will work on Windows, system specific functions like syscall.Mkfifo have equivalent implementation on the other OS --
  • Tests (Unit test, E2E tests, manual test on universal and k8s) --
  • Do you need to update UPGRADE.md? --
  • Does it need to be backported according to the backporting policy? --
  • Do you need to explicitly set a > Changelog: entry here or add a ci/ label to run fewer/more tests?

Signed-off-by: Lukasz Dziedziak <lukidzi@gmail.com>
@lukidzi lukidzi requested a review from a team as a code owner July 25, 2023 21:50
@lukidzi lukidzi requested review from jakubdyszkiewicz, lobkovilya and michaelbeaumont and removed request for a team July 25, 2023 21:50
Signed-off-by: Lukasz Dziedziak <lukidzi@gmail.com>
This reverts commit 8f57626.

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>
This reverts commit 53b4777.

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>
…utine on cancel

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>
@michaelbeaumont michaelbeaumont changed the title fix(kuma-cp): don't spawn new goroutine on every tick in SimpleWatchdog fix(kuma-cp): don't leak goroutine on every tick in SimpleWatchdog Jul 26, 2023
@lahabana lahabana added the ci/run-full-matrix PR: Runs all possible e2e test combination (expensive use carefully) label Jul 26, 2023
@michaelbeaumont michaelbeaumont merged commit 1f97d44 into kumahq:master Jul 26, 2023
5 checks passed
@github-actions
Copy link
Contributor

github-actions bot commented Jul 26, 2023

backporting to release-2.3 with action

backporting to release-2.0 with action
backporting to release-2.2 with action

kumahq bot pushed a commit that referenced this pull request Jul 26, 2023
…7348)

* fix(kuma-cp): ensure we call cancel on watchdog tick scope, exit goroutine on cancel

Signed-off-by: Lukasz Dziedziak <lukidzi@gmail.com>
Signed-off-by: Mike Beaumont <mjboamail@gmail.com>
Co-authored-by: Mike Beaumont <mjboamail@gmail.com>
kumahq bot pushed a commit that referenced this pull request Jul 26, 2023
…7348)

* fix(kuma-cp): ensure we call cancel on watchdog tick scope, exit goroutine on cancel

Signed-off-by: Lukasz Dziedziak <lukidzi@gmail.com>
Signed-off-by: Mike Beaumont <mjboamail@gmail.com>
Co-authored-by: Mike Beaumont <mjboamail@gmail.com>
kumahq bot pushed a commit that referenced this pull request Jul 26, 2023
…7348)

* fix(kuma-cp): ensure we call cancel on watchdog tick scope, exit goroutine on cancel

Signed-off-by: Lukasz Dziedziak <lukidzi@gmail.com>
Signed-off-by: Mike Beaumont <mjboamail@gmail.com>
Co-authored-by: Mike Beaumont <mjboamail@gmail.com>
kumahq bot pushed a commit that referenced this pull request Jul 26, 2023
…7348)

* fix(kuma-cp): ensure we call cancel on watchdog tick scope, exit goroutine on cancel

Signed-off-by: Lukasz Dziedziak <lukidzi@gmail.com>
Signed-off-by: Mike Beaumont <mjboamail@gmail.com>
Co-authored-by: Mike Beaumont <mjboamail@gmail.com>
kumahq bot pushed a commit that referenced this pull request Jul 26, 2023
…7348)

* fix(kuma-cp): ensure we call cancel on watchdog tick scope, exit goroutine on cancel

Signed-off-by: Lukasz Dziedziak <lukidzi@gmail.com>
Signed-off-by: Mike Beaumont <mjboamail@gmail.com>
Co-authored-by: Mike Beaumont <mjboamail@gmail.com>
michaelbeaumont added a commit that referenced this pull request Jul 26, 2023
…ackport of #7348) (#7355)

fix(kuma-cp): don't leak goroutine on every tick in SimpleWatchdog (#7348)

* fix(kuma-cp): ensure we call cancel on watchdog tick scope, exit goroutine on cancel

Signed-off-by: Lukasz Dziedziak <lukidzi@gmail.com>
Signed-off-by: Mike Beaumont <mjboamail@gmail.com>
Co-authored-by: Łukasz Dziedziak <lukidzi@gmail.com>
Co-authored-by: Mike Beaumont <mjboamail@gmail.com>
michaelbeaumont added a commit that referenced this pull request Jul 26, 2023
…ackport of #7348) (#7352)

fix(kuma-cp): don't leak goroutine on every tick in SimpleWatchdog (#7348)

* fix(kuma-cp): ensure we call cancel on watchdog tick scope, exit goroutine on cancel

Signed-off-by: Lukasz Dziedziak <lukidzi@gmail.com>
Signed-off-by: Mike Beaumont <mjboamail@gmail.com>
Co-authored-by: Łukasz Dziedziak <lukidzi@gmail.com>
Co-authored-by: Mike Beaumont <mjboamail@gmail.com>
michaelbeaumont added a commit that referenced this pull request Jul 26, 2023
…ackport of #7348) (#7351)

fix(kuma-cp): don't leak goroutine on every tick in SimpleWatchdog (#7348)

* fix(kuma-cp): ensure we call cancel on watchdog tick scope, exit goroutine on cancel

Signed-off-by: Lukasz Dziedziak <lukidzi@gmail.com>
Signed-off-by: Mike Beaumont <mjboamail@gmail.com>
Co-authored-by: Łukasz Dziedziak <lukidzi@gmail.com>
Co-authored-by: Mike Beaumont <mjboamail@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport ci/run-full-matrix PR: Runs all possible e2e test combination (expensive use carefully)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants