Variable-looping-vus/stages is a start/stopping instances a little flaky #1473
Comments
mstoykov
changed the title
mstoykov
added a commit
that referenced
this issue
Jun 9, 2020
Also fix VLV not respecting gracefulStop since 270fd91, add test for it and try to make some other VLV tests more stable. The original code (pre 270fd91) did go through all the raw steps (ramp up/down) and the graceful stop ones concurrently and when the raw steps ended it will return from Run, but will start a goroutine to go through the remainign graceful stop ones. The most important of which ( at least in my current understanding) being the last one which is the actuall gracefulStop end which will make all VUs stop. The reason why this actually worked is that it also waiting on all activeVUs to have ended being active, being returned in the new terminology, before it actually cancels the context for all of them. So if the VUs manage to end iterations in this time the executor will end earlier if not the gracefuStop will make them. 270fd91 broke the returning of VUs which lead to the executor never returning and moving the cancel "before" the waiting for activeVUs fixed that at the cost of gracefulStop not being taken into account, but with no tests, nobody noticed. Here I basically revert that especially because vuHandle now returns VUs when stopped. fixes #1473
mstoykov
added a commit
that referenced
this issue
Jun 9, 2020
Also fix VLV not respecting gracefulStop since 270fd91, add test for it and try to make some other VLV tests more stable. The original code (pre 270fd91) did go through all the raw steps (ramp up/down) and the graceful stop ones concurrently and when the raw steps ended it will return from Run, but will start a goroutine to go through the remainign graceful stop ones. The most important of which ( at least in my current understanding) being the last one which is the actuall gracefulStop end which will make all VUs stop. The reason why this actually worked is that it also waiting on all activeVUs to have ended being active, being returned in the new terminology, before it actually cancels the context for all of them. So if the VUs manage to end iterations in this time the executor will end earlier if not the gracefuStop will make them. 270fd91 broke the returning of VUs which lead to the executor never returning and moving the cancel "before" the waiting for activeVUs fixed that at the cost of gracefulStop not being taken into account, but with no tests, nobody noticed. Here I basically revert that especially because vuHandle now returns VUs when stopped. fixes #1473
mstoykov
added a commit
that referenced
this issue
Jun 10, 2020
Also fix VLV not respecting gracefulStop since 270fd91, add test for it and try to make some other VLV tests more stable. The original code (pre 270fd91) did go through all the raw steps (ramp up/down) and the graceful stop ones concurrently and when the raw steps ended it will return from Run, but will start a goroutine to go through the remainign graceful stop ones. The most important of which ( at least in my current understanding) being the last one which is the actuall gracefulStop end which will make all VUs stop. The reason why this actually worked is that it also waiting on all activeVUs to have ended being active, being returned in the new terminology, before it actually cancels the context for all of them. So if the VUs manage to end iterations in this time the executor will end earlier if not the gracefuStop will make them. 270fd91 broke the returning of VUs which lead to the executor never returning and moving the cancel "before" the waiting for activeVUs fixed that at the cost of gracefulStop not being taken into account, but with no tests, nobody noticed. Here I basically revert that especially because vuHandle now returns VUs when stopped. fixes #1473 Co-authored-by: na-- <n@andreev.sh>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It seems like sometimes variable looping vus will gracefully stop a VU while ramping up and then just start it again. After some additional logs, it is easy to spot
As is seen
rawStepgoes a little faster sometimes and gets 2-3 steps ahead ofgracefulLimitand when we get agracefulLimitthat is basically for some steps ago and we stop a VU or two... gracefully :D.This (at least for me) only happens if the machine is under load which is not all that hard especially when ramping up so ...
The code in question reads from two different channels that get events every time something needs to happen.
https://github.com/loadimpact/k6/blob/83d0fcf2ecf44db27275cdf7ea887c05edf9bd06/lib/executor/variable_looping_vus.go#L638-L673
I do think it will be much simpler and easier if we zip (combine) the two giving
gracefulLimitthe preference and just go through the new slice ...This also will definitely be more performant, not that 2 goroutines and 2 channels are too much ... but we can skip them :D
The text was updated successfully, but these errors were encountered: