Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stats are reset when re-balancing users across slave nodes #1168

Closed
heyman opened this issue Nov 20, 2019 · 10 comments
Closed

Stats are reset when re-balancing users across slave nodes #1168

heyman opened this issue Nov 20, 2019 · 10 comments
Labels

Comments

@heyman
Copy link
Member

@heyman heyman commented Nov 20, 2019

When running locust with --reset-stats in distributed mode, the stats are reset every time a new slave connects (when the re-balancing of the simulated users occur). The stats should only be reset when the initial hatching is complete, and not when re-balancing users.

@max-rocket-internet
Copy link
Contributor

@max-rocket-internet max-rocket-internet commented Nov 21, 2019

I am not using --reset-stats option, I'm using the default docker image locustio/locust:0.13.1 and locust is started like this:

# Slave
/usr/local/bin/python /usr/local/bin/locust --slave --locustfile=/mnt/locust/main.py --host=https://xxxxx.com --master-host=locust-xxx

# Master
/usr/local/bin/python /usr/local/bin/locust --master --locustfile=/mnt/locust/main.py --host=https://xxxxx.com

But I still see this issue 🙂

@heyman
Copy link
Member Author

@heyman heyman commented Nov 21, 2019

Hm, that's strange. I'm not able to reproduce the issue without --reset-stats. I guess I have to do some more digging (I'll try and see if I can reproduce it using the Docker image).

@max-rocket-internet
Copy link
Contributor

@max-rocket-internet max-rocket-internet commented Nov 22, 2019

OK let me know if you want me to debug or test something.

@matti
Copy link

@matti matti commented Nov 27, 2019

I'm also having the same issue as @max-rocket-internet
Locust

@max-rocket-internet
Copy link
Contributor

@max-rocket-internet max-rocket-internet commented Jan 2, 2020

Still seeing this in version 0.13.5:

Screen Shot 2020-01-02 at 16 34 29

@carstendev
Copy link

@carstendev carstendev commented Jan 23, 2020

I was able to replicate this or a (related?) issue, by doing the following:

  • Start a test without --reset-stats with 500 max users and a hatch rate of 1
  • Lower the number of max users after one minute (still during the hatching phase) to 450

Locust then proceeds to calculate users_to_kill = 500 - 450 and kills 50 users immediately.

Issue here is that as the hatching is still ongoing, and 50 of the at this point only 60 spawned users are killed. I would expect locust to kill users based on the actual spawned users, not just the max setting.

If this is in fact not the same issue, I can also open another issue for this.

@heyman
Copy link
Member Author

@heyman heyman commented Jan 23, 2020

I've managed to reproduce the issue using the steps described by @carstendev. It's actually a different issue than the original post, but with similar symptoms.

I'm working on a fix now.

heyman added a commit that referenced this issue Jan 23, 2020
…sers are still hatching. #1168

Removed LocustRunner.num_clients in favour of using LocustRunner.user_count which should be less prone to errors (since it’s the actual number of locust greenlets). Introduced MasterLocustRunner.target_user_count which is needed so that we can rebalance locust users across the connected slaves while users are till hatching.
@heyman
Copy link
Member Author

@heyman heyman commented Jan 23, 2020

I've now pushed changes (576f7ea) that should hopefully fix this. Would be great if someone could try out the latest master to confirm that the issue has been fixed.

@carstendev
Copy link

@carstendev carstendev commented Jan 24, 2020

I've tried to reproduce the issue with the latest master, but it seems to be fixed now 👍.

@heyman heyman closed this as completed Jan 24, 2020
@max-rocket-internet
Copy link
Contributor

@max-rocket-internet max-rocket-internet commented Jan 27, 2020

Awesome, thanks @heyman!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants