Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Locust freezes after a while (python 3.7) #843

Closed
ahmetb opened this issue Jul 15, 2018 · 3 comments
Closed

Locust freezes after a while (python 3.7) #843

ahmetb opened this issue Jul 15, 2018 · 3 comments

Comments

@ahmetb
Copy link

ahmetb commented Jul 15, 2018

Hi, I've been using locust to generate some load to for a demo application.

Description of issue

For some reason, after sending a few requests, locust decides to stop doing that.

UPDATE: Switching to python3.6.6 from 3.7.0 does not exhibit this issue.

I was keeping locust version and all the same and my HTTP endpoints still work just fine when I call them manually, locust started to freeze.

I run locust with parameters:

locust --host="http://${FRONTEND_ADDR}" --no-web -c 10 -r 10 

and monitor its output via standard output.

Expected behavior

Locust should continue to run forever since -n is not specified.

Actual behavior

Locust freezes and logs (stdout) stop showing any progress.

  • my HTTP endpoint is healthy and up, I manually verify this

  • This used to work, but in the past few days locust started to freeze and stop sending requests

  • I sometimes see "segmentation fault" in the stderr and my container restarts:

    ./loadgen.sh: line 10:     6 Segmentation fault      (core dumped) locust --host="http://${FRONTEND_ADDR}" --no-web -c "${USERS:-10}"
    
  • Logs from locust container:

[2018-07-15 18:17:56,792] loadgenerator-564c4fdd64-w54vb/INFO/locust.main: Starting Locust 0.8.1
[2018-07-15 18:17:56,793] loadgenerator-564c4fdd64-w54vb/INFO/locust.runners: Hatching and swarming 10 clients at the rate 10 clients/s...
 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s
--------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------
 Total                                                              0     0(0.00%)                                       0.00

[2018-07-15 18:17:57,803] loadgenerator-564c4fdd64-w54vb/INFO/locust.runners: All locusts hatched: WebsiteUser: 10
[2018-07-15 18:17:57,804] loadgenerator-564c4fdd64-w54vb/INFO/locust.runners: Resetting stats

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                              0     0(0.00%)       0       0       0  |       0    0.00
 GET /cart                                                          0     0(0.00%)       0       0       0  |       0    0.00
 POST /cart                                                         0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/0PUK6V6EV0                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/2ZYFJ3GM2N                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/66VCHSJNUP                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/6E92ZMYYFZ                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/OLJCESPC7Z                                            0     0(0.00%)       0       0       0  |       0    0.00
 POST /setCurrency                                                  0   1(100.00%)       0       0       0  |       0    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Total                                                              0     1(0.00%)                                       0.00

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                              0     0(0.00%)       0       0       0  |       0    0.00
 GET /cart                                                          0     0(0.00%)       0       0       0  |       0    0.00
 POST /cart                                                         0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/0PUK6V6EV0                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/2ZYFJ3GM2N                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/66VCHSJNUP                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/6E92ZMYYFZ                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/L9ECAV7KIM                                            0   1(100.00%)       0       0       0  |       0    0.00
 GET /product/OLJCESPC7Z                                            0     0(0.00%)       0       0       0  |       0    0.00
 POST /setCurrency                                                  0   1(100.00%)       0       0       0  |       0    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Total                                                              0     2(0.00%)                                       0.00

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                              0     0(0.00%)       0       0       0  |       0    0.00
 GET /cart                                                          0     0(0.00%)       0       0       0  |       0    0.00
 POST /cart                                                         0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/0PUK6V6EV0                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/2ZYFJ3GM2N                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/66VCHSJNUP                                            0   1(100.00%)       0       0       0  |       0    0.00
 GET /product/6E92ZMYYFZ                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/L9ECAV7KIM                                            0   1(100.00%)       0       0       0  |       0    0.00
 GET /product/OLJCESPC7Z                                            0     0(0.00%)       0       0       0  |       0    0.00
 POST /setCurrency                                                  2    1(33.33%)     168      97     239  |      97    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Total                                                              2   3(150.00%)                                       0.00

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                              0     0(0.00%)       0       0       0  |       0    0.00
 GET /cart                                                          1     0(0.00%)     488     488     488  |     490    0.00
 POST /cart                                                         1     0(0.00%)     167     167     167  |     170    0.00
 GET /product/0PUK6V6EV0                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/1YMWWN1N4O                                            1     0(0.00%)     299     299     299  |     300    0.00
 GET /product/2ZYFJ3GM2N                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/66VCHSJNUP                                            0   1(100.00%)       0       0       0  |       0    0.00
 GET /product/6E92ZMYYFZ                                            0     0(0.00%)       0       0       0  |       0    0.00
 GET /product/L9ECAV7KIM                                            0   1(100.00%)       0       0       0  |       0    0.00
 GET /product/OLJCESPC7Z                                            0     0(0.00%)       0       0       0  |       0    0.00
 POST /setCurrency                                                  2    1(33.33%)     168      97     239  |      97    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Total                                                              5    3(60.00%)                                       0.00

[... some time passes ..]

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                              4     0(0.00%)     124      75     224  |      78    0.00
 GET /cart                                                         10     0(0.00%)     209      68     488  |     210    0.40
 POST /cart                                                        10     0(0.00%)     126      48     315  |     110    0.50
 POST /cart/checkout                                                4     0(0.00%)     326     230     383  |     330    0.10
 GET /product/0PUK6V6EV0                                            6     0(0.00%)     136      69     187  |     130    0.30
 GET /product/1YMWWN1N4O                                            8     0(0.00%)     129      33     299  |     110    0.30
 GET /product/2ZYFJ3GM2N                                            6     0(0.00%)     204     118     341  |     160    0.10
 GET /product/66VCHSJNUP                                            5    1(16.67%)     190     112     282  |     220    0.20
 GET /product/6E92ZMYYFZ                                            9     0(0.00%)     173      64     326  |     160    0.10
 GET /product/9SIQT8TOJO                                            1     0(0.00%)      78      78      78  |      78    0.00
 GET /product/L9ECAV7KIM                                            2    1(33.33%)     215     160     270  |     160    0.00
 GET /product/LS4PSXUNUM                                            6     0(0.00%)     143      34     244  |     110    0.30
 GET /product/OLJCESPC7Z                                           14     0(0.00%)     197      38     397  |     170    0.10
 POST /setCurrency                                                 10     1(9.09%)     130      38     239  |     120    0.20
--------------------------------------------------------------------------------------------------------------------------------------------
 Total                                                             95     3(3.16%)                                       2.60

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                              4     0(0.00%)     124      75     224  |      78    0.00
 GET /cart                                                         11     0(0.00%)     197      68     488  |     210    0.40
 POST /cart                                                        11     0(0.00%)     119      48     315  |     110    0.30
 POST /cart/checkout                                                4     0(0.00%)     326     230     383  |     330    0.10
 GET /product/0PUK6V6EV0                                            6     0(0.00%)     136      69     187  |     130    0.20
 GET /product/1YMWWN1N4O                                            8     0(0.00%)     129      33     299  |     110    0.20
 GET /product/2ZYFJ3GM2N                                            6     0(0.00%)     204     118     341  |     160    0.10
 GET /product/66VCHSJNUP                                            5    1(16.67%)     190     112     282  |     220    0.10
 GET /product/6E92ZMYYFZ                                            9     0(0.00%)     173      64     326  |     160    0.20
 GET /product/9SIQT8TOJO                                            2     0(0.00%)     145      78     212  |      78    0.00
 GET /product/L9ECAV7KIM                                            2    1(33.33%)     215     160     270  |     160    0.10
 GET /product/LS4PSXUNUM                                            7     0(0.00%)     152      34     244  |     200    0.30
 GET /product/OLJCESPC7Z                                           14     0(0.00%)     197      38     397  |     170    0.20
 POST /setCurrency                                                 10     1(9.09%)     130      38     239  |     120    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Total                                                             99     3(3.03%)                                       2.20

[freezes here, no more new logs arrive]

Environment settings (for bug reports)

  • OS: Linux (inside a python:3 docker container)
  • Python version: 3.7.0
  • Locust version: 0.8.1

Steps to reproduce (for bug reports)

I'm not sure if I can provide any good repro of this. :( My HTTP endpoint works just fine manually, but Locust stops sending requests.

EDIT: uploaded my locustfile.py here: https://gist.github.com/ahmetb/2645deefff78216933779eeb4043a5e2

@ahmetb
Copy link
Author

ahmetb commented Jul 15, 2018

I reduced -n 10 to -n 5, now instead of freezing at ~100 requests, it made it to 343 requests, only to segfault later:

 Total                                                            343     4(1.17%)                                       0.90

./loadgen.sh: line 10:     6 Segmentation fault      (core dumped) locust --host="http://${FRONTEND_ADDR}" --no-web -c "${USERS:-10}" -r 10 -n 99999

Then after it restarted, now it's stuck again on 233 requests. :(

For posterity, my locustfile is here: https://gist.github.com/ahmetb/2645deefff78216933779eeb4043a5e2

@ahmetb ahmetb changed the title Locust freezes after a while Locust freezes after a while (python 3.7) Jul 15, 2018
@ahmetb
Copy link
Author

ahmetb commented Jul 15, 2018

After a bit of debugging and switching to python:3.6.6 image, this no longer reproduces. So I'm updating title to match that. Not sure why python:3.7.0 was causing this issue.

@aldenpeterson-wf
Copy link
Contributor

Currently, no one has verified Locust vs 3.7.0 and CI isn't run vs it.

mtwo pushed a commit to mtwo/hipstershop-demo that referenced this issue Jul 18, 2018
python3.7 is causing lockup in locust.
See locustio/locust#843

Signed-off-by: Ahmet Alp Balkan <ahmetb@google.com>
ahmetb added a commit to GoogleCloudPlatform/microservices-demo that referenced this issue Aug 3, 2018
python3.7 is causing lockup in locust.
See locustio/locust#843

Signed-off-by: Ahmet Alp Balkan <ahmetb@google.com>
neptune-web added a commit to neptune-web/kubernetes-microservices-sam that referenced this issue Jun 12, 2022
python3.7 is causing lockup in locust.
See locustio/locust#843

Signed-off-by: Ahmet Alp Balkan <ahmetb@google.com>
oplik0 pushed a commit to oplik0/bso that referenced this issue Jun 6, 2024
python3.7 is causing lockup in locust.
See locustio/locust#843

Signed-off-by: Ahmet Alp Balkan <ahmetb@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants