Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Locust docker version 1.4.0 using 100% CPU on idle #1629

Closed
simakuutio opened this issue Nov 16, 2020 · 17 comments
Closed

Locust docker version 1.4.0 using 100% CPU on idle #1629

simakuutio opened this issue Nov 16, 2020 · 17 comments
Labels

Comments

@simakuutio
Copy link

simakuutio commented Nov 16, 2020

Test environment:
Ubuntu 18.04.1
Docker version 19.03.6

Environment stays the same only the docker image version is changed.
Running the docker container with the same locustifle.py which is an example from https://hub.docker.com/r/locustio/locust

from locust import HttpUser, task

class QuickstartUser(HttpUser):
    def on_start(self):
        self.client.post
("/login", json={"username":"foo", "password":"bar"})

    @task
    def hello_world(self):
        self.client.get("/hello")
        self.client.get("/world")

    @task(3)
    def view_item(self):
        for item_id in range(10):
            self.client.get(f"/item?id={item_id}", name="/item")

When running locustio/locust version 1.3.2 the idle CPU load is around 0.0% and stable.

Running locustio/locust version 1.4.0 the idle CPU load goes up to 100.0% at once when the container is started and stays between 96.7% and 100.0%.

@simakuutio simakuutio added the bug label Nov 16, 2020
@max-rocket-internet
Copy link
Contributor

What is consuming the CPU?

@heyman
Copy link
Member

heyman commented Nov 16, 2020

Maybe "caused" by #1626?

@heyman
Copy link
Member

heyman commented Nov 16, 2020

@simakuutio:

Since Locust 1.4, the default wait time for simulated locust users is zero. Therefore I recommend that you specify a wait time for QuickstartUser.

from locust import HttpUser, task, between

class QuickstartUser(HttpUser):
    # wait between 1 and 5 seconds between tasks
    wait_time = between(1, 5)
    ...

@simakuutio
Copy link
Author

If this is the recommended way, example files should be also updated accordingly.

@heyman
Copy link
Member

heyman commented Nov 16, 2020

If this is the recommended way, example files should be also updated accordingly.

I agree.

@eskallberg
Copy link

eskallberg commented Nov 16, 2020

Hi

The same result comes with or without wait_time parameter

Following example locustfile also uses 100% CPU

import time
from locust import HttpUser, task, between

class QuickstartUser(HttpUser):
    wait_time = between(1, 2)

    @task
    def index_page(self):
        self.client.get("/hello")
        self.client.get("/world")

    @task(3)
    def view_item(self):
        for item_id in range(10):
            self.client.get(f"/item?id={item_id}", name="/item")
            time.sleep(1)

        def on_start(self):
            self.client.post("/login", json={"username":"foo", "password":"bar"})

Command locust is using 100% CPU inside the Docker container

@cyberw
Copy link
Collaborator

cyberw commented Nov 16, 2020

I cant reproduce 100% cpu usage on any of the suggested test plans.

Do you mean 100% cpu on the load gen or on the server, or are you running them both on the same machine?

Is it possible that it is somehow related to the keyboard input, which was also introduced in 1.4?

@cyberw
Copy link
Collaborator

cyberw commented Nov 16, 2020

@eskallberg are you also on Ubuntu?

@eskallberg
Copy link

Yes

@cyberw
Copy link
Collaborator

cyberw commented Nov 16, 2020

@simakuutio Btw, when you say "idle CPU load" what do you mean exactly? I guess you mean "while running load", but that is kind of the opposite of "idle", so I just want to be sure?

@heyman
Copy link
Member

heyman commented Nov 16, 2020

Btw, when you say "idle CPU load" what do you mean exactly? I guess you mean "while running load"

That's what I thought as well. I can reproduce the issue with just a few number of simulated users for the original example when running against a HTTP server that can handle the load. I haven't been able to reproduce it when the test isn't running. About to test @eskallberg 's example in Docker container now.

@cyberw
Copy link
Collaborator

cyberw commented Nov 16, 2020

I can reproduce this in docker/Ubuntu and it also prints the following exception when shutting down. I'm 99% confident this is an issue with keyboard input (on Ubuntu, not MacOS, CentOS or Windows), not with wait times.

I will check with @DennisKrone if he can fix it.

docker run -p 8089:8089 -v $PWD:/mnt/locust locustio/locust -f /mnt/locust/quickstart.py
[2020-11-16 12:10:57,399] c07ede557d74/INFO/locust.main: Starting web interface at http://0.0.0.0:8089 (accepting connections from all network interfaces)
[2020-11-16 12:10:57,412] c07ede557d74/INFO/locust.main: Starting Locust 1.4.0
[2020-11-16 12:11:02,415] c07ede557d74/WARNING/root: CPU usage above 90%! This may constrain your throughput and may even give inconsistent response time measurements! See https://docs.locust.io/en/stable/running-locust-distributed.html for how to distribute the load over multiple CPU cores or machines
^CTraceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 88, in input_listener_func
    input = poller.poll()
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 36, in poll
    return sys.stdin.read(1)
  File "/usr/local/lib/python3.8/codecs.py", line 319, in decode
    def decode(self, input, final=False):
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 854, in gevent._gevent_cgreenlet.Greenlet.run
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 98, in input_listener_func
    poller.cleanup()
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 31, in cleanup
    termios.tcsetattr(self.stdin, termios.TCSANOW, self.tattr)
AttributeError: 'UnixKeyPoller' object has no attribute 'tattr'
2020-11-16T12:11:07Z <Greenlet at 0x7fcc45787e10: input_listener_func> failed with AttributeError

[2020-11-16 12:11:07,514] c07ede557d74/CRITICAL/locust.main: Unhandled exception in greenlet: <Greenlet at 0x7fcc45787e10: input_listener_func>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 88, in input_listener_func
    input = poller.poll()
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 36, in poll
    return sys.stdin.read(1)
  File "/usr/local/lib/python3.8/codecs.py", line 319, in decode
    def decode(self, input, final=False):
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 854, in gevent._gevent_cgreenlet.Greenlet.run
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 98, in input_listener_func
    poller.cleanup()
  File "/usr/local/lib/python3.8/site-packages/locust/input_events.py", line 31, in cleanup
    termios.tcsetattr(self.stdin, termios.TCSANOW, self.tattr)
AttributeError: 'UnixKeyPoller' object has no attribute 'tattr'
KeyboardInterrupt
2020-11-16T12:11:07Z
[2020-11-16 12:11:07,527] c07ede557d74/INFO/locust.main: Running teardowns...
[2020-11-16 12:11:07,527] c07ede557d74/INFO/locust.main: Shutting down (exit code 2), bye.
[2020-11-16 12:11:07,527] c07ede557d74/INFO/locust.main: Cleaning up runner...
[2020-11-16 12:11:07,528] c07ede557d74/WARNING/locust.runners: CPU usage was too high at some point during the test! See https://docs.locust.io/en/stable/running-locust-distributed.html for how to distribute the load over multiple CPU cores or machines

@cyberw
Copy link
Collaborator

cyberw commented Nov 16, 2020

Aha, the problem is that when running docker, there is no interactive terminal. Add -it to your docker run command as a temp workaround.

@eskallberg
Copy link

I will try that and let you know how it goes with -it

About the idle, no heavy test running, just the container up.

@eskallberg
Copy link

I will try that and let you know how it goes with -it

About the idle, no heavy test running, just the container up.

with -it it works fine.

@heyman
Copy link
Member

heyman commented Nov 16, 2020

I can reproduce this in docker/Ubuntu and it also prints the following exception when shutting down. I'm 99% confident this is an issue with keyboard input (on Ubuntu, not MacOS, CentOS or Windows), not with wait times.

Nice find! The #1630 should fix it.

@cyberw
Copy link
Collaborator

cyberw commented Nov 16, 2020

Fixed in #1631

@cyberw cyberw closed this as completed Nov 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants