ft: LoadTestShapes with custom user classes #2181

samuelspagl · 2022-08-31T15:14:32Z

Hey Locust Team,

I've been using locust for a while now, and really like to work with it. Still we achieved to run into some shortcomings for the project I am currently working on.

With this PR I would like to propose a new feature:

The possibility to create custom LoadTestShapes, where a step / tick only creates a specified set of users.

Why I need this feature / other maybe also need it:

In our Testscenario we need to make sure that a certain amount of locust users is definitely present upfront. Trying to achieve this with weighting or fixed_users is possible but from my experience also mixed with a bit of luck. As we are already using LoadTestShapes I asked myself, whether it is possible to include the users in the stages for example.

How to use it:

Using this would look like this for example:

stages = [
        {"duration": 60, "users": 10, "spawn_rate": 10, "user_classes": [WebsiteUserA]},
        {"duration": 100, "users": 50, "spawn_rate": 10, "user_classes": [WebsiteUserB]},
        {"duration": 180, "users": 100, "spawn_rate": 10, "user_classes": [WebsiteUserA, WebsiteUserB]},
        {"duration": 220, "users": 30, "spawn_rate": 10},
    ]

In this case the first step would only create users of the class WebsiteUserA, the second step WebsiteUserB and so on.
I also included a new example called staging_user_classes.py.

Is it backwards compatible:

Yeah, it should be. If the "user_classes"-entry is missing, it will use all the existing user classes as input, as it was before.

How it was achieved:

I needed to adjust three files:

shape.py
runners.py
dispatch.py

The maybe most important change (and maybe there's an even better solution for that) was, that in every new_dispatch() call (this happens for each tick / step once), also a new user_generator is being created. But in the end there were only a few minimal changes to the code done.

I really am looking forward hearing your thought.

Thanks a lot in advance.

locust/dispatch.py

cyberw · 2022-08-31T17:41:32Z

Sounds like a good idea! I'll give some comments, but then we also need:

Tests (integration tests, maybe distributed as well as maybe some low level tests)
Documentation

I'd be really interested to hear @mboutet and @max-rocket-internet 's opinions on this too.

max-rocket-internet · 2022-09-02T14:45:37Z

I think this is also what was mentioned in #2151

max-rocket-internet · 2022-09-02T14:48:41Z

I'd be really interested to hear @mboutet and @max-rocket-internet 's opinions on this too.

If it doesn't affect existing uses of LoadTestShape, new tests are added and old tests pass then I think it's awesome 🎉

This PR looks quite simple but I don't have much understanding of how a load test works with multiple User classes. My use of locust has only ever been with a single User.

samuelspagl · 2022-09-02T15:15:26Z

I think this is also what was mentioned in #2151

Yeah it's quite similar but an approach from a different direction.
My PR would just use one LoadTestShape, #2151 would use multiple concurrently, if I understood it correctly.

So using multiple users is working quite well, even without this PR. Our setup is just quite complex, also ensuring that some users have been spawned before others.
I will try to create new tests next week, but I still need to understand your test structure. And sooner or later I will probably need at least some advice 😁.

mboutet · 2022-09-02T17:18:33Z

It's a great feature indeed!

Make sure to add tests. In particular, add tests in test_dispatch.py to validate the behavior for ramp-up, ramp-down, addition and removal of workers, etc.

samuelspagl · 2022-09-05T09:15:39Z

It's a great feature indeed!

Make sure to add tests. In particular, add tests in test_dispatch.py to validate the behavior for ramp-up, ramp-down, addition and removal of workers, etc.

I added some tests. I will additionally add those for addition and removal of workers. The ramp-down, behaviour should be unchanged, as the user_classes property is not taken into account for this process, right? Should I still include tests for that? @mboutet

FYI @cyberw , @max-rocket-internet , @mboutet I had some issues getting an environment to work as expected. In the end I needed to copy all dependencies from the setup.cfg and put them into a Pipfile for pipenv. I couldn't get it working with venvor pip. I am developing on an ARM M1 Macbook.

cyberw · 2022-09-05T11:35:46Z

What was your error message? Unfortunately I only have an intel mac so I cant test it out for myself...

…asses are included.

samuelspagl · 2022-09-05T15:26:48Z

@cyberw Those are only the last log messages, because the whole would be quite long. But if wanted I can also post the whole thing.

  running build_ext
  running configure
  /private/var/folders/zb/r6xl1vyn38n3f1r93661zsl40000gn/T/pip-build-env-lcqqicr7/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py:262: UserWarning: Unknown distribution option: 'cffi_modules'
    warnings.warn(msg)
  Settings obtained from pkg-config: {'library_dirs': ['/opt/homebrew/Cellar/zeromq/4.3.4/lib'], 'include_dirs': ['/opt/homebrew/Cellar/zeromq/4.3.4/include', '/opt/homebrew/Cellar/libsodium/1.0.18_1/include'], 'libraries': ['zmq']}
  {'libraries': ['zmq'], 'include_dirs': ['/opt/homebrew/Cellar/zeromq/4.3.4/include', '/opt/homebrew/Cellar/libsodium/1.0.18_1/include'], 'library_dirs': ['/opt/homebrew/Cellar/zeromq/4.3.4/lib'], 'runtime_library_dirs': [], 'extra_link_args': ['-Wl,-rpath', '-Wl,/opt/homebrew/Cellar/zeromq/4.3.4/lib']}
  Configure: Autodetecting ZMQ settings...
      Custom ZMQ dir:
  ************************************************
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -Werror=implicit-function-declaration -I/opt/homebrew/Cellar/zeromq/4.3.4/include -I/opt/homebrew/Cellar/libsodium/1.0.18_1/include -Izmq/utils -c build/temp.macosx-10.14-arm64-cpython-38/scratch/vers.c -o build/temp.macosx-10.14-arm64-cpython-38/scratch/vers.o
  clang -undefined dynamic_lookup -Wl,-rpath -Wl,/opt/homebrew/Cellar/zeromq/4.3.4/lib build/temp.macosx-10.14-arm64-cpython-38/scratch/vers.o -L/opt/homebrew/Cellar/zeromq/4.3.4/lib -lzmq -o build/temp.macosx-10.14-arm64-cpython-38/scratch/vers
  ld: warning: dylib (/opt/homebrew/Cellar/zeromq/4.3.4/lib/libzmq.dylib) was built for newer macOS version (12.0) than being linked (11.0)
      ZMQ version detected: 4.3.4
  ************************************************
  building 'zmq.backend.cython._device' extension
  creating build/temp.macosx-10.14-arm64-cpython-38/zmq
  creating build/temp.macosx-10.14-arm64-cpython-38/zmq/backend
  creating build/temp.macosx-10.14-arm64-cpython-38/zmq/backend/cython
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -Werror=implicit-function-declaration -DHAVE_SYS_UN_H=1 -I/opt/homebrew/Cellar/zeromq/4.3.4/include -I/opt/homebrew/Cellar/libsodium/1.0.18_1/include -Izmq/utils -I/Users/samuelspagl/work/locust/venv/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -c zmq/backend/cython/_device.c -o build/temp.macosx-10.14-arm64-cpython-38/zmq/backend/cython/_device.o
  zmq/backend/cython/_device.c:25:10: fatal error: 'Python.h' file not found
  #include "Python.h"
           ^~~~~~~~~~
  1 error generated.
  error: command '/usr/bin/clang' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for pyzmq
  Building wheel for zope.interface (setup.py) ... done
  Created wheel for zope.interface: filename=zope.interface-5.4.0-cp38-cp38-macosx_10_14_arm64.whl size=219808 sha256=c1b18af1d317b2ae68edaf5783554084fb0b6c03316765c0311e2f3c40598181
  Stored in directory: /Users/samuelspagl/Library/Caches/pip/wheels/f6/d5/8a/522a527f3831d7baa52a67b0d6f45c5872aad25058e4a34b16
Successfully built locust greenlet psutil zope.interface
Failed to build gevent pyzmq
ERROR: Could not build wheels for gevent, pyzmq which use PEP 517 and cannot be installed directly
WARNING: You are using pip version 21.1.2; however, version 22.2.2 is available.
You should consider upgrading via the '/Users/samuelspagl/work/locust/venv/bin/python -m pip install --upgrade pip' command.

samuelspagl · 2022-09-05T15:29:17Z

And one question, for running the test_main.py the locust command needs to be registered in the system right? Right now these tests fail locally, because it doesn't find the command. Do I need to add something additionally?

cyberw · 2022-09-05T15:49:10Z

'Python.h' file not found typically means that you have only installed the python binary and not its headers. Did you install python using brew or some other method?

cyberw · 2022-09-05T15:51:35Z

And one question, for running the test_main.py the locust command needs to be registered in the system right? Right now these tests fail locally, because it doesn't find the command. Do I need to add something additionally?

Yes. You may also need to add the things listed in tox.ini under testenv:

codecov
mock
retry
pyquery
cryptography

samuelspagl · 2022-09-06T08:03:11Z

@cyberw Thanks that helped a lot. I had some issues with the relative imports done in some of the tests, but after resolving them locally nearly everything seems to work.

I have one failing test and am pretty clueless about it. I would be happy to get some guidance from you guys :)

======================================================================
FAIL: test_distributed_tags (locust.test.test_main.DistributedIntegrationTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/samuelspagl/work/locust/locust/test/test_main.py", line 1231, in test_distributed_tags
    self.assertIn("task1", stdout_worker)
AssertionError: 'task1' not found in ''

----------------------------------------------------------------------
Ran 511 tests in 404.766s

FAILED (failures=1, skipped=1)

This is the only one who's failing. So functioning wise everything is working as expected (Tested it with a custom_shape locust file from the examples in standalone and distributed mode).

cyberw · 2022-09-06T09:11:08Z

Try reordering the assertions to check stderr or exit code first, that may give more info...

samuelspagl · 2022-09-06T09:26:55Z

This is the error:

AssertionError: 'Traceback' unexpectedly found in 
'[2022-09-06 11:22:39,756] Samuels-MacBook-Pro.local/DEBUG/locust.main: Connected to locust master: 127.0.0.1:5557\n
[2022-09-06 11:22:39,756] Samuels-MacBook-Pro.local/INFO/locust.main: Starting Locust 2.8.7.dev194\n
[2022-09-06 11:22:40,821] Samuels-MacBook-Pro.local/DEBUG/locust.runners: Spawning additional {"UserSubclass": 1} ({"UserSubclass": 0, "SecondUser": 0} already running)...\n
[2022-09-06 11:22:40,821] Samuels-MacBook-Pro.local/DEBUG/locust.runners: 1 users spawned\n
[2022-09-06 11:22:40,821] Samuels-MacBook-Pro.local/DEBUG/locust.runners: All users of class UserSubclass spawned\n
[2022-09-06 11:22:40,821] Samuels-MacBook-Pro.local/DEBUG/locust.runners: 0 users have been stopped, 1 still running\n
[2022-09-06 11:22:40,822] Samuels-MacBook-Pro.local/ERROR/locust.user.task: No tasks defined on UserSubclass. Use the @task decorator or set the \'tasks\' attribute of the User (or mark it as abstract = True if you only intend to subclass it)\nTraceback (most recent call last):\n  File "/Users/samuelspagl/work/locust/locust/user/task.py", line 340, in run\n    self.schedule_task(self.get_next_task())\n  File "/Users/samuelspagl/work/locust/locust/user/task.py", line 480, in get_next_task\n    raise Exception(\nException: No tasks defined on UserSubclass. Use the @task decorator or set the \'tasks\' attribute of the User (or mark it as abstract = True if you only intend to subclass it)\n\n
[2022-09-06 11:22:41,763] Samuels-MacBook-Pro.local/INFO/locust.runners: Got quit message from master, shutting down...\n
[2022-09-06 11:22:41,763] Samuels-MacBook-Pro.local/DEBUG/locust.runners: Stopping all users\n
[2022-09-06 11:22:41,763] Samuels-MacBook-Pro.local/DEBUG/locust.runners: Stopping Greenlet-0\n
[2022-09-06 11:22:41,764] Samuels-MacBook-Pro.local/DEBUG/locust.runners: 1 users have been stopped, 0 still running\n
[2022-09-06 11:22:41,764] Samuels-MacBook-Pro.local/DEBUG/locust.main: Running teardowns...\n
[2022-09-06 11:22:41,764] Samuels-MacBook-Pro.local/INFO/locust.main: Shutting down (exit code 0)\n
[2022-09-06 11:22:41,765] Samuels-MacBook-Pro.local/DEBUG/locust.main: Cleaning up runner...\n'

samuelspagl · 2022-09-06T09:32:19Z

And this is the locust file from the test.

    def test_distributed_tags(self):
        content = (
            MOCK_LOCUSTFILE_CONTENT
            + """
from locust import tag
class SecondUser(HttpUser):
    host = "http://127.0.0.1:8089"
    wait_time = between(0, 0.1)
    @tag("tag1")
    @task
    def task1(self):
        print("task1")

    @tag("tag2")
    @task
    def task2(self):
        print("task2")
"""

samuelspagl · 2022-09-06T10:13:03Z

@cyberw Okay so the test test_distributed_tags(self) which is included in the test_main.py works if in the in the content variable the MOCK_LOCUSTFILE_CONTENT is removed.
The MOCK_LOCUSTFILE_CONTENT is also where the in the error logs included UserSubclass is coming from.

Is this a bug in the test?

The working test looks like this:

 content = ( """
from locust import tag, HttpUser, task, between, User
class SecondUser(HttpUser):
    host = "http://127.0.0.1:8089"
    wait_time = between(0, 0.1)
    @tag("tag1")
    @task
    def task1(self):
        print("task1")

    @tag("tag2")
    @task
    def task2(self):
        print("task2")
"""
        )
        with mock_locustfile(content=content) as mocked:

cyberw · 2022-09-06T10:19:24Z

Hmm... sorry, I cant really think of why it should fail like that, and I dont see any error in the test.

Is it possible that your changes actually broke something?

samuelspagl · 2022-09-06T11:06:11Z

@cyberw Okay so with some thought I found out why. Yeah there were also some changes that broke it.

But this issue has two sides:

 content = (
            MOCK_LOCUSTFILE_CONTENT
            + """
from locust import tag
class SecondUser(HttpUser):
    host = "http://127.0.0.1:8089"
    wait_time = between(0, 0.1)
    @tag("tag1")
    @task
    def task1(self):
        print("task1")

    @tag("tag2")
    @task
    def task2(self):
        print("task2")
"""

This is the code how it was. The thing is that content in this case is including 2 structures of a locust file. The one included in MOCK_LOCUSTFILE_CONTENT and the one added as text. The result is that two users are existing: UserSubclass and SecondUser. In the beginning of initialising the dispatcher, all of the classes are sorted by name. That's why SecondUser is dispatched before UserSubclass and the reason the test was working beforehand.

As I implemented the feature I forgot that each time I update the user_classes I also need to sort them.

I changed that, and locally it works as expected. But still if the user would not be called SecondUser but UserTestStuff the test would also fail because, UserSubclass is going to be deployed before UserTestStuff.

cyberw · 2022-09-06T20:37:02Z

Ah, I see now. The problem is that UserSubclass has no tasks tagged with tag1, and thus instantiating it throws that exception. So unrelated to your change, and I should probably adjust the exception message if there are tags. But it is still good that you sort your users (for consistency)

It is starting to look pretty good now. Do we need a distributed test? Probably not, as workers wont know any difference..

docs/custom-load-shape.rst

locust/runners.py

cyberw · 2022-09-06T21:00:58Z

locust/runners.py

@@ -486,6 +494,9 @@ def _start(self, user_count: int, spawn_rate: float, wait: bool = False) -> None
        if wait and user_count - self.user_count > spawn_rate:
            raise ValueError("wait is True but the amount of users to add is greater than the spawn rate")

+        # if user_classes is None:
+        #    user_classes = self.user_classes
+
        for user_class in self.user_classes:
            if self.environment.host:


What happens if we combine a -H/--host parameter with shape-configured Users? I guess that will still work?

So normally it should. I'll try it out tomorrow :)

@cyberw So I tried it with a few example locust files, and it seemed to work.

If you want to be sure, you can also try it out. :)

locust/runners.py

locust/shape.py

samuelspagl · 2022-09-06T21:20:39Z

@cyberw Do you want to resolve the threads yourself, or should I do it? :)

cyberw · 2022-09-06T21:44:39Z

Go ahead and resolve as you fix stuff, I'm too lazy :)

cyberw · 2022-09-06T21:53:09Z

I made a PR (#2186) to log a warning when tag filtering made it so that there were no tasks left. Probably not the most beautiful solution, but should be better than before.

cyberw · 2022-09-07T07:29:06Z

looking really good now. just need to rename Runner.shape_last_state to shape_last_tick as well, to be consistent (sorry for derailing your PR a little bit :)

samuelspagl · 2022-09-07T07:49:35Z

looking really good now. just need to rename Runner.shape_last_state to shape_last_tick as well, to be consistent (sorry for derailing your PR a little bit :)

Don't worry, code style and consistency is important.

@cyberw An off-topic question for the tests. Is there a reason that in some of the tests the imports are relative and not absolute? Just out of curiosity :)

cyberw · 2022-09-07T08:42:26Z

Tbh, I dont know why, could be just other people writing them :)

Thanks for this PR, merging now!

samuelspagl · 2022-09-07T09:32:02Z

Thanks for all the comments @cyberw :)

Do you already know when a new release is going to be created?

cyberw · 2022-09-07T12:10:52Z

Thanks for all the comments @cyberw :)

Do you already know when a new release is going to be created?

Done. 2.12.0 should be available in less than 10 minutes.

Add possibility to extend the stages with control over userclasses

0346a4e

cyberw reviewed Aug 31, 2022

View reviewed changes

locust/dispatch.py Outdated Show resolved Hide resolved

samuelspagl added 4 commits September 1, 2022 09:11

add better type hinting and docstring

e569c87

add better type hinting

9080834

add documentation entry

c621217

changes according to flake8, black, mypy

a39b3a9

adding some tests

7a3cdc5

samuelspagl added 2 commits September 5, 2022 17:06

Updating Tests

9367ca8

Small fix in new dispatch, not creating a new generator if all usercl…

70b551b

…asses are included.

samuelspagl added 2 commits September 6, 2022 09:13

Updated files according to flake, blake and mypy

c131241

fix arguments of new_dispatch

713477c

add sorting of userclasses

389f68a

typing according to python 3.7

fdabf81