Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ft: LoadTestShapes with custom user classes #2181

Merged

Conversation

samuelspagl
Copy link
Contributor

Hey Locust Team,

I've been using locust for a while now, and really like to work with it. Still we achieved to run into some shortcomings for the project I am currently working on.

With this PR I would like to propose a new feature:

The possibility to create custom LoadTestShapes, where a step / tick only creates a specified set of users.

Why I need this feature / other maybe also need it:

In our Testscenario we need to make sure that a certain amount of locust users is definitely present upfront. Trying to achieve this with weighting or fixed_users is possible but from my experience also mixed with a bit of luck. As we are already using LoadTestShapes I asked myself, whether it is possible to include the users in the stages for example.

How to use it:

Using this would look like this for example:

stages = [
        {"duration": 60, "users": 10, "spawn_rate": 10, "user_classes": [WebsiteUserA]},
        {"duration": 100, "users": 50, "spawn_rate": 10, "user_classes": [WebsiteUserB]},
        {"duration": 180, "users": 100, "spawn_rate": 10, "user_classes": [WebsiteUserA, WebsiteUserB]},
        {"duration": 220, "users": 30, "spawn_rate": 10},
    ]

In this case the first step would only create users of the class WebsiteUserA, the second step WebsiteUserB and so on.
I also included a new example called staging_user_classes.py.

Is it backwards compatible:

Yeah, it should be. If the "user_classes"-entry is missing, it will use all the existing user classes as input, as it was before.

How it was achieved:

I needed to adjust three files:

  • shape.py
  • runners.py
  • dispatch.py

The maybe most important change (and maybe there's an even better solution for that) was, that in every new_dispatch() call (this happens for each tick / step once), also a new user_generator is being created. But in the end there were only a few minimal changes to the code done.

I really am looking forward hearing your thought.

Thanks a lot in advance.

locust/dispatch.py Outdated Show resolved Hide resolved
@cyberw
Copy link
Collaborator

cyberw commented Aug 31, 2022

Sounds like a good idea! I'll give some comments, but then we also need:

  • Tests (integration tests, maybe distributed as well as maybe some low level tests)
  • Documentation

I'd be really interested to hear @mboutet and @max-rocket-internet 's opinions on this too.

@max-rocket-internet
Copy link
Contributor

I think this is also what was mentioned in #2151

@max-rocket-internet
Copy link
Contributor

I'd be really interested to hear @mboutet and @max-rocket-internet 's opinions on this too.

If it doesn't affect existing uses of LoadTestShape, new tests are added and old tests pass then I think it's awesome 🎉

This PR looks quite simple but I don't have much understanding of how a load test works with multiple User classes. My use of locust has only ever been with a single User.

@samuelspagl
Copy link
Contributor Author

samuelspagl commented Sep 2, 2022

I think this is also what was mentioned in #2151

Yeah it's quite similar but an approach from a different direction.
My PR would just use one LoadTestShape, #2151 would use multiple concurrently, if I understood it correctly.

So using multiple users is working quite well, even without this PR. Our setup is just quite complex, also ensuring that some users have been spawned before others.
I will try to create new tests next week, but I still need to understand your test structure. And sooner or later I will probably need at least some advice 😁.

@mboutet
Copy link
Contributor

mboutet commented Sep 2, 2022

It's a great feature indeed!

Make sure to add tests. In particular, add tests in test_dispatch.py to validate the behavior for ramp-up, ramp-down, addition and removal of workers, etc.

@samuelspagl
Copy link
Contributor Author

It's a great feature indeed!

Make sure to add tests. In particular, add tests in test_dispatch.py to validate the behavior for ramp-up, ramp-down, addition and removal of workers, etc.

I added some tests. I will additionally add those for addition and removal of workers. The ramp-down, behaviour should be unchanged, as the user_classes property is not taken into account for this process, right? Should I still include tests for that? @mboutet

FYI @cyberw , @max-rocket-internet , @mboutet I had some issues getting an environment to work as expected. In the end I needed to copy all dependencies from the setup.cfg and put them into a Pipfile for pipenv. I couldn't get it working with venvor pip. I am developing on an ARM M1 Macbook.

@cyberw
Copy link
Collaborator

cyberw commented Sep 5, 2022

What was your error message? Unfortunately I only have an intel mac so I cant test it out for myself...

@samuelspagl
Copy link
Contributor Author

@cyberw Those are only the last log messages, because the whole would be quite long. But if wanted I can also post the whole thing.

  running build_ext
  running configure
  /private/var/folders/zb/r6xl1vyn38n3f1r93661zsl40000gn/T/pip-build-env-lcqqicr7/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py:262: UserWarning: Unknown distribution option: 'cffi_modules'
    warnings.warn(msg)
  Settings obtained from pkg-config: {'library_dirs': ['/opt/homebrew/Cellar/zeromq/4.3.4/lib'], 'include_dirs': ['/opt/homebrew/Cellar/zeromq/4.3.4/include', '/opt/homebrew/Cellar/libsodium/1.0.18_1/include'], 'libraries': ['zmq']}
  {'libraries': ['zmq'], 'include_dirs': ['/opt/homebrew/Cellar/zeromq/4.3.4/include', '/opt/homebrew/Cellar/libsodium/1.0.18_1/include'], 'library_dirs': ['/opt/homebrew/Cellar/zeromq/4.3.4/lib'], 'runtime_library_dirs': [], 'extra_link_args': ['-Wl,-rpath', '-Wl,/opt/homebrew/Cellar/zeromq/4.3.4/lib']}
  Configure: Autodetecting ZMQ settings...
      Custom ZMQ dir:
  ************************************************
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -Werror=implicit-function-declaration -I/opt/homebrew/Cellar/zeromq/4.3.4/include -I/opt/homebrew/Cellar/libsodium/1.0.18_1/include -Izmq/utils -c build/temp.macosx-10.14-arm64-cpython-38/scratch/vers.c -o build/temp.macosx-10.14-arm64-cpython-38/scratch/vers.o
  clang -undefined dynamic_lookup -Wl,-rpath -Wl,/opt/homebrew/Cellar/zeromq/4.3.4/lib build/temp.macosx-10.14-arm64-cpython-38/scratch/vers.o -L/opt/homebrew/Cellar/zeromq/4.3.4/lib -lzmq -o build/temp.macosx-10.14-arm64-cpython-38/scratch/vers
  ld: warning: dylib (/opt/homebrew/Cellar/zeromq/4.3.4/lib/libzmq.dylib) was built for newer macOS version (12.0) than being linked (11.0)
      ZMQ version detected: 4.3.4
  ************************************************
  building 'zmq.backend.cython._device' extension
  creating build/temp.macosx-10.14-arm64-cpython-38/zmq
  creating build/temp.macosx-10.14-arm64-cpython-38/zmq/backend
  creating build/temp.macosx-10.14-arm64-cpython-38/zmq/backend/cython
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -Werror=implicit-function-declaration -DHAVE_SYS_UN_H=1 -I/opt/homebrew/Cellar/zeromq/4.3.4/include -I/opt/homebrew/Cellar/libsodium/1.0.18_1/include -Izmq/utils -I/Users/samuelspagl/work/locust/venv/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -c zmq/backend/cython/_device.c -o build/temp.macosx-10.14-arm64-cpython-38/zmq/backend/cython/_device.o
  zmq/backend/cython/_device.c:25:10: fatal error: 'Python.h' file not found
  #include "Python.h"
           ^~~~~~~~~~
  1 error generated.
  error: command '/usr/bin/clang' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for pyzmq
  Building wheel for zope.interface (setup.py) ... done
  Created wheel for zope.interface: filename=zope.interface-5.4.0-cp38-cp38-macosx_10_14_arm64.whl size=219808 sha256=c1b18af1d317b2ae68edaf5783554084fb0b6c03316765c0311e2f3c40598181
  Stored in directory: /Users/samuelspagl/Library/Caches/pip/wheels/f6/d5/8a/522a527f3831d7baa52a67b0d6f45c5872aad25058e4a34b16
Successfully built locust greenlet psutil zope.interface
Failed to build gevent pyzmq
ERROR: Could not build wheels for gevent, pyzmq which use PEP 517 and cannot be installed directly
WARNING: You are using pip version 21.1.2; however, version 22.2.2 is available.
You should consider upgrading via the '/Users/samuelspagl/work/locust/venv/bin/python -m pip install --upgrade pip' command.

@samuelspagl
Copy link
Contributor Author

And one question, for running the test_main.py the locust command needs to be registered in the system right? Right now these tests fail locally, because it doesn't find the command. Do I need to add something additionally?

@cyberw
Copy link
Collaborator

cyberw commented Sep 5, 2022

'Python.h' file not found typically means that you have only installed the python binary and not its headers. Did you install python using brew or some other method?

@cyberw
Copy link
Collaborator

cyberw commented Sep 5, 2022

And one question, for running the test_main.py the locust command needs to be registered in the system right? Right now these tests fail locally, because it doesn't find the command. Do I need to add something additionally?

Yes. You may also need to add the things listed in tox.ini under testenv:

codecov
mock
retry
pyquery
cryptography

@samuelspagl
Copy link
Contributor Author

@cyberw Thanks that helped a lot. I had some issues with the relative imports done in some of the tests, but after resolving them locally nearly everything seems to work.

I have one failing test and am pretty clueless about it. I would be happy to get some guidance from you guys :)

======================================================================
FAIL: test_distributed_tags (locust.test.test_main.DistributedIntegrationTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/samuelspagl/work/locust/locust/test/test_main.py", line 1231, in test_distributed_tags
    self.assertIn("task1", stdout_worker)
AssertionError: 'task1' not found in ''

----------------------------------------------------------------------
Ran 511 tests in 404.766s

FAILED (failures=1, skipped=1)

This is the only one who's failing. So functioning wise everything is working as expected (Tested it with a custom_shape locust file from the examples in standalone and distributed mode).

@cyberw
Copy link
Collaborator

cyberw commented Sep 6, 2022

Try reordering the assertions to check stderr or exit code first, that may give more info...

@samuelspagl
Copy link
Contributor Author

This is the error:

AssertionError: 'Traceback' unexpectedly found in 
'[2022-09-06 11:22:39,756] Samuels-MacBook-Pro.local/DEBUG/locust.main: Connected to locust master: 127.0.0.1:5557\n
[2022-09-06 11:22:39,756] Samuels-MacBook-Pro.local/INFO/locust.main: Starting Locust 2.8.7.dev194\n
[2022-09-06 11:22:40,821] Samuels-MacBook-Pro.local/DEBUG/locust.runners: Spawning additional {"UserSubclass": 1} ({"UserSubclass": 0, "SecondUser": 0} already running)...\n
[2022-09-06 11:22:40,821] Samuels-MacBook-Pro.local/DEBUG/locust.runners: 1 users spawned\n
[2022-09-06 11:22:40,821] Samuels-MacBook-Pro.local/DEBUG/locust.runners: All users of class UserSubclass spawned\n
[2022-09-06 11:22:40,821] Samuels-MacBook-Pro.local/DEBUG/locust.runners: 0 users have been stopped, 1 still running\n
[2022-09-06 11:22:40,822] Samuels-MacBook-Pro.local/ERROR/locust.user.task: No tasks defined on UserSubclass. Use the @task decorator or set the \'tasks\' attribute of the User (or mark it as abstract = True if you only intend to subclass it)\nTraceback (most recent call last):\n  File "/Users/samuelspagl/work/locust/locust/user/task.py", line 340, in run\n    self.schedule_task(self.get_next_task())\n  File "/Users/samuelspagl/work/locust/locust/user/task.py", line 480, in get_next_task\n    raise Exception(\nException: No tasks defined on UserSubclass. Use the @task decorator or set the \'tasks\' attribute of the User (or mark it as abstract = True if you only intend to subclass it)\n\n
[2022-09-06 11:22:41,763] Samuels-MacBook-Pro.local/INFO/locust.runners: Got quit message from master, shutting down...\n
[2022-09-06 11:22:41,763] Samuels-MacBook-Pro.local/DEBUG/locust.runners: Stopping all users\n
[2022-09-06 11:22:41,763] Samuels-MacBook-Pro.local/DEBUG/locust.runners: Stopping Greenlet-0\n
[2022-09-06 11:22:41,764] Samuels-MacBook-Pro.local/DEBUG/locust.runners: 1 users have been stopped, 0 still running\n
[2022-09-06 11:22:41,764] Samuels-MacBook-Pro.local/DEBUG/locust.main: Running teardowns...\n
[2022-09-06 11:22:41,764] Samuels-MacBook-Pro.local/INFO/locust.main: Shutting down (exit code 0)\n
[2022-09-06 11:22:41,765] Samuels-MacBook-Pro.local/DEBUG/locust.main: Cleaning up runner...\n'

@samuelspagl
Copy link
Contributor Author

And this is the locust file from the test.

    def test_distributed_tags(self):
        content = (
            MOCK_LOCUSTFILE_CONTENT
            + """
from locust import tag
class SecondUser(HttpUser):
    host = "http://127.0.0.1:8089"
    wait_time = between(0, 0.1)
    @tag("tag1")
    @task
    def task1(self):
        print("task1")

    @tag("tag2")
    @task
    def task2(self):
        print("task2")
"""

@samuelspagl
Copy link
Contributor Author

@cyberw Okay so the test test_distributed_tags(self) which is included in the test_main.py works if in the in the content variable the MOCK_LOCUSTFILE_CONTENT is removed.
The MOCK_LOCUSTFILE_CONTENT is also where the in the error logs included UserSubclass is coming from.

Is this a bug in the test?

The working test looks like this:

 content = ( """
from locust import tag, HttpUser, task, between, User
class SecondUser(HttpUser):
    host = "http://127.0.0.1:8089"
    wait_time = between(0, 0.1)
    @tag("tag1")
    @task
    def task1(self):
        print("task1")

    @tag("tag2")
    @task
    def task2(self):
        print("task2")
"""
        )
        with mock_locustfile(content=content) as mocked:

@cyberw
Copy link
Collaborator

cyberw commented Sep 6, 2022

Hmm... sorry, I cant really think of why it should fail like that, and I dont see any error in the test.

Is it possible that your changes actually broke something?

@samuelspagl
Copy link
Contributor Author

@cyberw Okay so with some thought I found out why. Yeah there were also some changes that broke it.

But this issue has two sides:

 content = (
            MOCK_LOCUSTFILE_CONTENT
            + """
from locust import tag
class SecondUser(HttpUser):
    host = "http://127.0.0.1:8089"
    wait_time = between(0, 0.1)
    @tag("tag1")
    @task
    def task1(self):
        print("task1")

    @tag("tag2")
    @task
    def task2(self):
        print("task2")
"""

This is the code how it was. The thing is that content in this case is including 2 structures of a locust file. The one included in MOCK_LOCUSTFILE_CONTENT and the one added as text. The result is that two users are existing: UserSubclass and SecondUser. In the beginning of initialising the dispatcher, all of the classes are sorted by name. That's why SecondUser is dispatched before UserSubclass and the reason the test was working beforehand.

As I implemented the feature I forgot that each time I update the user_classes I also need to sort them.

I changed that, and locally it works as expected. But still if the user would not be called SecondUser but UserTestStuff the test would also fail because, UserSubclass is going to be deployed before UserTestStuff.

@cyberw
Copy link
Collaborator

cyberw commented Sep 6, 2022

Ah, I see now. The problem is that UserSubclass has no tasks tagged with tag1, and thus instantiating it throws that exception. So unrelated to your change, and I should probably adjust the exception message if there are tags. But it is still good that you sort your users (for consistency)

It is starting to look pretty good now. Do we need a distributed test? Probably not, as workers wont know any difference..

docs/custom-load-shape.rst Outdated Show resolved Hide resolved
locust/runners.py Outdated Show resolved Hide resolved
locust/runners.py Outdated Show resolved Hide resolved
locust/runners.py Outdated Show resolved Hide resolved
@@ -486,6 +494,9 @@ def _start(self, user_count: int, spawn_rate: float, wait: bool = False) -> None
if wait and user_count - self.user_count > spawn_rate:
raise ValueError("wait is True but the amount of users to add is greater than the spawn rate")

# if user_classes is None:
# user_classes = self.user_classes

for user_class in self.user_classes:
if self.environment.host:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if we combine a -H/--host parameter with shape-configured Users? I guess that will still work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So normally it should. I'll try it out tomorrow :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cyberw So I tried it with a few example locust files, and it seemed to work.

If you want to be sure, you can also try it out. :)

locust/runners.py Outdated Show resolved Hide resolved
locust/shape.py Outdated Show resolved Hide resolved
@samuelspagl
Copy link
Contributor Author

@cyberw Do you want to resolve the threads yourself, or should I do it? :)

@cyberw
Copy link
Collaborator

cyberw commented Sep 6, 2022

Go ahead and resolve as you fix stuff, I'm too lazy :)

@cyberw
Copy link
Collaborator

cyberw commented Sep 6, 2022

I made a PR (#2186) to log a warning when tag filtering made it so that there were no tasks left. Probably not the most beautiful solution, but should be better than before.

@cyberw
Copy link
Collaborator

cyberw commented Sep 7, 2022

looking really good now. just need to rename Runner.shape_last_state to shape_last_tick as well, to be consistent (sorry for derailing your PR a little bit :)

@samuelspagl
Copy link
Contributor Author

looking really good now. just need to rename Runner.shape_last_state to shape_last_tick as well, to be consistent (sorry for derailing your PR a little bit :)

Don't worry, code style and consistency is important.

@cyberw An off-topic question for the tests. Is there a reason that in some of the tests the imports are relative and not absolute? Just out of curiosity :)

@cyberw
Copy link
Collaborator

cyberw commented Sep 7, 2022

Tbh, I dont know why, could be just other people writing them :)

Thanks for this PR, merging now!

@cyberw cyberw merged commit 96986de into locustio:master Sep 7, 2022
@samuelspagl
Copy link
Contributor Author

Thanks for all the comments @cyberw :)

Do you already know when a new release is going to be created?

@cyberw
Copy link
Collaborator

cyberw commented Sep 7, 2022

Thanks for all the comments @cyberw :)

Do you already know when a new release is going to be created?

Done. 2.12.0 should be available in less than 10 minutes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants