Artillery-playwright process seems to hang on fargate container

[zoe-jobson-IL]

WE are trying to scale out our first load test, so we are trying to run with several fargate machines each with 25 VUs on. in many test runs there is one which gets stuck until it times out.

WHen I look at the ends of the fargate cloudwatch logs the sucessful ones tend to look like this

2024-02-01T16:01:59.254+00:00 | vusers.session_length:
  | 2024-02-01T16:01:59.254+00:00 | min: ......................................................................... 10396.5
  | 2024-02-01T16:01:59.254+00:00 | max: ......................................................................... 59254.8
  | 2024-02-01T16:01:59.254+00:00 | mean: ........................................................................ 32310.5
  | 2024-02-01T16:01:59.254+00:00 | median: ...................................................................... 30647.1
  | 2024-02-01T16:01:59.254+00:00 | p95: ......................................................................... 44815.4
  | 2024-02-01T16:01:59.254+00:00 | p99: ......................................................................... 44815.4
  | 2024-02-01T16:01:59.254+00:00 | vusers.skipped: ................................................................ 110
  | 2024-02-01T16:02:02.884+00:00 | Finished with code 0
  | 2024-02-01T16:02:02.886+00:00 | no ensure spec
  | 2024-02-01T16:02:03.444+00:00 | Completed 123.3 KiB/123.3 KiB (596.2 KiB/s) with 1 file(s) remaining upload: ./output.txt to s3://artilleryio-test-data-049107765258/test-runs/t373c_a3t5658m6cxc3becqx5tc58zrtj8y_p96p/worker-log-7dd1660df0c5487594904252f9bb1da6.txt
  | 2024-02-01T16:02:03.499+00:00Copylog: s3://artilleryio-test-data-049107765258/test-runs/t373c_a3t5658m6cxc3becqx5tc58zrtj8y_p96p/worker-log-7dd1660df0c5487594904252f9bb1da6.txt | log: s3://artilleryio-test-data-049107765258/test-runs/t373c_a3t5658m6cxc3becqx5tc58zrtj8y_p96p/worker-log-7dd1660df0c5487594904252f9bb1da6.txt
  | 2024-02-01T16:02:03.499+00:00 | cleanup called, signal:
  | 2024-02-01T16:02:03.499+00:00 | EXIT
  | 2024-02-01T16:02:03.988+00:00 | Message body: {"event": "workerDone"}
  | 2024-02-01T16:02:03.988+00:00 | Message attributes: {"testId": {"StringValue": "t373c_a3t5658m6cxc3becqx5tc58zrtj8y_p96p", "DataType": "String"}, "workerId": {"StringValue": "7dd1660df0c5487594904252f9bb1da6", "DataType": "String"}}
  |  

and the failing one looks like this, NOTE, the last message before the container gets the TERM is truncated.

  | 2024-02-01T16:02:01.196+00:00 | vusers.session_length:
-- | -- | --
  | 2024-02-01T16:02:01.196+00:00 | min: ......................................................................... 27333.9
  | 2024-02-01T16:02:01.196+00:00 | max: ......................................................................... 27333.9
  | 2024-02-01T16:02:01.196+00:00 | mean: ........................................................................ 27333.9
  | 2024-02-01T16:02:01.196+00:00 | median: ...................................................................... 27181.5
  | 2024-02-01T16:02:01.196+00:00 | p95: ......................................................................... 27181.5
  | 2024-02-01T16:02:01.196+00:00 | p99: ......................................................................... 27181.5
  | 2024-02-01T16:29:23.146+00:00 | cleanup called, signal:
  | 2024-02-01T16:29:23.146+00:00 | TERM
  | 2024-02-01T16:29:23.146+00:00 | Interrupted with TERM, stopping
  | 2024-02-01T16:29:23.146+00:00 | /artillery/loadgen-worker: line 420: wait: pid 341 is not a child of this shell
  | 2024-02-01T16:29:23.146+00:00 | cleanup called, signal:
  | 2024-02-01T16:29:23.146+00:00 | EXIT

As a separate thing (I'm not sure if you want a different discussion or not,

How can I define phase(s) so that my VUs ramp up by no more than 2 per second overall up to a 25 VU limit per container, and stay at 25 for a given period of time, I'm having difficulty defining this without ending up with a sudden burst of users - I am working round a known limit of 2 users per second for sign-in right now) at the moment I have

  phases:
  - duration: 100 # careful this is the duration for creating new users and not the duration of the test
    arrivalCount: 20 # number to add in period 20 users = 80% CPU (16 processor) used
    name: arrival  # we can't add more than 2 per second and have to account for total number of test boxes (5) could chunk more
  - duration: 600
    arrivalRate: 1
    maxVusers: 20
    name: steadyState

WHich causes a burst of extra users when the second phase starts, also the second phase seem to cause a lot of skipped users

[hassy]

@zoe-jobson-IL would you be able to send me the full CloudWatch log for a failing worker on hveldstra@artillery.io? It seems that the worker encounters some sort of an error earlier and then the error handling code errors out itself here: https://github.com/artilleryio/artillery/blob/main/packages/artillery/lib/platform/aws-ecs/worker/loadgen-worker#L416-L424

On the ramp behavior: there's no way in Artillery right now to say "move on to the next phase when a condition is met" (in your case the condition is 25 active VUs). The number of active VUs at any given point in time will depend on the duration of the session of each of those VUs. For example if each VU takes exactly 2 seconds, you will never reach 25 active VUs with an arrivalRate of 2/second:

second 0 - test starts
second 1 - +2 VUs
second 2 - +2 VUs, the first 2 still running = 4 active
second 3 - +2 VUs, the first 2 VUs now done, the second 2 VUs still running = 4 active
second 4 - +2 and -2 again, and so on

Using maxVusers is the right way to build up to some number of active VUs, but the arrivalRate will need to be tweaked depending on how long your VUs take on average.

[zoe-jobson-IL]

Sent.

[zoe-jobson-IL]

The trouble I am having with arrival rate is that I can't see how to set it to less than 1 per second, which means that when I have more than a couple of containers running the login endpoint is likely to get slammed.

[robin-miro]

I enabled DEBUG=* from Fargate with Playwright because I met the same issue.

The Fargate worker will exit with Code 6, then the Artillery CLI will also gonna be exit with the code 6(CLI_ERROR), this gives very incorrect information.

One of the reasons is we haven't detached/try-catch the listeners such as our customised page.on() etc before we end the test, so the listeners are still running even Artillery worker finishes the job.

Here is my example: page.close() is called by the Artillery worker already, but my listener is still running

From the code, the artillery playwright will call page.close() from

artillery/packages/artillery-engine-playwright/index.js

Line 250 in 6e56af4

await page.close();

so it means that we must remove/try-catch all Listeners from tests before the end of test, to avoid the listener is still running before Worker called page.close()

[hassy]

thanks for looking into it @robin-miro!

We're propagating exit codes from workers to the CLI on purpose so that there's some indication of what went wrong, but there's lots of room to improve error reporting in cases like this.

On the issue with custom page.on() listeners - I wonder if we should call page.removeAllListeners() before calling page.close() to prevent this kind of an exception. Another option could be to allow the Playwright test function to close the page rather than do it by default in Artillery. Do you have any thoughts?