Fargate - arrivalCount seems doesn't work as expected

[robin-miro]

Version info:

Artillery: 2.0.5
Node.js:   v21.6.1
OS:        darwin

Running this command:

config:
      phases:
        - duration: 50m
          arrivalCount: 300

run-fargate --count = 10

I expected to see this happen:
Based on the doc, it should mean creating 300 VU in 3000 seconds, which means creating 1 VU per 10 seconds around, as I said fargate count equals 10, then means 10 VU per 10 seconds around in total.

Instead, this happened:

not enough VUs are created, instead of 10 VU per 10 seconds, it randomly just creates 2 VU per 10 seconds. Those missing VUs are also Not failed.

Additional notice
When I switch to 50m with 600 VUs, meaning 1 VU per 5 seconds, it becomes a bit better.

Why this is important
When we generate a latency benchmark instead of a Load/Stress test, we expect the load can be similar and with a small and stable load, meaning the VUs are roughly can have the same amount, and help to compare the latency benchmarks from different regions.

Idea
Because it's hard to check all workers from CloudWatch, as It's hard to figure out which Fargate/Worker doesn't generate VUs, maybe it gonna be a bit easier to verify it with Artillery Cloud.

[robin-miro]

More findings:

• I checked the summary vu.completed from each fargate, every thing is 100% correct, the number is expected

Meaning, the issue is happened from Fargate -> SQS -> AWS CLI, however I'm not sure how to check more details

[hassy]

thanks @robin-miro! will try to replicate here

[hassy]

btw @robin-miro feel free to drop me a line on hveldstra@artillery.io for an Artillery Cloud invite, would love for you to try it

[robin-miro]

Hey @hassy , thanks for sharing, I will drop you an email to have a look for collaboration for using the cloud.

JFYI: I used this workaround and resolved my issue, and there is no blocker anymore to generate a fixed amount of users in a short period. I can get a very stable load now.

Just share some very interesting findings maybe you can reproduce in your test environment.

Case 1 :
10 Fargates
same scenario
same AWS region
config:

      phases:
        - duration: 10m 
          arrivalCount: 120 

Results:

vusers.completed: .............................................................. 376
vusers.created: ................................................................ 376
vusers.failed: ................................................................. 0

Case 2 :
10 Fargates
same scenario
same AWS region
config:

      phases:
        - duration: 60 
          arrivalCount: 12 # per fargate, 5 sec will have a new virtual user
        - duration: 60 
          arrivalCount: 12 # per fargate, 5 sec will have a new virtual user
        - duration: 60 
          arrivalCount: 12 # per fargate, 5 sec will have a new virtual user
        - duration: 60 
          arrivalCount: 12 # per fargate, 5 sec will have a new virtual user
        - duration: 60 
          arrivalCount: 12 # per fargate, 5 sec will have a new virtual user
        - duration: 60 
          arrivalCount: 12 # per fargate, 5 sec will have a new virtual user
        - duration: 60 
          arrivalCount: 12 # per fargate, 5 sec will have a new virtual user
        - duration: 60 
          arrivalCount: 12 # per fargate, 5 sec will have a new virtual user
        - duration: 60 
          arrivalCount: 12 # per fargate, 5 sec will have a new virtual user
        - duration: 60 
          arrivalCount: 12 # per fargate, 5 sec will have a new virtual user

Results:

vusers.completed: .............................................................. 1200
vusers.created: ................................................................ 1200
vusers.failed: ................................................................. 0

[hassy]

I'm not able to reproduce this on v2.0.5 yet. I'm wondering if it's a bug in the built-in HTML reporter (it hasn't been updated in a very long time).

Here's a couple of reports from tests I ran:

1. 5m duration, arrivalCount of 30, with 10 Fargate workers. 10 VUs are created every 10s as expected (purple line in the first chart on the page): https://app.artillery.io/share/sh_241f1dfb796c17b0da86b205284bef26c5a992cdb3fc2195239821f98a0dbc13
2. 10m duration, arrivalCount 120, 10 workers. 20 VUs are created as expected every 10s: https://app.artillery.io/share/sh_62eb4e34813fc67f04a4ffbfa001d130db82e20d92e09790713bb21151188c3a

If you could share the text log from one of the runs that didn't produce the expected number of VUs that would be really useful. Thank you!

[hassy]

Going to close this for now, but @robin-miro feel free to reopen if you run into the issue again.