How to define the number of iterations for multi-threaded testing

[avoitenko]

Hello @AntyaDev
I am trying to define a number of iterations for multi-threaded testing.
My expectation is to increase the test speed when I increase the number of threads.
Please consider the code below:

`public class IterationTest
{
//--- config
public readonly int Threads = 1; // number of threads
public readonly int Iterations = 500; // number of iterations for each thread
public readonly TimeSpan Duration = TimeSpan.FromSeconds(50); // duration

public ConcurrentDictionary<int, int> threadCounter = new ConcurrentDictionary<int, int>();

//---
public void Run()
{
    //---
    var scenario = Scenario.Create("scenario1", async context =>
    {
        await Task.Delay(10);

        int threadIndex = context.ScenarioInfo.ThreadNumber;
        int iterationIndex = context.InvocationNumber;

        //--- DO TEST
        if (iterationIndex <= Iterations)
        {
            //--- do some actions
            context.Logger.Information("TEST, thread: {0}, iterations: {1}",
                threadIndex,
                iterationIndex);

            return Response.Ok();
        }


        //--- save iteration value for each thread
        if (threadCounter.ContainsKey(threadIndex))
        {
            threadCounter[threadIndex] = iterationIndex;
        }
        else
        {
            threadCounter.TryAdd(threadIndex, iterationIndex);
        }

        //--- detect to stop the sceanrio
        if (threadCounter.Count == Threads)
        {
            bool stop = true;
            foreach (var it in threadCounter)
            {
                if (it.Value < Iterations)
                    stop = false;
            }

            //---
            if (stop)
            {
                context.Logger.Information("STOP TEST");
                //--- stop
                context.StopCurrentTest("stop the " + context.ScenarioInfo.ScenarioName);
                //--- does not work
                //context.StopScenario(context.ScenarioInfo.ScenarioName, "stop the " + context.ScenarioInfo.ScenarioName);

            }
        }

        return Response.Fail();
    })
    .WithoutWarmUp()
    .WithMaxFailCount(int.MaxValue)
    .WithLoadSimulations(Simulation.KeepConstant(Threads, Duration));

    //---
    NBomberRunner
        .RegisterScenarios(scenario)
        .Run();
}

}`

When I set Threads=1, Iterations = 500, the test takes 8 seconds, request count │ all = 501, ok = 500
When I set Threads=10, Iterations = 50, the test takes 2 seconds, request count │ all = 715, ok = 500

I need your advice on how to improve its performance because we have 215 wrong iterations.
My general requirements - invoke a certain amount of Iterations for certain threads number.
Maybe you have a good solution because I trying to migrate my code from K6 to NBomber.
K6 has the native config parameter for it which is called 'iterations'.
If possible please add the parameter iterations in to WithLoadSimulations
Thanks.

[AntyaDev]

Hi @avoitenko
Could you please send me a link to k6 docs about iterations?

[AntyaDev]

One more thing, context.InvocationNumber gives you a number of invocations per ScenarioCopy/instance. It's not Global InvocationNumber for the whole test. I don't remember the reason why it's not global. Maybe it does make sense to fix it and make it global :)

[avoitenko]

@AntyaDev
Hello.
This is the link to K6 options:
https://k6.io/docs/using-k6/scenarios/executors/per-vu-iterations/
It was pretty useful for me to set the iteration number.

[avoitenko]

One more thing, context.InvocationNumber gives you a number of invocations per ScenarioCopy/instance. It's not Global InvocationNumber for the whole test. I don't remember the reason why it's not global. Maybe it does make sense to fix it and make it global :)

InvocationNumber should represent the number of iterations per thread but not the whole scenario. It works as expected.
My goal is to implement the iterations parameter per thread like in K6.
But it takes extra coding like in my example above.
Is it possible to extend options in KeepConstant simulation and add 'Iterations' parameter?

[AntyaDev]

Hi @avoitenko ,

Yes, we can add this. Sounds reasonable.

[avoitenko]

Great. Waiting for implementation.
Thanks.

[AntyaDev]

Hi @avoitenko
Any updates?

btw: if you interested we have a roadmap and we added your requests about iterations: https://nbomber.com/docs/getting-started/roadmap

[AntyaDev]

Hi @avoitenko ,
Just wanted to inform you that NBomber v5.4 changed the behavior of context.InvocationNumber - and now it gives you aggregated value and not per instance value.

Also, NBomber v5.4 added support of two new LoadSimulations:

• Iterations For Constant
• Iterations For Inject