slowness test

[oulfr]

Q	A
ParaTest version	6.6.5
PHP version	8.0

Summary

I have a lot of unit tests that use a real database for testing each test is isolated and can run in a separate thread, the sequential run takes 30 minutes, and with the option : -f -p32 it take 20-18 minutes.

My configuration is like that:

• 4 cores
• 8 logique processors

Current behavior

Knowing that one processor can run a lot of threads, it will be better if paratest use thread in place of processor, I don't know if it's a limitation of paratest or PHP language, but what I know is in other programation language we can run a lot of thread even if we have just one processor.

Expected behavior

Run each test in separate thread not a processor

[mfn]

Are you sharing anything on the database side between the parallel runs, like same database connection/tables?

[Slamdunk]

As far as I know, in PHP the user (as well as this library) can't handle processes nor CPU threads, the engine does it all.

By the way there's something fundamentally wrong in your setup if you get those results.

In many mainstream computers I've worked with, going above 12 or 16 processes in Paratest the result doesn't scale anymore, because the bootstrap operations become too dominant.

But in every well coded test suite, and with --runner WrapperRunner enabled, there is a inverse proportion between nproc and elapsed time.

With --runner WrapperRunner --processes 8 you must have a run that ends in less than 5 minutes if the sequential run takes 30 minutes, otherwise your DB setup is likely the culprit

[oulfr]

@Slamdunk: I don't think it's a DB setup, even if it's the case with multi-threading the time must decrease when we run a lot of thread 50 or 100.

This is how my test works:

For each test :-

1. I create a new schema, then a fresh database I'm in the multi tenancy project,
2. Inside of each test I have many requests to mysql database- to check data- (not in memory db)
3. At the end, I delete the schema
4. This scenario is for each test.

The problem here is why paratest is limited to process not thread

[Slamdunk]

The problem here is why paratest is limited to process not thread

If going from -p1 to -p2 doesn't exactly half the execution time, paratest cannot be the culprit, and neither can be processes/threads when you have more than one CPU.

Creating and deleting a db is extremely time consuming, try a table-truncation-teardown

[Jean85]

Creating and deleting a db is extremely time consuming, try a table-truncation-teardown

+1 on this, the culprit is totally your database: creating/deleting a schema is not an operation built for concurrency, so it probably has a lot of locks that make the execution of your tests nearly sequential, even when using Paratest. It also forces the DB to flush to the disk, so you're an order of magnitude slower.

[oulfr]

@Jean85 and @Slamdunk : thank you for your response i will do a search in mysql create/delete schema and concurrency.
But what I still not understood is: I have 8 logical processors and each processor can handle a lot of threads, but paratest use processors to handle parallel testing with -p isn't that? is a limitaion of php ?

[Jean85]

What we're saying is that from within PHP, we cannot directly govern how the CPU is consumed: Paratest just launches multiple PHPUnit processes, and leaves to the OS the job of actually leveraging all the available CPUs. So, if your tests aren't constrained by something else, you should see your CPU consumption increase with direct correlation to the number of parallel tests running.

In a normal situation, you should see a 100% CPU consumption; in your case, we just suspect that the DB is acting as a bottleneck, stopping tests from proceeding at full speed.

[oulfr]

@Jean85 : thank you, yes the issue of my test is a lot of create delete schemas for each test and not related to paratest.

Thank you all.

[oulfr]

Ok found a solution before each create/delete:

SET unique_checks=0; SET foreign_key_checks=0; SET GLOBAL innodb_stats_on_metadata=0;

and after:
SET unique_checks=1; SET foreign_key_checks=1; SET GLOBAL innodb_stats_on_metadata=1;