Implement a max jobs per worker budget

[staabm]

implements rectorphp/rector#8192

[staabm]

will be AFK for a few hours

[staabm]

reduced the jobsize, which leads to less memory used

[samsonasik]

config/config.php need to be updated as well:

rector-src/config/config.php

Line 16 in 038f809

$rectorConfig->parallel(120, 16, 20);

could you also update the different between $jobSize vs MAX_CHUNKS_PER_WORKER ? Thank you.

[staabm]

I dropped the args in this file, as these just reflect the defaults and therefore don't need to be kept in sync

could you also update the different between $jobSize vs MAX_CHUNKS_PER_WORKER ?

where?

[samsonasik]

here seems ok

↓

#4965 (review)

[TomasVotruba]

Looks good 👍

[staabm]

renaming because the previous name suggested it would be a single job

[staabm]

with the parameter adjustments at this point we can run rector across mautic with

    $rectorConfig->paths([
        __DIR__.'/app/',
        __DIR__.'/plugins',
    ]);

with a peak of 14.75 GB. it never grows higher then that:

time php ../rector-src/bin/rector.php --dry-run --no-progress-bar --no-diffs --clear-cache
                                               
 [OK] 10 files would have changed (dry-run) by Rector                                                                   
                                                                                                                       
php ../rector-src/bin/rector.php --dry-run --no-progress-bar --no-diffs   379.85s user 11.00s system 798% cpu 48.939 total

---

with rector 0.18.2 it peaks at 16.5 GB RAM and takes

➜  mautic git:(5.x) ✗ time php ../rector-src/bin/rector.php --dry-run --no-progress-bar --no-diffs --clear-cache
                                                                                                                        
 [OK] 30 files would have changed (dry-run) by Rector                                                                                                                                                                                      

php ../rector-src/bin/rector.php --dry-run --no-progress-bar --no-diffs   516.54s user 8.36s system 766% cpu 1:08.51 total

=> with this PR (and all the already merged optimizations) we are 20 seconds faster and take ~1.75 GB less memory

[samsonasik]

it is a bit ambiguous the different between $jobSize vs MAX_CHUNKS_PER_WORKER, could you write a comment here? Thank you.

[samsonasik]

@TomasVotruba let's quick merge and revert in case of it doesn't work :)

Thank you @staabm

[samsonasik]

@staabm I am testing in CodeIgniter 4 project, when we define:

$rectorConfig->parallel(240, 8, 1);

the procss got 2 times slower, but when we define default value:

$rectorConfig->parallel();

the performance seems equals, I mention you in PR test at:

• [Rector] Testing Rector dev-main codeigniter4/CodeIgniter4#7918

so in case other CodeIgniter4 project maintainer has different use case, they can give you feedback faster :)

[staabm]

the procss got 2 times slower,

thats expected. the smaller the jobsize is and the smaller the chunks-per-worker is, the slower the rector job will get.
but it also uses less memory.

the higher the jobsize and the higher the chunks-per-worker are, rector gets faster but requires more memory.

its a balance between memory and speed. before this PR all default settings were set for maximum speed but without limits for memory. thats the actual problem the bigger the project gets we analyze.

with this PR we have a mean to cut the memory peaks.

performance and memory also depends whether rector just scans the codebase and does not need todo anything or whether it needs to refactor something

[staabm]

@Flyingmana @yguedidi you might be interessted in testing rector@dev-main which now uses process re-spawning to recover from memory leaks and memory spikes