feat: Ability to run Fixer with parallel runner 🎉 #7777

Empty for now, because logic for running fixers must be moved here. Basically it's a PoC of command being hidden - works on PHP 7.4 and 8.3 with latest Composer packages on each version. Command's arguments/options are more or less what we need in the worker, but it may be changed later.

The same collection of files has to be determined for both sequential and parallel run. See PR7777#discussion_r1526648174

It's working, but needs polishing here and there. It speeds up analysis significantly (like 5x for this codebase on my computer).

Parallel runner must be introduced slowly, and start as an opt-in. When it's stable enough, we can change it to be default one.

Using `ParallelConfig::detect()` can be used for utilising all available resources. Tested by running analysis natively on the host and inside Docker (with core limit set on the OrbStack level). In both scenarios it detects available cores correctly.

…o the error manager

Detected by Composer Require Checker

Cache check is done during file iteration - if file is in cache and wasn't changed, then it's skip by iterator and doesn't even go to the worker. That's why `ReadonlyCacheManager` used on worker's side is a bit superfluous, but I thought it's safer to use it instead of `NullCacheManager`. Anyway, each file is stored to cache when worker reports analysis result to the parallelisation operator, so it's basically similar to sequential analysis, because file write is done only in one process.

Co-authored-by: Julien Falque <julien.falque@gmail.com>

… PHPStan narrowed types

Use absolute file path for reporting analysis result and convert it to relative path when processing on parallelisation operator's side. It gives the same effect in terms of caching, but does not require `DirectoryInterface::getAbsolutePath()`.

It improves caching, because cache save can be triggered more frequently and errors or interrupting during processing large chunk won't cause analysis info being lost (which would lead to re-analysing file that was already processed).

- extract constants to dedicated class with more fitting name - use constants in every place where communication between runner and worker occurs

There were 2 problems: - when child processes were spawned, `ArgvInput` was created to pass the arguments/options to the worker, but when fix/check command was executed via command tester, `ArgvInput` was using PHPUnit's argv from top-level command (and was failing with "The "--configuration" option does not exist.") - Passing `InputInterface` instance from command to process (did an experiment with static property set in command to skip passing input through the Runner instance) was not enough because output was invalid and tests were failing. I hope we can figure out something better after discussion on code review 😉.

Some of the tests are little hacky at this point, maybe it's possible to refactor a bit to achieve better testability. Good enough for now, though.

Passing root process' input to the runner and then to `ProcessFactory` allows switching `InputInterface` implementation and test process creation.

Tested on: - PHP 8.3 - PHP 8.3 with --prefer-lowest - PHP 7.4 with --prefer-lowest

- it can be helpful when using parallel config but there's need to disable it temporarily (it's easier to add option to the command than editing config file) - it solves the internal problem with parallel runner being used for running tests that operate on command's output (parallel run produces undeterministic output because files are not processed in the strict order)

https://twitter.com/_Codito_/status/1763374111571472558 Needs more tests!

Broken during rebase.

…unix-like OS

The linting is done on the workers' side before fixers are applied, so we will collect any syntax errors anyway.

…ring value read

…command

- always expect CPU(s) usage info in the output - always use dist config for PHAR parallel test (because local config can use sequential analysis)

* syntax sugar * sequential option - no shortcut, as we do not use shortcut for other any other option * explicitly mark `ProcessFactory::getCommandArgs` as `@private`, to indicate it shall not be called from outside of this class, ref PHP-CS-Fixer#7777 (comment) * `WorkerCommand` options written as single-liners, so unified as all other commands and easier to copy-paste * add explicit comment for hardcoded option * remove unused var * rename `ParallelRunnerConfigInterface` to `ParallelAwareConfigInterface` * make `ParallelConfig` only DTO and externalize factory methods to dedicated class * `Runner` - don't reevaluate `maxFixerPerProcess` in each loop * explicitly call generator, avoid simple 'job' as too similar to job-id * rename variables related to file chunks

* explicitly crash on unexpected parallel action * explicitly crash on unexpected events count, i.e. when someone wrongly refactors EventBus * no need to calculate relative path + validate if result has proper size * introduce `ErrorsManager::reportWorkerError()`: don't report `WorkerError` via `report(Error)` method

…-in via env variable

* use cache nicely, without recalculation of hash * no need for read-only cache, anymore

…or manager - Remove invalid comment (no idea why it was here) - rename loop's variable to not confuse it with `WorkerError` class that represents unrecoverable errors on worker side

It's actually more natural to name these actions after initiating side - when we're writing to the output, we should use action related to the place where we're writing, and when we're reading the message, action should point to the callee. See: PHP-CS-Fixer#7777 (comment)

See: #5 (comment)

- handled process errors (linting, failed fixes etc.) are now collected with ~original exception as source when possible. These don't break the main process, unless `--stop-on-violation` is used, it was like that before too, but `ParallelisationException` was used for recreating error's source, which was not correct. - other errors (socket in/out errors caught by React, or unhandled errors that kill the worker) are re-thrown in the main process that effectively stop further execution

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Ability to run Fixer with parallel runner 🎉 #7777

feat: Ability to run Fixer with parallel runner 🎉 #7777

Commits on May 15, 2024