Extended message queue consumer configuration with parameters "maxIdl…

[Neos2007]

Extended message queue consumer configuration with parameters "maxIdleTime", "sleep" and "onlySpawnWhenMessageAvailable". Created logic for checking/handling these parameters during firing/executing queue consumers. Updated unit tests.

Description (*)

Current pull request extends existed message queue consumers logic and adds support of new consumers configuration parameters: "maxIdleTime", "sleep" and "onlySpawnWhenMessageAvailable".
PR was created based on "Improvements to message queue consumer processes" (Problem 1: not enough options to keep consumers under control).

Description new consumers parameters:

"maxIdleTime" - Describes max time (in seconds) of waiting for a new message from the queue.
"sleep" - Time (in seconds) to sleep before checking if a new message is available in the queue.
"onlySpawnWhenMessageAvailable" - (boolean value) identifies that consumer should be spawned only if there are available message in the related queue.

Parameters "maxIdleTime" and "sleep" are currently used only in Magento\Framework\MessageQueue\CallbackInvoker class (so they will be handled only for Consumers which was fired with defined "maxNumberOfMessages" parameter)
Parameter "onlySpawnWhenMessageAvailable" currently checks/validates via ConsumersRunner cron (\Magento\MessageQueue\Model\Cron\ConsumersRunner).

All new parameters for consumers could be configured either via queue_consumer.xml configuration files (as attributes for the consumer node) or via env.php file:

    <consumer name="consumer1" queue="queue1" handler="handlerClassOne::handlerMethodOne" consumerInstance="consumerClass1" connection="connection1" maxMessages="200" maxIdleTime="500" sleep="5" onlySpawnWhenMessageAvailable="true"/>

 'consumers' =>[
     'consumer1' => [
        'name' => 'consumer1',
        'queue' => 'queue1',
        'consumerInstance' => 'consumerClass1',
        'handlers' => [
            0 => [
                'type' => 'handlerClassOne',
                'method' => 'handlerMethodOne'
            ],
        ],
        'connection' => 'connection1',
        'maxMessages' => '200',
        'maxIdleTime' => '500',
        'sleep' => '5',
        'onlySpawnWhenMessageAvailable' => true
    ]
]

Manual testing scenarios (*)

For testing could be extended existed consumers configuration (via queue_consumer.xml or via env.php file).
Testing 'onlySpawnWhenMessageAvailable' parameter.

1. Configure 'onlySpawnWhenMessageAvailable' parameter as "true" for one of the consumers.
2. Make sure that cron run is allowed for cron consumers in env.php (parameter 'cron_consumers_runner/cron_run' should be set as true or should not be defined, parameter 'cron_consumers_runner/consumers' should be empty or include all consumers).
3. Make sure that all messages related to the consumer queue were processed and there are no new messages.
4. Disable external triggering of running Magento cron jobs.
5. Kill the PHP (if run) processes that trigger running needed consumer.
6. Run Magento cron CLI command with '--group="consumers"' parameter (php bin/magento cron:run --group="consumers).
7. Wait 15-20 seconds after executing CLI command and Check currently executed php processes. There should not be a process which runs tested consumer.
8. Publish new messages in the queue that is processed by the tested consumer.
9. Re-run magento CLI cron:run command for "consumers" group.
10. Check currently executed php processes. There should be the php process that executes (starts) your consumer.
    You can test consumers behavior with and without 'onlySpawnWhenMessageAvailable' parameter debugging process running consumers via cron (debug \Magento\MessageQueue\Model\Cron\ConsumersRunner::run() method execution).

Testing "maxIdleTime" and "sleep" methods are could be done via the debugging processing consumer process.

Questions or comments

Contribution checklist (*)

• Pull request has a meaningful description of its purpose
• All commits are accompanied by meaningful commit messages
• All new or changed code is covered with unit/integration tests (if applicable)
• All automated tests passed successfully (all builds are green)

[m2-assistant]

Hi @Neos2007. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

• @magento give me test instance - deploy test instance based on PR changes
• @magento give me 2.4-develop instance - deploy vanilla Magento instance

For more details, please, review the Magento Contributor Guide documentation.

[m2-assistant]

Hi @Neos2007, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

[ghost]

@Neos2007 unfortunately, only members of the maintainers team are allowed to assign developers to the pull request

[m2-assistant]

Hi @Neos2007, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

[m2-assistant]

Hi @Neos2007, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

[Neos2007]

@magento run all tests

[adifucan]

Hi @Neos2007! What is the status of this PR? Are you going to proceed with it?
Last builds were green. So why was this PR not merged? Now Semantic Version check fails because of updated xsd schema with new attributes. But it's not a big deal I guess.
Please, let me know the status of your PR. Thanks!

[adifucan]

Please, check my review comment about CallbackInvoker::invoke $sleep variable. No more review comments from my side.
I also tested your solution: maxIdleTime, sleep, onlySpawnWhenMessageAvailable options work fine.
I ran all tests against your code and only Semantic Version Checker fails with minor changes because of new options added to xsd schema. I guess it should be approved.
Please, let me know if you need any assistance with your PR. Thanks! Good job!

[vzabaznov]

Hi @Neos2007
Thank you for your contribution, we found this PR very helpful and we are going to proceed with it by our internal team.
Once we merge it, status of this will be changed to merged.
Thanks.

[m2-assistant]

Hi @Neos2007, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.