Added support for process fork to mview indexation

[MateuszMesek]

Description (*)

Added support for process fork to MView indexation

Related Pull Requests

#36155

Fixed Issues (if relevant)

1. ChangelogBatchWalker does not work in multiple threads #38246

Manual testing scenarios (*)

1. Prepare MView action and use Magento\Indexer\Model\ProcessManager inside
2. Set environment variable MAGE_INDEXER_THREADS_COUNT=2
3. Run indexer_update_all_views cron job

Contribution checklist (*)

• Pull request has a meaningful description of its purpose
• All commits are accompanied by meaningful commit messages
• All new or changed code is covered with unit/integration tests (if applicable)
• README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
• All automated tests passed successfully (all builds are green)

[m2-assistant]

Hi @MateuszMesek. Thank you for your contribution!
Here are some useful tips on how you can test your changes using Magento test environment.

Add the comment under your pull request to deploy test or vanilla Magento instance:

• @magento give me test instance - deploy test instance based on PR changes
• @magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

• @magento run all tests - run or re-run all required tests against the PR changes
• @magento run <test-build(s)> - run or re-run specific test build(s)
  For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names.

Allowed build names are:

1. Database Compare
2. Functional Tests CE
3. Functional Tests EE
4. Functional Tests B2B
5. Integration Tests
6. Magento Health Index
7. Sample Data Tests CE
8. Sample Data Tests EE
9. Sample Data Tests B2B
10. Static Tests
11. Unit Tests
12. WebAPI Tests
13. Semantic Version Checker

You can find more information about the builds here
ℹ️ Run only required test builds during development. Run all test builds before sending your pull request for review.

---

For more details, review the Code Contributions documentation.
Join Magento Community Engineering Slack and ask your questions in #github channel.

[MateuszMesek]

@magento run all tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

[MateuszMesek]

@magento run all tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

[andrewbess]

Hello @MateuszMesek
Thank you for your contribution
Fixes look nice for me
Approved

[engcom-Charlie]

@magento run all tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

[engcom-Hotel]

@magento run all tests

[engcom-Hotel]

Hello @MateuszMesek,

Thanks for the collaboration!

❌ QA not Passed

We are unable to reproduce the issue in the latest development branch 2.4-develop. Please refer to the below steps we have followed to reproduce the original issue:

1. Set up a vanilla instance with the latest development branch i.e. 2.4-develop.

2. Added MAGE_INDEXER_THREADS_COUNT and set its value as 4
   

3. Run bin/magento setup:upgrade

4. Add products in the instance:
   

5. Start MySQL show full processlist with the interval of every 1 second with the command below to monitor the queries:
   mysqladmin --verbose -u root -p -i 1 processlist

6. Run cron via the below command:
   bin/magento cron:run --group=index

But the process went well and we do not see any Delete query in the process list.

Please let us know if we missed anything to reproduce the issue.

Thanks

[engcom-Hotel]

Hello @MateuszMesek,

Have you had a chance to check this comment, #38248 (comment)?

Thanks

[MateuszMesek]

Hello @engcom-Hotel

This issue occurs only when Mview action use Magento\Indexer\Model\ProcessManager.
Please check Manual testing scenarios in my first comment, #38248 (comment)

This issue can be reproduce only by prepare custom code, you can't reproduce in vanilla instance.

[engcom-Hotel]

Hello @MateuszMesek,

Thanks for the reply!

We have tried with custom mview action using Magento\Indexer\Model\ProcessManager, but the issue is still not reproducible for us.

Please find attached the module for reference:

Magz.zip

Let us know if we have missed anything.

Thanks

[MateuszMesek]

Hello @engcom-Hotel

I updated code of your module to allow test our issue.

Magz.zip

My steps to replicate issue are:

• Install module
• Enable module
  ./bin/magento mod:ena Magz_Merchandizing
• Enable schedule mode of new indexer
  ./bin/magento indexer:set-mode schedule merchandizing_popular
• Add changes to changelog table
  INSERT INTO merchandizing_popular_order_cl (entity_id) VALUES (1), (2), (3), (4), (5), (6), (7), (8);
• Run indexer in multi threads mode
  MAGE_INDEXER_THREADS_COUNT=4 ./bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1

[engcom-Hotel]

Hello @MateuszMesek,

Thanks for the updated code, but the issue is still not reproducible for us with the latest development branch i.e. 2.4-develop. I request you to please try to reproduce the issue in the latest development branch and let us know if this is still reproducible for you.

Thanks

[engcom-Hotel]

Hello @MateuszMesek,

I trust you've reviewed this comment. We are still unable to reproduce the issue with the latest development branch i.e. 2.4-develop. Kindly reconsider this and inform us if anything has been overlooked.

Thanks

[MateuszMesek]

Hello @engcom-Hotel

Yes, I can still reproduce this issue on latest commit (c971859) in development branch 2.4-develop.

We have modified code of Magz_Merchandizing module to display processed ids in output in case when we call cron job.

On my local environment in case when I add list of ids to changelog table (INSERT INTO merchandizing_popular_order_cl (entity_id) VALUES (1), (2), (3), (4), (5), (6), (7), (8);)
And next when I call cron job then on output I see:

MAGE_INDEXER_THREADS_COUNT=4 ./bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1

1
Ran jobs by schedule.

And inside var/log/cron.log file:

[2024-03-19T08:05:01.374918+00:00] main.CRITICAL: PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'magento2.merchandizing_popular_order_cl__tmp65f9472d57b710_49733668' doesn't exist in /var/www/html/magento2/lib/internal/Magento/Framework/DB/Statement/Pdo/Mysql.php:90

Please remember about have installed pcntl module in PHP.
You can check if correct method was used by Magento\Indexer\Model\ProcessManager by add var_dump(__METHOD__) inside simpleThreadExecute and multiThreadsExecute methods of this class.
Output example in my local environment with this var_dump is:

MAGE_INDEXER_THREADS_COUNT=4 ./bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1
string(57) "Magento\Indexer\Model\ProcessManager::multiThreadsExecute"

1
Ran jobs by schedule.

[engcom-Hotel]

Hello @MateuszMesek,

Appreciate your response!

Before I proceed, could you please resolve the conflicts? Once that's done, we can move forward with the next steps.

Thanks

[MateuszMesek]

@magento run all tests

[MateuszMesek]

@engcom-Hotel done

[engcom-Hotel]

Thanks @MateuszMesek for the reply!

We are able to reproduce the issue on the development branch.

✔️ QA Passed

Added support for process fork to MView indexation

Manual testing scenario:

• Install module
• Enable module
  ./bin/magento mod:ena Magz_Merchandizing
• Enable schedule mode of new indexer
  ./bin/magento indexer:set-mode schedule merchandizing_popular
• Add changes to changelog table
  INSERT INTO merchandizing_popular_order_cl (entity_id) VALUES (1), (2), (3), (4), (5), (6), (7), (8);
• Run indexer in multi threads mode
  MAGE_INDEXER_THREADS_COUNT=4 ./bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1

Actual Result: ✔️
Running on several threads, the indexer works.
Or to be more precise:
Running on several threads, the ChangelogBatchWalker doesn't delete the table used by other threads.

After: ✔️

Before: ✖️

Tested all the manual scenarios, no impact on regression testing.

[engcom-Hotel]

@magento run Functional Tests B2B, Functional Tests EE, WebAPI Tests

[engcom-Hotel]

@magento run Functional Tests B2B, Functional Tests EE, WebAPI Tests

[engcom-Hotel]

@magento run Functional Tests B2B, Functional Tests EE, WebAPI Tests

[engcom-Hotel]

@magento run Functional Tests EE

[engcom-Hotel]

Failed tests seem flaky to me. Please refer to the below screenshots for reference:

WebAPI Tests

_Test Run 1_

Test Run 2

Functional Tests B2B

_Test Run 1_

Test Run 2

Hence moving the PR into merge in progress.