[Issue] Avoids endless loop of indexers being marked as invalid

[m2-assistant]

This issue is automatically created based on existing pull request: #29196: Avoids endless loop of indexers being marked as invalid

Magento 2.4-develop

Description (*)

This was discovered on a particular project where we noticed that the indexers catalogrule_rule and catalog_category_product are constantly being marked as Reindex required.

The cronjob indexer_reindex_all_invalid is running and is supposed to fix this, but that doesn't happen.
If however, I stop the cronjobs, and manually execute bin/magento indexer:reindex catalogrule_rule catalog_category_product we see a different behavior and only catalog_category_product is then marked as Reindex required.. If I then execute either bin/magento indexer:reindex catalog_category_product or let the cronjob indexer_reindex_all_invalid take over again, everything is getting back to a stable state and all indexers are marked as Ready.

So this means that the execution of the indexer_reindex_all_invalid cronjob and bin/magento indexer:reindex {all-invalid-indexers} are behaving differently which isn't correct in my opinion.

We've seen this happening with Magento OS 2.3.5, and I was able to reproduce it on the 2.4-develop branch (using commit f39a2e0)

I currently only have a way to reproduce it by using the database of this particular project and we also need a particular module from Amasty installed as well (the Improved Layered Navigation module version 2.14.3).
But in my opinion it's not a bug in the Amasty module but a problem in core Magento since we see a difference in behavior in how invalid indexers are reindexed if you do it manually or if the cronjob executes it.

After searching around a bit I noticed a particular commit which was only applied to the manual reindexing command: MAGETWO-51540: Linked indexers catalog_category_product and catalog_product_cateogry do not set valid status after shell run.

So I now introduced these same changes to the code which is used by the indexer_reindex_all_invalid cronjob and after testing these changes on both our project (on Magento 2.3.5) as on my test on the 2.4-develop branch, it looks like it fixes the issue.

In my opinion, this is a pretty important bugfix, because this bug causes the server of this particular project to have a high CPU load because of the constant reindexing which happens with every cron run which is not necessary.

In my opinion, we should also try to refactor the code so both bin/magento indexer:reindex as the cronjob indexer_reindex_all_invalid should use the same code so we can avoid in the future where one fix is applied to one part and forgotten about in the other part.
I've currently chosen not to refactor this so it's easier to review these changes, and maybe somebody can follow up in a later PR to refactor this?

Related Pull Requests

Fixed Issues (if relevant)

1. None that I could find

Manual testing scenarios (*)

1. Have magerun2 available to manually execute cronjobs (https://files.magerun.net/)
2. Setup a clean Magento
3. Install Amasty's Improved Layered Navigation module version 2.14.3
4. Use a specific database from our project (sorry, I can't publicly share this database)
5. Run bin/magento setup:upgrade
6. Run bin/magento indexer:reindex so all indexers are marked as valid
7. Invalidate the two specific 2 indexers: bin/magento indexer:reset catalogrule_rule catalog_category_product
8. Look at the indexers statuses: bin/magento indexer:status (the two indexers should be marked as invalid)
9. Run php n98-magerun2.phar sys:cron:run indexer_reindex_all_invalid
10. Look at the indexers statuses: bin/magento indexer:status (the two indexers are still marked as invalid)
11. Run php n98-magerun2.phar sys:cron:run indexer_reindex_all_invalid
12. Look at the indexers statuses: bin/magento indexer:status (the two indexers are still marked as invalid)
13. Run bin/magento indexer:reindex catalogrule_rule catalog_category_product
14. Look at the indexers statuses: bin/magento indexer:status (only one indexers is marked as invalid: catalog_category_product)
15. Run php n98-magerun2.phar sys:cron:run indexer_reindex_all_invalid
16. Look at the indexers statuses: bin/magento indexer:status (All indexers are now finally marked as valid)

With this PR applied, after step 8, we can jump immediately to step 13. This is probably still not 100% fixing the problem since I would expect all indexers to be marked as valid after this, but maybe that Amasty module is intervening here, not exactly sure without more deeper debugging.

If somebody needs the database from this project, I'm willing to provide it to you in private via Slack for example, but it can contain confidential information, so please I'm asking internal Magento devs to not abuse the information in the database.

Thanks!

Questions or comments

Maybe a reproducible case is written down in internal ticket MAGETWO-51540 which you can try to reproduce using the indexer_reindex_all_invalid cronjob?

Contribution checklist (*)

• Pull request has a meaningful description of its purpose
• All commits are accompanied by meaningful commit messages
• All new or changed code is covered with unit/integration tests (if applicable)
• All automated tests passed successfully (all builds are green)

[magento-engcom-team]

✅ Confirmed by @engcom-Alfa
Thank you for verifying the issue. Based on the provided information internal tickets MC-38495 were created

Issue Available: @engcom-Alfa, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

[magento-engcom-team]

Hi @m2-assistant[bot]. Thank you for your report.
The issue has been fixed in #29196 by @hostep in 2.4-develop branch
Related commit(s):

• f0c645307e031f9c6b8beb31f18c801dc62c02e3
• 07d4ac67ce54e3b4be771f526975819d1d907d8b
• e9359f1ce92b61890df259114325e575c6dd02dd
• 6a58f7ef0cf75b830f7a7c462d8fa7d010de317e
• bc2aab7c2efec20cfdc83c62765da5c380dcb39e
• f9f3725e1326a375cecbd8fd1c70125d35adeaba

The fix will be available with the upcoming 2.4.2 release.