-
Notifications
You must be signed in to change notification settings - Fork 9.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
catalog_product_index_price reindexing occasionally resulting in missing products on site #35616
Comments
Hi @rob-gonz. Thank you for your report.
Make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, Add a comment to the issue:
For more details, review the Magento Contributor Assistant documentation. Add a comment to assign the issue: To learn more about issue processing workflow, refer to the Code Contributions.
🕙 You can find the schedule on the Magento Community Calendar page. 📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket. ✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel |
Please note. The documentation states that catalog_product_index_price using table switching (https://developer.adobe.com/commerce/php/development/components/indexing/optimization/). It does not use table switching when the indexer is running off of the cronjob. We have confirmed this - maybe it does on occasion. Typically it does not. |
Hi @engcom-Delta. Thank you for working on this issue.
|
@rob-gonz We are experiencing some very similar issues on 2.4.3-p2 with a product database of 30K products Thx!! |
Hi @engcom-November. Thank you for working on this issue.
|
@engcom-November This is a scale problem with the M2 handling of catalog_product_index_price indexer. You are not testing the scenario described. As stated above, the number of products you have used to test this scenario is significantly insufficient. You have not replicated the scenario I described and that is why you were not able to reproduce the issue. The speed at which the database updates is beyond the capability of a human to run a select search at the cmd line for catalog_product_index_price. The 462 rows is complete in ~ 1 second, if not much less, once the query is made. Even then, the insert command used will cause table locking for a majority of the request because the table has to lock while building its index making it even harder for your Select to detect an issue at such a small count. We are experiencing this on a catalog_product_index_price index table with > 16 million rows, with the default batch size of 10k. You will only experience the issue if you have more than 10k products. Even then, you will only be able to detect the issue if you have more than probably 50k products. Preferably, you should test with a minimum of 100k products in the catalog_product_index_price. Even then a script will be necessary to pull the count(*) of catalog_product_index_price 2 to 3 times per second. Then it will become obvious what we are describing. At the size of our catalog_product_index_price it is much easier to observe. For example: When you have 24/7 traffic and are constantly running automated price, inventory, and content updates the indexer is always reindexing something. Meaning there is a significantly higher chance for the collection to not get the proper products it would normally. This is what we are experiencing. |
@duckchip, glad to hear others pay attention to indexers as much as we do. We do not currently have a fix for this. We know what needs to be fixed, but we are looking to Adobe to fix the issue because they are making significant database design changes in the coming versions. It's also a core issue they need to address. I've heard that a 2.4.4+ update will have a direct effect on this indexer table. |
@engcom-November, This is affecting users' experience. When can we expect a fix? |
Thank you @rob-gonz for sharing this. We have similar symptoms in one of our shops and I think we are experiencing the same issue here. I want to share some of my findings related to this issue. TLDR: As already mentioned Occasionally, for no reason we have some categories in frontend showing the message: "We can't find products matching the selection.", although there are products in these categories. After clearing the cache, everything is fine again. There was not way to reproduce this, until I found this issue and gave it another try. I created a test script to reproduce this issue. I will attach it here. Let me explain, how it works:
You will see the count changing during reindex process. It is only a small time window but in some scenarios it may be enough to encounter empty categories in frontend. I think we are encountering this issue so often because we are doing a lot of price updates in this shop which leads to many price index updates. Digging into deeper into the code, I think the root cause is inside If I remove this function call everything seems to be fine and the issue does not appear anymore. I'm aware that this is not a proper solution but I will try it as a workaround because some orphaned entries inside I can confirm this issue on a Mageto 2.4.4 installation. I did not test latest 2.4-develop, but I expect the issue to be still present because code of |
@bka, thank you for the work you did to provide even more evidence of the issue and provide detailed findings here for the community. Cheers! We'll see what we can come up with when we explore the issue again. |
Very well-known issue, sorry I haven't noticed that before. You can easily fix it with the patch from @magento/creativestyle team: |
@lbajsarowicz thanks for that hint |
Honestly this is just the tip of the iceberg, most of the indexes have the same behavior (lack of transaction), there are much more patches we created for similar issues as yours: https://github.com/magesuite/magento-patches/blob/master/essential/magento/module-inventory-indexer/wrap-inventory-indexer-with-transaction.patch |
Great info here! Why aren't these fixes being contributed back as pull requests to Magento? (I understand the PR process is a big pain in the ass here, but at least it would make the fixes more visible to the entire community) |
To be honest, nobody from our side had time to create PRs taking into account how long the process takes. Since we are open source with our patches we are open for anyone to take those changes and prepare proper PRs based on them. |
Hi @engcom-Echo. Thank you for working on this issue.
|
I've noticed 2 commits that got merged here 11 hours ago, under the name "ACP2E-1028: Current Indexer implementation for large catalogs is causing items to not appear on site", that sounds related to this issue report. Unfortunately there is no clean merge commit, it's part of a huge merge commit with a bunch of different tickets, so here are the individual commits: Not sure if these will fix the problem, but the description sounds related... |
Hello @rob-gonz, Thanks for raising your concern and make patience. We have tried to reproduce the issue in Magento 2.4-develop branch, but for us, the issue is not reproducible. We have followed the below steps:
As per your suggestion we have check on more than 10 k product but we are not getting the any differences. Please find the below screen shot for your references. Screen for count of catalog_product_index_price and catalog_product_index_price_replica table. Please check if i missed anything and kindly check on Magento 2.4-develop branch and provide more information with exact steps and screenshot so that we can reproduce this issue. Thanks |
@engcom-Echo , I appreciate you are still trying to reproduce - I really do. However, you failed to follow the steps I outlined. If they are not followed exactly you can not reproduce the issue. The initial reindex is NOT the problem. It is once a reindex is occurring based on a cronjob. The issue also only occurs during the reindex NOT after. Running the reindex directly runs a reindex that generates the replica then swaps the tables. The cronjob reindexes for this table does it in place.
With only 40k products you are still going to be challenged to see it by hand. It can not be done after a reindex. It has to be done during a reindex to see the problem. Running a script will greatly increase your chances of reproducing it. You still may have to try a time or two. Maybe add multiple storefronts and associate the products so you can multiply the number of products in the index. I could reproduce it by hand because my catalog_product_price_index table has 15+ million rows. I had over an hour in which I could reproduce the issue. Meaning there was over an hour in which items on my site could be queried, perceived to not exist, and thus be cached as if they did not exist for many hours after resulting in lost sales. If it helps you at all this: https://github.com/magesuite/magento-patches/blob/9271a0fe59ca008c04ad1b73b6581a1d4582754e/essential/magento/module-catalog/m242-wrap-price-indexer-with-transaction.patch#L1-L79 posted by @lbajsarowicz actually fixes the issue. |
Hi @engcom-Hotel. Thank you for working on this issue.
|
Hello @rob-gonz, Thanks for the detailed steps for issue reproduction. However, we have tried to reproduce the issue in the latest 2.4-develop branch but it seems that the flow is working as expected. We have a script as stated by you in this #35616 (comment). I am attaching the same below for reference: I printed the logs when we have the count of both the tables (catalog_product_index_price_replica, catalog_product_index_price) not equal. and we have observed that the count is not matching for some point of time because Magento tries to copy data from catalog_product_index_price to catalog_product_index_price_replica table. Each time 60000 records have been inserted. Please refer to the below log file for reference: We have tried it with almost 58000 records: I think we are missing some steps. Thanks |
Dear @rob-gonz, We have noticed that this issue has not been updated for a period of 14 Days. Hence we assume that this issue is fixed now, so we are closing it. Please raise a fresh ticket or reopen this ticket if you need more assistance on this. Regards |
Preconditions and environment
Steps to reproduce
This bug is hard to reproduce when you want it to occur. It may be easier to reproduce if you were to have everything set up just perfectly. This includes scripts controlling which and how many products to pull in a product collection on a clean cache(no cache).
Expected result
catalog_product_index_price (or any table index) should NEVER have the number of items in it reduced. Item information should always be here it should never be removed at any time, for any purpose.
Actual result
What will happen is products that are removed from the catalog_product_index_price index for the purpose of reindexing will be missing from the queries/collections and they will get cached that way in magento2. If you are running a properly configured environment this will also cause the frontend categories to also be missing these products and can be cached not only in M2 but in edge caching. Moreover, when using a 3rd party search tool these products will be missing entirely from the search.
Additional information
Why is this an issue?
We have experienced occasional loss of sales due to missing products on our site. We have been tracking this issue for a bit. We couldn't believe it at first and it still feels like this is a major oversight unless we are missing something.
How did you discover the bug
We realized that when doing a manually reindex (forced) it would create a replica table and rename. That's fine - that works great. However, when reindex is running from the cronjob it is using the DELETE command to delete a batch of N products and then insert a batch of N products. This means that at any given time during indexing you could be missing up to 10,000 versions/variations/pricing of a product.
How do you expect Adobe to Fix This
Stop deleting items from this table and then inserting them. Use the 'REPLACE INTO' Query to make updates. If there are records that do need to be removed those should be removed through a separate deletion or by periodically running reindex that involves a table rename.
** Additional note**
Consider not using batch size paging that uses 'count(*)' to determine what items to page through next. When you get up beyond 8 or 10 million records in a database it slows the performance down. Use the entity_id to key off for paging it is SIGNIFICANTLY faster. We have already modified an indexer because it was taking several hours to run. We increased the speed 7x with this change.
Release note
No response
Triage and priority
The text was updated successfully, but these errors were encountered: