Detaching category from product causes massive product url regeneration

[bartoszkubicki]

Preconditions

1. Magento 2.2.5
2. PHP 7.1

Steps to reproduce

1. Have large catalog - I tested it against medium profile
2. Open product form of any product
3. Detach category, which has a lot of product (possibly some low level one, inheriting straight from root)
4. Save product

Expected result

1. Product saved in some fait time, max a few seconds.

Actual result

1. Products saving about 3-4 minutes.

Actually I have debug it and this is trace to track the problem
https://github.com/magento/magento2/blob/2.2/app/code/Magento/Catalog/Controller/Adminhtml/Product/Save.php:113
https://github.com/magento/magento2/blob/2.2/app/code/Magento/Catalog/Model/CategoryLinkManagement.php:assignProductToCategories()
https://github.com/magento/magento2/blob/2.2/app/code/Magento/Catalog/Model/CategoryLinkRepository.php:deleteByIds() -> which triggers event catalog_category_save_after
Observer https://github.com/magento/magento2/blob/2.2/app/code/Magento/CatalogUrlRewrite/Observer/CategoryProcessUrlRewriteSavingObserver.php - lines 112-3 triggers regeneration of all saved category's products. For example for first level category of catalog having 30k products it can be 5-10k products.

Whole process is completely pointless - removing category from one product triggers massive url rewrites regeneration, which is processed during http request.

[magento-engcom-team]

Hi @bartoszkubicki. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento-engcom-team give me $VERSION instance

where $VERSION is version tags (starting from 2.2.0+) or develop branches (for example: 2.3-develop).
For more details, please, review the Magento Contributor Assistant documentation.

@bartoszkubicki do you confirm that you was able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• yes
• no

[magento-engcom-team]

Hi @engcom-backlog-nazar. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label G1 Passed will be added to the issue automatically. Please, edit issue description if needed, until label G1 Passed appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add G2 Passed label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Verify that the issue is reproducible on 2.2-develop branch.

  Details

  - Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

• 6. Add label acknowledged once verification is complete.

• 7. Make sure that automatic system confirms that report is acknowledged.

[jakoblundsted]

Same issue, if you change something on a category (fx. meta description) with a good amount of products on a storeview.

[bartoszkubicki]

@magento-engcom-team If you tell me how to fix it I could prepare pull request. I need information how te rearrange events or observers, because maybe sometimes this operation should be done (i mean regeneration of product urls). Maybe if we can do it in 2.3 we could put url regeneration on queue?

Anyway, I think we should remove all-product regeneration from category save and add that observer maybe to more accurate operations and events.

[hostep]

@bartoszkubicki: great idea!
And maybe put it behind an option: 'async url rewrite generation enabled': yes/no, because shops with a small catalog don't really need this I think.
Then we will also finally have an official way to re-generate url rewrites, to correct already incorrectly generated ones (which still happens when having multiple storeviews with different url_key values over storeviews).

It would also be great if Magento devs could shed some light on plans in that direction, maybe this is already been worked on internally, but we don't know it yet...

[kandy]

@hostep, the main problem with async url rewrite is it cannot resolve conflict immediately and as a result, you will gets inconsistency and some products/categories will not have valid url

[kandy]

Also, you can use patch that change the way how url rewrites is generated

[hostep]

@kandy: makes sense, but the async process could then skip the product/categories with conflicting values.

Next to that, a grid in the backend of Magento could be added to give a very user friendly overview of all conflicting url_key values of products/categories, and could allow the shop owner to manually correct those one by one.

A feature like that would be awesome I think.

[magento-engcom-team]

@engcom-backlog-nazar Thank you for verifying the issue. Based on the provided information internal tickets MAGETWO-95176, MAGETWO-95177 were created

[pemann]

The fact that some non-related actions will change a product's url without creating a valid url redirect is a major(ish) issue. Having 404 pages as a search result will be punished by google and thus will reduce our sales.

[nickpiro]

I'm having an issue where we reorganized categories and now when we export products, the URL_KEY column is blank on most products. Is this related you think?

[smcpaiva]

I wonder how some big projects with Mangrove 2 have a good experience. And what ability to actually work with a large number of products and categories in the open source version.
In my case there are 25 thousand categories in 5 levels. And no product. And to create, edit or delete a product, in the admin, it is about 4 minutes of delay.
And in the end it has generated the error "No data to save"
But if I delete or reduce the categories it works again and decreases this time.
I have already tested different server configurations and magento versions :(

[cpartica]

hi @bartoszkubicki we're about to do this in 2.3.1
Can you provide more exact steps about what "Detaching category" means? So our quality engineers can understand it.
What level of category do you try to detach? Do you go through product page? do you go through category page, do you do this through api? All those cases?

[bartoszkubicki]

@cpartica I think all question you've asked have been already resolved while describing the issue:

• I think level doesn't matter as long as this category has a lot of products (thousands to see the effect)
• I do it on product page
• I have it tested only on product page, but I believe it can be a problem throughut other entry points, as it is caused by massive rewrites operations, which are triggered on low-level category event (catalog_category_save_after).

[nthurston]

I also have huge load times when trying to attach a product to a category through API on 2.2.6.

I have a category with a few thousand products and it takes nearly 90s to add another product through the category link repository API (POST to catalogCategoryLinkRepositoryV1).

[magento-engcom-team]

Hi @Jakhotiya. Thank you for working on this issue.
Looks like this issue is already verified and confirmed. But if your want to validate it one more time, please, go though the following instruction:

• 1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 2. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 3. Verify that the issue is reproducible on 2.2-develop branch.

  Details

  - Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

• 4. If the issue is not relevant or is not reproducible any more, feel free to close it.

[Jakhotiya]

@cpartica Are you working on this? I was thinking, if it is possible to leverage "Use Categories Path for Product URLs" setting. If it's set to No, can we skip URL rewrite generation for products when they are detached from a category??

While looking into the default url_rewrite table I saw that no matter what the configuration is "category path product URL's" are always generated. This also leads to unnecessarily large url_rewrite table. This table is optimised for read operations but as a result of using unique indexes "INSERT" queries become slow.
Is there way to respect "Use Categories Path for Product URLs" setting?

[hostep]

@Jakhotiya: that's not a good idea with the current state of Magento where there is no official way to regenerate url rewrites, please see my comment in: #15086 (comment)

[sdzhepa]

Hello @bartoszkubicki @jakoblundsted @pemann @Jakhotiya @hostep

Thank you for contribution and collaboration!

The corresponding internal ticket MAGETWO-95176 was fixed, delivered and closed by Magento team

Please see details in the next commits:

• cf316b4
• ee86554