Redis / AOF - Magento 2.4.6-p3

[Nuranto]

Summary

With redis for session, configured with appendonly=yes (AOF), we noticed that after upgrading from 2.4.6-p2 to 2.4.6-p3, storage was increasing very very fast (More than 5Go used for 300Mo of actual data). Our redis servers were crashing because of that (volume getting full within 24hours...). We guess the session are updated more frequently than before.

A hotfix was to switch to RDB.

I'm not sure it is related to 2.4.6-p3, could be another composer dependency upgrade. But I wanted to let you know, maybe others have experienced this issue too ?

Examples

Proposed solution

No response

Release note

No response

Triage and priority

• Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[m2-assistant]

Hi @Nuranto. Thank you for your report.
To speed up processing of this issue, make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, Add a comment to the issue:

• @magento give me 2.4-develop instance - upcoming 2.4.x release
• For more details, review the Magento Contributor Assistant documentation.
• Add a comment to assign the issue: @magento I am working on this
• To learn more about issue processing workflow, refer to the Code Contributions.

---

Join Magento Community Engineering Slack and ask your questions in #github channel.
⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.
🕙 You can find the schedule on the Magento Community Calendar page.
📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

[Nuranto]

Update : We still have issues with RDB. So the issue could be the number of sessions that increased, or the size of them. We'll continue investigate.

[ananth-iyer]

@Nuranto it could be related to Magento CSP:
Please check: #29964

[Nuranto]

I'm not sure, this ticket seems related to cache. We only have issues with sessions.

[hostep]

Not many session related changes at first sight: https://github.com/magento/magento2/compare/2.4.6-p2..2.4.6-p3

Maybe one of the recent changes in these 3 repositories causes it?

• https://github.com/colinmollenhour/php-redis-session-abstract
• https://github.com/colinmollenhour/Cm_Cache_Backend_Redis
• https://github.com/colinmollenhour/credis

[Nuranto]

Yes, that was my thought too.
But you're right, redis dependencies have been updated at the same time, I'll look into it, thanks.

[Garde85]

I have a similar issue. After upgrade Magento from 2.3.5 to 2.4.6-p2, Redis memory start to grow very very fast and arrived to 11 GB.
In my case memory saturation seems not to be related to session, because I use 2 different Redis instances: one for sessions and one for caches. Instance that manage sessions does not have any problems, while instance that manage backend cache saturate his memory.
I observer that the isseu seems to be related to 2 type of magento cache:

• layout
• block_html

if i flush theese caches, memory size of redis return to (around) 300MB.

[m2-assistant]

Hi @engcom-November. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
• 3. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
• 4. Verify that the issue is reproducible on 2.4-develop branch
  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
• 5. Add label Issue: Confirmed once verification is complete.
• 6. Make sure that automatic system confirms that report has been added to the backlog.

[engcom-November]

Hello @Nuranto,

Thank you for the report and collaboration!

We configured redis for session with readonly, and upgraded from 2.4.6-p2 to the latest 2.4.7-beta2.
But we didn't see any significant increase in redis size.
Please take a look at the screenshot:
This is the reddis size with 2.4.6-p2

And this is when upgraded to 2.4.7-beta2

Please let us know if we are missing anything.

Thank you.

[JoostWan]

Same problem over here will multiple shops and Redis page cache.

[goivvy]

@Nuranto Were you able to fix it? What version of redis are you running, was it upgraded along with Magento?

[amcguireweb]

We're currently updating to 2.4.6-p3 in hopes to solve our Redis issues, this does not sound promising. Has anyone determined the issue yet?

[Nuranto]

Hi @goivvy, no we did not upgrade redis along with magento. We're running on redis 7.0.14. And we stabilized this by disabling AOF

[hostep]

I just noticed this commit made in core magento that limits 2 of the redis packages to a certain version (according to the commit message - in broken English - it sounds like newer versions of these packages have performance issues): 9369884

[engcom-November]

@Nuranto, @hostep we were not able to reproduce this issue on vanilla 2.4-develop instance with redis AOF turned on.
Are there any additional things required in order to reproduce this issue, for example should we have a large number of data in magento instance or large number of traffic is required?
could you please provide more information on this.

Thank you.

[amcguireweb]

@Nuranto, @hostep we were not able to reproduce this issue on vanilla 2.4-develop instance with redis AOF turned on. Are there any additional things required in order to reproduce this issue, for example should we have a large number of data in magento instance or large number of traffic is required? could you please provide more information on this.

Thank you.

We're all experiencing the issue. It's not new to the most recent version, it's been a problem through all past versions. I am guessing you installed vanilla magento, with no data, and no traffic? You'll need a large catalog and simulated traffic to reproduce this issue.

[kestraly]

Currently experiencing this issue on 3 different setups.
I've switched over to RDB, which has indeed solved much of the issue. But I noticed Redis continued to spike. All 3 versions are on Magento 2.4.5 and all 3 versions had "Cache User Defined Attributes" - YES.
Cache EAV types and attributes - whilst enabled seems to grow without limitation. Whilst disabled, the Redis database stabilises

[github-jira-sync-bot]

✅ Jira issue https://jira.corp.adobe.com/browse/AC-13309 is successfully created for this GitHub issue.

[m2-assistant]

✅ Confirmed by @engcom-Hotel. Thank you for verifying the issue.
Issue Available: @engcom-Hotel, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

[engcom-Hotel]

Hello,

As mentioned here #38148 (comment), the issue is unable to reproduce easily, I think we can confirm this issue as so many users are facing the same issue.

We will investigate more on this during the fixing of this issue. We will keep you posted on this.

Thanks

[TalesDuque]

Hi everyone,

I recently faced the same issue with a very large catalog, with a total of 2+ million products, meaning a lot of PDP layouts.

We came to the conclusion that the ID and SKU specific PDP Layouts were generating individual cache entries for each individual product, which lead to a huge amount of cache keys in Redis after people visited different product pages.

After some time, we were looking at around 5+ million cache keys stored in REDIS, which was causing the issues we saw previously.

We then implemented a solution to disable the feature that allows the individual layouts by ID and SKU, and we're now sitting at only 35k cache keys stored in REDIS after around 15h after the deployment and after the cache was cleared. It is a significant reduction in the amount of cached keys, reducing by a ton the amount of memory used.

This check was done by going into the REDIS CLI client and typing in MEMORY STATS . This returns a "keys.count" array key with the amount of keys used in REDIS.

As a comparison, I ran the check on a smaller website, which has around 90k products, and got the result of around 100k keys. Keep in mind that in this smaller website's case, the SKU/ID specific layouts are still enabled.

The solution consists of implementing plugins on:

Magento\Framework\View\Model\Layout\Merge
and
Magento\Framework\View\EntitySpecificHandlesList

These plugins prevent the addition of the 'CATALOG_PRODUCT_VIEW_ID' and 'CATALOG_PRODUCT_VIEW_SKU' to the layout, further preventing the generation of these individual layout keys in REDIS.

Of course, our solution implies that you cannot create ID/SKU specific layout files by specifying the catalog_product_view_id/sku_... layout, but I'd say it's still recommended to create a custom product type and use it for the purpose of customizing certain PDPs, and it would also make the layout reusable in future products.

[hostep]

Even though my comment isn't session related but cache related, maybe others may find this info useful.

For the people having this Redis issue, can you try to disable Lua mode in the Redis configuration for your default cache in your app/etc/env.php file and see if that improves things?

    'cache' => [
        'frontend' => [
            'default' => [
                'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
                'backend_options' => [
                    'use_lua' => '0',                        <=====

Lua was enabled by default in colinmollenhour/cache-backend-redis version 1.17.0 which was released a few weeks before this ticket got created.

We've seen significant improvement in Redis usage by disabling Lua.

See this related discussion about it btw: colinmollenhour/Cm_Cache_Backend_Redis#181

---

For the issue @TalesDuque mentions, there is this module that does the same: https://github.com/Vendic/module-optimize-cache-size
And also we have this PR that tries to bring in configuration options to enable/disable those in core-magento: #39718