Magento Cache Grows Uncontrollably

[Shimon2]

The Magento cache on our site keeps growing.

We Flush the Magento Cache. This brings disk usage to about 30.88GB (29%)
We warm the page cache with a utility that goes through the entire sitemap and warms the cache. This is done to improve the responsiveness of the site. This brings disk usage to about 50.7 GB (48)%.
The disk usage continues to climb at about 12 GB (10%) / day.

This means that if we do not flush the cache every 4-5 days we run out of disk space and the site chokes.

Preconditions

Magento 2.2.2.
PHP 7.0.30
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.5.1804 (Core)
Release: 7.5.1804
Codename: Core

Steps to reproduce

1. enter df command - see that the disk is very full (80% or so_
2. clear magento cache - see that the disk is at 30.88GB 29%
3. warm cache with all pages. - see that the disk is at 50.7 GB 48%

4. wait 2-3 hours - see that the disk is at 53.3 GB 49%
5. repeat step 4 and see that the disk usage continues to climb

Expected result

disk usage should plateau at some point.

Actual result

1. Disk usage will grow to 100%, choking the site

[magento-engcom-team]

Hi @Shimon2. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento-engcom-team give me {$VERSION} instance

where {$VERSION} is version tags (starting from 2.2.0+) or develop branches (2.2-develop +).
For more details, please, review the Magento Contributor Assistant documentation.

@Shimon2 do you confirm that you was able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• yes
• no

[orlangur]

@Shimon2 please find out on your side which caches are growing. Also, 29-48-80% are not any useful numbers, always operate MB values.

We warm the page cache with a utility that goes through the entire sitemap and warms the cache

Why not use Varnish? Using filesystem Full Page Cache is not a recommended approach for production and obviously there is no mechanism which would control the total size of stored cache.

@Shimon2, thank you for your report.
We were not able to reproduce this issue by following the steps you provided. If you'd like to update it, please reopen the issue.

[hostep]

I've seen this behaviour a couple of times (very sporadic), but haven't found time yet to investigate, easiest solution is to just flush the cache, but that's not really the correct solution.

@orlangur:

Why not use Varnish? Using filesystem Full Page Cache is not a recommended approach for production and obviously there is no mechanism which would control the total size of stored cache.

Not everyone will be able to use Varnish unfortunately.
Maybe some kind of configurable limit should be added to Magento to keep the cache size under control somehow? I think that's how Varnish works as well.

[orlangur]

Maybe some kind of configurable limit should be added to Magento to keep the cache size under control somehow? I think that's how Varnish works as well.

In Varnish this is builtin, for Magento it would be a feature request. I don't think such mechanism should be a part of Magento at all, as to me having some script monitoring disk usage and removing oldest cache entries is much more reliable.