Very high consumption of storage for cvat_cvat_cache_db

[subhamagrawal1]

Actions before raising this issue

• I searched the existing issues and did not find anything similar.
• I read/searched the docs

Steps to Reproduce

When running cvat on a self-hosted format on ec2 instance with 32gb ram and 250gb storage, the docker logs of cvat_server show high usage of disk > 90.9% and keeps crashing the server, the screenshot below shows the error on docker logs.

The below shows my df - h output for storage size

Docker stats screenshot

On stopping the containers around 15gb of storage gets freed

On deleting the cache db around 35Gb of storage gets free

After starting the containers back up, the storage is at good levels, on using the app for 1 - 2 days and doing anntotaions same issue repeats

Expected Behavior

The containers don't crash at 90.9% warning and notify of storage issue, cache_db should not consume for e.g more than 20Gb of storage.

Possible Solution

No response

Context

No response

Environment

- cvat_version: v2.14.0
- docker version: 20.10.22
- environment : AWS t3.2xlarge
- Linux: Ubuntu 18.04

[bsekachev]

The containers don't crash at 90.9% warning and notify of storage issue

If you accidentally get 100% disk space usage, you may have big problems with the instance.
So, just warning will not work. You may disable this check if you believe it is not necessary for you. Please, refer to: #7180

cache_db should not consume for e.g more than 20Gb of storage.

I agree, that size has to be limited somehow, but it should be configurable. Now keys are from this database are removed after 24 hours. You may try to decrease the value.

[subhamagrawal1]

@bsekachev agree on the warning and that it is necessary, i would definitely retain the warning to avoid big problems.
Regarding cache db settings, could you guide me on how to decrease the time value for the keys?
Thanks a ton for the quick response!

[bsekachev]

cvat/cvat/settings/base.py

Line 543 in 483502a

'TIMEOUT' : 3600 * 24, # 1 day

[subhamagrawal1]

@bsekachev after making the required changes, what steps do i need to take to build and deploy it back? And what are the data that is stored in cache, does it affect annotations in any way?

[bsekachev]

does it affect annotations in any way

Not, it is only temporary cache to store ready chunks with images.

docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build

[subhamagrawal1]

@bsekachev I made the changes and when trying to build get the error shown below. Could you please guide me on next steps?

[subhamagrawal1]

@bsekachev could you please guide me on the next steps.

[azhavoro]

@bsekachev please use a latest docker and docker compose plugin instead of docker-compose v1

[bsekachev]

@subhamagrawal1 FYI