-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Very high consumption of storage for cvat_cvat_cache_db #7968
Comments
If you accidentally get 100% disk space usage, you may have big problems with the instance.
I agree, that size has to be limited somehow, but it should be configurable. Now keys are from this database are removed after 24 hours. You may try to decrease the value. |
@bsekachev agree on the warning and that it is necessary, i would definitely retain the warning to avoid big problems. |
Line 543 in 483502a
|
@bsekachev after making the required changes, what steps do i need to take to build and deploy it back? And what are the data that is stored in cache, does it affect annotations in any way? |
Not, it is only temporary cache to store ready chunks with images.
|
@bsekachev I made the changes and when trying to build get the error shown below. Could you please guide me on next steps? |
@bsekachev could you please guide me on the next steps. |
@bsekachev please use a latest docker and docker compose plugin instead of docker-compose v1 |
@subhamagrawal1 FYI |
Actions before raising this issue
Steps to Reproduce
When running cvat on a self-hosted format on ec2 instance with 32gb ram and 250gb storage, the docker logs of cvat_server show high usage of disk > 90.9% and keeps crashing the server, the screenshot below shows the error on docker logs.
The below shows my df - h output for storage size
Docker stats screenshot
On stopping the containers around 15gb of storage gets freed
On deleting the cache db around 35Gb of storage gets free
After starting the containers back up, the storage is at good levels, on using the app for 1 - 2 days and doing anntotaions same issue repeats
Expected Behavior
The containers don't crash at 90.9% warning and notify of storage issue, cache_db should not consume for e.g more than 20Gb of storage.
Possible Solution
No response
Context
No response
Environment
The text was updated successfully, but these errors were encountered: