[Support]: Frigate fills the drive, without deleting old recordings #21778
Replies: 4 comments 29 replies
|
You've identified a real limitation in how Frigate manages storage. Frigate tracks recordings based on its database metadata, not actual filesystem usage(1)(2). This means old recordings that exist on disk but aren't in Frigate's database won't be cleaned up automatically. Current BehaviorFrigate includes an emergency cleanup mechanism: when there's less than an hour of storage remaining, it deletes the oldest 2 hours of recordings(1). However, this only applies to recordings Frigate knows about in its database. The storage display in the UI shows what Frigate tracks in its database (16.64 TiB used out of 18.99 TiB total), which doesn't account for orphaned files from outside Frigate's tracking system(1). What Likely HappenedYour year-old recordings suggest either:
Recommended Actions
record:
sync_recordings: TrueThis performs a two-way sync between the recordings database and disk on startup and once daily(3).
Network Storage ConsiderationsSince you're using NFS, ensure your network connection is robust (Gigabit Ethernet minimum) and the share can handle continuous write loads(4). Some users have found NFS performs better than SMB for recordings(2). 📚 Sources: Have another question? Just tag @inkeep. |
|
Sync recordings was false, thank you. But that doesn't solve the issue that Frigate reports fake information in the UI, and it doesn't take the expected action to keep itself functional. |
|
We have seen this occasionally in the past when mounts are setup incorrectly leading to the wrong size information being reported. Frigate uses a basic python utility to check the free space of volumes |
|
I am confused on the request here, the main card in the storage page already shows the breakdown of free and used storage. The card in your screenshot is just a general overview of what storage Frigate's known recordings are using, with the breakdown below. To explain the data sources: We can get the overall free / used / total stats of a volume, but we can't verify what % of that volume is used by recordings because that requires spinning up the HDDs. The area in your screenshot is specifically showing what storage is used by Frigate's recordings according to the DB. We can not make this more accurate due to the reasons stated above. The main storage card on the storage page, which is a larger breakdown of the volume for recordings, is calculated using this same overall volume data by looking at the recordings in the DB for each camera, subtracting that total space from the used space of the volume to get "other", and then using the overall volume data for "free". For the emergency cleanup Frigate uses the same python util, it does not use the DB in any way for this check, so that is not related. |
|
It might be quicker to sketch my suggestion. 
Include the total used space on this gauge. I think you're saying you already have enough information to do this. |
|
Right, but that information is already shown below, in a way that has more room to be clear about what it means. |
|
It's a misleading UI element, it clearly shows "you have plenty of free space" when you have none. You have to click on the (!) symbol, or read the manual, to find out otherwise, which is not good design. You've confirmed that the fix is easy, but you don't want to do it, so my questions have been answered. |
|
I understand the information is not at the top, however
Is false. You can just look directly below where it is clear how much free space there is. I don't understand why you're suggesting this information is not on the page. It's easy to make the change you're suggesting but it has the likelihood of being just as confusing as there's not enough space to clearly show what everything means. We originally had it more succinct and we got a lot more feedback that it was confusing. So far this is the main negative feedback we've had on this page as it is now. In any case, thanks for the feedback. |
|
I now suspect the issue has something to do with me trialling 0.17.0 Beta 2. It was working fine on 0.16.3 for an extended period, even with those orphaned recordings. When the beta failed, I downgraded to confirm the issue was still present in stable, and I saw that the UI issue was present, and none of the expected functions worked. Perhaps the stable release was unable to recover from the broken state that it was now in? Or it needed more time to begin deleting recordings? |
|
I'm not saying this is what happened, but I wonder if Frigate is holding onto a stale estimate of the total size of the volume? NFS shares appear to shrink, as space fills up on the same volume, but outside of the share. So if the algorithm, that decides when to delete old recordings, holds an out of date total size, it might not realise that the new size is basically equal to the used space. |
|
These shares are separate ZFS datasets, on the same storage pool. Notice how the free space is equal, but the size (reported by |
|
The logic is here, it does a fresh check every time https://github.com/blakeblackshear/frigate/blob/dev/frigate%2Fstorage.py#L98-L109 |
Uh oh!
There was an error while loading. Please reload this page.
Checklist
Describe the problem you are having
I know that Frigate only tracks its own recordings, and it doesn't know about other files on the same drive. But hear me out.
Frigate failed because the drive was full. The drive is an NFS share that's only used by Frigate's recordings. But, there were old recordings from a year ago, well outside of the retention policy.
Expected behaviour: Frigate cleans up old recordings to avoid this. (At least, the ones that it does know about.)
Actual behaviour: Frigate stops recording, fills up its own cache, and then stops detecting. It silently loses all functionality.
I suggest that Frigate needs to check on the actual drive free space when deciding if it should clean up recordings. The UI could report both the usage that Frigate tracks, as well as the free space reported by the filesystem, in a manner that allows the user to detect when the drive has unexpectedly gained a lot of files that Frigate doesn't know about.
By reporting plenty of free space in the Frigate UI, the user is tricked into not checking whether the filesystem agrees.
Steps to reproduce
Version
0.16.3-9034454
In which browser(s) are you experiencing the issue with?
Chromium
Frigate config file
Not given.docker-compose file or Docker CLI command
Not given.Relevant Frigate log output
Relevant go2rtc log output
Operating system
Debian
Install method
Docker Compose
Network connection
Wired
Camera make and model
Not relevant.
Screenshots of the Frigate UI's System metrics pages
Any other information that may be helpful
No response
All reactions