Repository navigation
Overwritten data retained after writes stop in v1.4.1 — expected behavior and a proposed fix #7700
Replies: 1 comment 1 reply
|
Thanks for the detailed report and the proposed patch. Currently, compaction and cleanup follow these paths:
In your test, the overwrites leave obsolete data without reaching the write-side compaction threshold. Since there are no subsequent reads or writes, there is no further opportunity to trigger compaction through those paths. Setting TrashDays=0 does not itself remove the old slice references. For workloads requiring reclamation after writes stop, we currently provide explicit maintenance commands: juicefs compact for a specific file or directory, or juicefs gc --compact for volume-wide compaction. gc --delete alone does not reclaim overwritten data that is still referenced by chunk metadata. One concern with the proposed approach is its impact on foreground workloads. A chunk being idle does not mean the client or storage backend is idle. Although the worker yields while other compactions are running, the additional compaction reads, writes, and deletions can still compete with application I/O during busy periods. For now, we’ll keep the current behavior and won’t adopt this change. We can revisit it if user reports show that this workload pattern is common and that manual compaction is insufficient, making automatic reclamation worth the additional overhead. |
Uh oh!
There was an error while loading. Please reload this page.
Hi, we observed that overwritten data can remain stored after file I/O stops, and would like to understand whether this is expected behavior.
Our setup uses JuiceFS v1.4.1 with TiKV for metadata and Ceph RADOS for data storage. We created 4 GiB of files, then fully overwrote the same files four times, writing 20 GiB in total. We then stopped all file reads and writes and observed storage usage for 60 minutes. TrashDays was set to 0, local caching and writeback were disabled, and normal background jobs remained enabled.
With upstream v1.4.1, the increase in Ceph pool data usage remained at 20 GiB, although the files contained only 4 GiB of current data. With the attached patch, the same test ended with an increase of 4 GiB. These figures exclude EC redundancy.
From reading the code, write-triggered compaction depends on the number of slice records in each chunk. Large overwrites can leave substantial obsolete data while the record count remains below the trigger threshold. If no further reads or writes occur, is there another background mechanism that should reclaim this data, or is retaining it an intentional trade-off to reduce compaction overhead?
The proposed patch keeps a bounded, in-memory list of chunks written by the client and checks them after a quiet period, reusing the existing compaction and deletion paths.
Reproduction scripts, original test logs, patches and regression tests are available here:
Report and supporting materials
We would appreciate feedback on both the observed behavior and the proposed approach. Even if this approach is not suitable for merging upstream, a technical review of its correctness, concurrency handling and resource costs would be very helpful. Suggestions for a better approach or an existing operational solution are also welcome.
All reactions