Add cross-file syncing between multiple open files, introduce LFS_O_SNAPSHOT #513

Compared to other filesystems, littlefs's handling of open files may come across as a bit odd, especially when you open the same file with multiple file handles. This commit addresses this by forcing all open readable file handles to be consistent if any open writable file handle is synced though either lfs_file_sync or lfs_file_close. This means open readable file handles always mirror the state of the filesystem on disk. To do this we again rely on the internal linked-list of open file handles, marking files as clean, copying over the written file cache, and synchronizing any custom attributes attached to the file handles. Note, this still needs cleanup and tests --- Why was the previous behavior? One of the nifty mechanism in littlefs is the ability to have multiple device-side copies of a file that share copy-on-write blocks of data. This is very useful for staging any amount of changes, which may live either in RAM caches or allocated-but-not-committed blocks on disk, that can be atomically updated in a single commit. After this change, littlefs still uses this update mechanism to track open files, meaning if you lose power, the entire file will revert to what was written at the last lfs_file_sync. Because this mechanism already exists, it was easy enough to rely on this to handle multiple open file handles gracefully. Each file handle gets its own copy-on-write copy of the contents at time of open, and and writes are managed independently of other open files. This behavior was idiosyncratic, but consistent, though after some time enough users raised feedback that this behavior needed to be reassessed. Now multiple open files should conform to what's found in other filesystem APIs, at a small code cost to manage syncing open files.

… dirty This is a bit of a complicated area for the custom-attribute API without much precedent. littlefs allows users to provide custom attributes in the lfs_file_config struct, which get written along with other file metadata. Sounds great on paper, but the devil is in the details. When does the metadata actually get written? What about this case? lfs_file_opencfg(lfs, file, "path", LFS_O_WRONLY, cfg_with_attrs); lfs_file_close(lfs, file); // does not write metadata This normally doesn't write out metadata! We've opened the file for writing, but made no changes, so normally littlefs doesn't bother to commit anything to disk. Before, as a courtesy, littlefs marked the file as dirty if it noticed the file was opened for writing with custom attributes, but this is inaccurate could to leave to problems after a file is synced: lfs_file_opencfg(lfs, file, "path", LFS_O_WRONLY, cfg_with_attrs); lfs_file_sync(lfs, file); change_attrs(); lfs_file_close(lfs, file); // does not write metadata Unfortunately, it isn't easy to know when metadata needs to be written. Custom attributes are provided as read-only pointers to buffers which may be updated without additional filesystem calls, this means we don't know if custom attributes have actually changed on the device side. If they haven't changed, writing out metadata on every sync would be wasteful. Another solution would be to compare our device-side attributes with the disk-side attributes every sync, but that would be even more expensive. --- So for now, the simpliest and most efficient solution wins. Custom attributes attached to open files, are not written unless the file data itself changes. Note that explicit calls to lfs_setattr always update on-disk attributes, and opening a file with LFS_O_CREATE | LFS_O_TRUNC will also always update the on-disk attributes (though not with just LFS_O_CREAT!). There are a few ways we could provide an API that manually forces a write of custom attributes, such as lfs_file_setattr, though without dynamic memory, providing these APIs gets a bit complicated. So for now we will see if users run into issues with the current scheme.

This was provided as a courtesy to hopefully make custom attributes more easy to use, however the zeroing turned out to be a bit complicated when syncing custom attributes across multiple open files. Implicitly zeroing trailing buffer space is also inconsistent with the other APIs in the filesystem, such as lfs_file_read, so this commit removes the behavior. If you need to handle differently sized custom attributes, you can either pre-zero the custom attribute buffers, or use lfs_getattr to find the on-disk size of custom attributes explicitly.

Related to changes to custom attribute and open file syncing

LFS_O_SNAPSHOT brings back some of littlefs's idiosyncratic behavior removed in the changes to open file syncing in a form that may be more useful for users. LFS_O_SNAPSHOT allows you to open a "snapshot" of a file. This is a cheap, local copy of a file who's changes are not reflected on disk. Internally, snapshot files use the same mechanism as pending writes. A separate, copy-on-write CTZ skip-list is created, with read-only references to the existing data blocks until a write occurs. The difference is that snapshot files are not enrolled in the mlist, meaning they won't get updates from open file syncs, and during close their contents are simply discarded. As an extra benefit, LFS_O_CREAT | LFS_O_SNAPSHOT is equivalent to Linux's O_TMPFILE, making it easy to create temporary, unnamed files. This may be useful for embedded development, where unnamed flash-backed buffers may provide a slower, but larger, alternative to RAM-backed buffers.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cross-file syncing between multiple open files, introduce LFS_O_SNAPSHOT #513

Add cross-file syncing between multiple open files, introduce LFS_O_SNAPSHOT #513

Commits on Dec 25, 2020

Add cross-file syncing between multiple open files, introduce LFS_O_SNAPSHOT #513

Are you sure you want to change the base?

Add cross-file syncing between multiple open files, introduce LFS_O_SNAPSHOT #513

Commits on Dec 25, 2020