New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate littlefs into mbed OS #5538

Merged
merged 58 commits into from Dec 1, 2017

Conversation

@geky
Member

geky commented Nov 20, 2017

What is littlefs?

littlefs is a little fail-safe filesystem designed for embedded systems.

It comes with three key features:

  1. Bounded RAM/ROM

    The littlefs is designed to work with a limited amount of memory. Recursion is avoided and dynamic memory is limited to configurable buffers that can be provided statically.

    Here's a comparison of the memory consumption between the FAT filesystem and littlefs:

    littlefs-memory

    By just switching from FAT to littlefs you will have saved 13 KB of ROM.

  2. Power-loss resilience

    The littlefs is designed for systems that may have random power failures. The littlefs has strong copy-on-write guaruntees and storage on disk is always kept in a valid state.

    The main drawback right now of the FAT filesystem is that there is no protection against power failures. Unless you have constrained when the device could lose power or be reset, you are basically relying on luck to not end up with a corrupted filesystem.

    The funnest way to demonstrate this is with the example program that is in littlefs's README.md. Spamming the reset button with the FAT filesystem will quickly lead to problems:

    fatfs-boot-count-demo littlefs-boot-count-demo

  3. Wear leveling

    Since the most common form of embedded storage is erodible flash memories, littlefs provides a form of dynamic wear leveling for systems that can not fit a full flash translation layer.

    To continue bashing on the FAT filesystem, here's a comparison of littlefs and FAT wearing down a block device with 3KB of static data and 3KB of dynamic data:

    littlefs-wear2

    The wear leveling algorithm in littlefs is not perfect, but it does give you the very nice property that expanding the size of storage increases the lifetime of the device.

    Here's an interactive simulation where you can see how the wear leveling behaves with different configurations:
    http://littlefs.geky.net/demo.html

How does it work?

I'm already pushing the limits of what is reasonable to shove into a PR, but I would encourage you to look into littlefs's DESIGN.md if you're interested.

How do we know it works?

The littlefs has quite a few stages of testing:

  1. A large set of unit tests run in a simulated block device on Linux. This provides coverage for forseeable issues. Any bug fixes come in with a unit test to prevent regressions.
    https://github.com/geky/littlefs/tree/master/tests

  2. The littlefs-fuse is used to hook up littlefs with the kernel. A subset of the unit tests are self-hosted on littlefs itself to provide a level of fuzz testing on the functional API.

  3. A set of integration tests run on the mbed filesystem API on a device with a physical storage device.
    https://github.com/ARMmbed/mbed-littlefs/tree/master/TESTS/filesystem

  4. A set of integration tests run on the retargeted stdlib API on a device with a physical storage device.
    https://github.com/ARMmbed/mbed-littlefs/tree/master/TESTS/filesystem_retarget

  5. A set of repeatable tests are ran in a simulated mbed block device with a full file-level filesystem check on every single erase cycle. This guarantees that the filesystem operations are atomic and the filesystem is never in an invalid state. (provided by @c1728p9).
    https://github.com/ARMmbed/mbed-littlefs/tree/master/TESTS/filesystem_recovery/resilience

  6. A set of repeatable tests are ran in a simulated mbed block device with simulated wear until the block device is exhausted. The block device is increased and the test is ran a second time. The test only passes if the increased size results in a proportional increase of filesystem lifetime. This guarantees that the properties around wear-leveling hold true. (provided by @c1728p9).
    https://github.com/ARMmbed/mbed-littlefs/tree/master/TESTS/filesystem_recovery/wear_leveling

  7. A set of repeatable tests are ran on a physical storage device on an mbed board hooked up to be reseted at various intervals. This is the highest level integration test and is intended to catch anything else that we may have missed. (provided by @c1728p9).
    https://github.com/ARMmbed/mbed-littlefs/tree/master/TESTS/filesystem_recovery/resilience_functional

  8. @c1728p9 has also set up a long-running soak test using a set of repeatable tests that run indefinitely. This is intended to catch any issues that may crop up from the system being ran for a large length of time.
    https://github.com/ARMmbed/mbed-littlefs-soaktest

In total, the tests take roughly 45 minutes to run on a single device with a single type of storage device.

How do I start using littlefs?

Switching from the FAT filesystem to littlefs is a easy as changing the class name from FATFileSystem to LittleFileSystem. If you're using storage-selector, littlefs is already available as the LITTLE option.

Here's a sed script:

sed -i 's/FATFileSystem/LittleFileSystem/g' $(find -regex '.*\.\(h\|c\|hpp\|cpp\)')

When should I use littlefs vs the FAT filesystem?

At this point, the only remaining benefit to using the FAT filesystem is when the user expects to be able to plug in the storage medium to a PC. There is littlefs-fuse, which is a start at supporting the ability to mount littlefs on PCs, but it is currently limited to Linux.

What else needs to be done?

The biggest task remaining is to make sure that CI is able to run with the incoming tests. I'll be working with @studavekar to make sure the CI is set up with correct hardware.

A lower priority task is to get the incoming filesystem tests working on the FAT filesystem as well. @deepikabhavnani has put a lot of work into this so it should just be an effort of migrating her PRs on mbed-littlefs.

Additionally, there are several small patches in various places that are required to get everything working:

cc @c1728p9, @deepikabhavnani, @studavekar, @sg-, @JanneKiiskila, @marcuschangarm, @MarceloSalazar

geky added some commits Jul 12, 2017

Squashed 'littlefs/' content from commit 663e953
git-subtree-dir: littlefs
git-subtree-split: 663e953a5073ac8e459f60d18375f887f14e21a2
Tweaked littlefs to fit into mbed
- Changed log statements to use the debug function
- Changed %d to %ld given the type of int32_t in arm-none-eabi-gcc. In
  mainstream gcc this is not the case and may cause problems to
  upstream.
Squashed 'littlefs/' changes from 663e953..c2283a2
c2283a2 Extended entry tag to support attributes
8795f0e Added build size output to CI
47db7a7 Added sanity check for compiling example
476915f Removed a few "what"s from the documentation

git-subtree-dir: littlefs
git-subtree-split: c2283a2753619d82a1fdf27d799cd53f2eef9a80
Increased default lookahead to 512 blocks
Because lookahead is stored efficiently as a bit-vector, this only
requires a ram increase of 48 bytes (2.1% of benchmark), but decreases
the SD benchmark runtime cost by 32 seconds (21.9% of benchmark).

Note this is unimportant on devices with byte-reads such as NOR flash.
Squashed 'littlefs/' changes from c2283a2..9843402
9843402 Fixed incorrect return value from lfs_file_seek
273cb7c Fixed problem with lookaheads larger than block device
d9367e0 Fixed collection of multiblock directories
a83b2fe Added checks for out-of-bound seeks
a8fa5e6 Fixed some corner cases with paths
26dd49a Fixed issue with negative modulo with unaligned lookaheads
0982020 Fixed issue with cold-write after seek to block boundary

git-subtree-dir: littlefs
git-subtree-split: 984340225befc1e2330dd3b88f4048373eda0fce
Converted tests into mbed-style integration tests
- TESTS/filesystem for mbed OS filesystem APIs
- TESTS/filesystem_retarget for mbed OS retargeted stdlib APIs

converted:
- test_dirs
- test_files
- test_seek
- test_parallel
Ported lfs_util functions to IAR intrinsics
required intrinsics for:
- lfs_ctz
- lfs_npw2
Squashed 'littlefs/' changes from 9843402..454b588
454b588 Updated SPEC.md and DESIGN.md based on recent changes
f3578e3 Removed clamping to block size in ctz linked-list
83d4c61 Updated copyright
539409e Refactored deduplicate/deorphan step to single deorphan step
2936514 Added atomic move using dirty tag in entry type
ac9766e Added self-hosting fuzz test using littlefs-fuse
9db1a86 Added specification document

git-subtree-dir: littlefs
git-subtree-split: 454b588f73032f9621c264fba280ab7b3a216402
Fixed incorrect instruction in IAR ctz implementation
The RBIT instruction reverses the bits of a word, not REV
Removed toolchain specific warnings
- Comparisons with differently signed integer types
- Incorrectly signed constant
- Unreachable default returns
- Leaked uninitialized variables in relocate goto statements
Squashed 'littlefs/' changes from 454b588..2ab150c
2ab150c Removed toolchain specific warnings
0825d34 Adopted alternative implementation for lfs_ctz_index
46e22b2 Adopted lfs_ctz_index implementation using popcount
4fdca15 Slight name change with ctz skip-list functions

git-subtree-dir: littlefs
git-subtree-split: 2ab150cc500d5b8233ec8ef6109efa363bf1d38c
Added simple high-level thread safety
All calls are blocking, so a single mutex is able
to garuntee synchronization across all relevant functions.
Cleaned up compilation with logging enabled
- Removed list of warnings on signedness of integers in printf
- Fixed issue with "true" in ifdef
Fixed issue with aggressively rounding down lookahead configuration
The littlefs allows buffers to be passed statically in the case
that a system does not have a heap. Unfortunately, this means we
can't round up in the case of an unaligned lookahead buffer.

Double unfortunately, rounding down after clamping to the block device
size could result in a lookahead of zero for block devices < 32 blocks
large.

The assert in littlefs does catch this case, but rounding down prevents
support for < 32 block devices.

The solution is to simply require a 32-bit aligned buffer with an
assert. This avoids runtime problems while allowing a user to pass
in the correct buffer for < 32 block devices. Rounding up can be
handled at higher API levels.
Fixed corner case with immediate exhaustion and lookahead==block_count
The previous math for determining if we scanned all of disk wasn't set
up correctly in the lfs_mount function. If lookahead == block_count
the lfs_alloc function would think we had already searched the entire
disk.

This is only an issue if we manage to exhaust a block on the first
pass after mount, since lfs_alloc_ack resets the lookahead region
into a valid state after a succesful block allocation.
Squashed 'littlefs/' changes from 2ab150c..3f31c8c
3f31c8c Fixed corner case with immediate exhaustion and lookahead==block_count
f4aeb83 Fixed issue with aggressively rounding down lookahead configuration
db51a39 Removed stray newline in LFS_ERROR for version

git-subtree-dir: littlefs
git-subtree-split: 3f31c8cba31e0f6cef5b02dba2e050d8df1168b7
@mbed-ci

This comment has been minimized.

mbed-ci commented Nov 29, 2017

Build : SUCCESS

Build number : 622
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/5538/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build

@mbed-ci

This comment has been minimized.

@mbed-ci

This comment has been minimized.

@sg-

sg- approved these changes Nov 30, 2017

Some work to be done to readme/links in readme that point at external repos that should be shut down and pointing at the release this is part of.

The logging using LFS rather than mbed highlights we still need to get that fixed. Please talk with @SenRamakri about what you've done here and make sure it can be replaced in a compatible way (for the stack implementation)

Otherwise LGTM 👍 and remind me to chat about later about assert

@0xc0170

This comment has been minimized.

Member

0xc0170 commented Nov 30, 2017

Some work to be done to readme/links in readme that point at external repos that should be shut down and pointing at the release this is part of.

Can you please update? we will retrigger tests asap. Uvisor is currently being built, device failure should be fixed now

@geky

This comment has been minimized.

Member

geky commented Nov 30, 2017

@sg-, Can update to be self contained. Is it fine to keep the external links to the FUSE and JavaScript wrappers in the related projects section? Or should I remove that section?

@geky geky force-pushed the geky:littlefs-staging branch 2 times, most recently Nov 30, 2017

@geky geky force-pushed the geky:littlefs-staging branch to c613030 Nov 30, 2017

@geky

This comment has been minimized.

Member

geky commented Nov 30, 2017

Talked with @sg- offline, he says this is fine, so we just need one last test run for sanity

@0xc0170

This comment has been minimized.

Member

0xc0170 commented Nov 30, 2017

/morph build

@mbed-ci

This comment has been minimized.

mbed-ci commented Nov 30, 2017

Build : SUCCESS

Build number : 631
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/5538/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build

@mbed-ci

This comment has been minimized.

@mbed-ci

This comment has been minimized.

@kegilbert

This comment has been minimized.

Contributor

kegilbert commented Nov 30, 2017

/morph uvisor-test

@kegilbert

This comment has been minimized.

Contributor

kegilbert commented Nov 30, 2017

/morph echo

@mbed-bot

This comment has been minimized.

mbed-bot commented Nov 30, 2017

I recieved your comment! Here's what you wrote!

/morph echo

@0xc0170 0xc0170 added ready for merge and removed needs: CI labels Dec 1, 2017

@0xc0170 0xc0170 merged commit 2e1c2a1 into ARMmbed:master Dec 1, 2017

10 checks passed

AWS-CI uVisor Build & Test Success
Details
ci-morph-build build completed
Details
ci-morph-exporter build completed
Details
ci-morph-test test completed
Details
continuous-integration/jenkins/pr-head This commit looks good
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
travis-ci/events Local events testing has passed
Details
travis-ci/littlefs Local littlefs testing has passed
Details
travis-ci/mbed2 Local mbed2 testing has passed
Details
travis-ci/tools Local tools testing has passed
Details

@0xc0170 0xc0170 removed the ready for merge label Dec 1, 2017

@geky geky referenced this pull request Feb 23, 2018

Merged

littlefs: Update to version 1.3 #6179

6 of 6 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment