2gb limit

[cgrozemuller]

I’ve formatted and mounted a 16gb sd card with the following commands:

./lfs --format /dev/sdb
./lfs /dev/sdb sd

Then I attempt to copy a big file to the sd card:

pv bigfile.zip > sd/bigfile.zip
Which results in:

1,97GiB 0:09:52 [5,44MiB/s] [==================================================================================>                                                                           ] 53% ETA 0:08:33
pv: write failed: Operation not permitted

Also writing multiple smaller files beyond 2^31 - 1 bytes fails. Is there a limit to the amount of data that can be written to the filesystem, using little-fs fuse?

Interestingly our microcontroller is able to write past this limit, but we can’t read the files beyond the 2gb limit with little-fs fuse.

edit: I'm on the master branch

[geky]

Hi @cgrozemuller, thanks for opening the issue, sorry about the delay. It looks like this was an oversight on my part.

littlefs uses 32-bit sizes for files, so it has a theoretical limit of 4GiB. However, several APIs pass this around as a signed 32-bit value, so some APIs break with more than 2GiB. File read/writes in FUSE end up calling several different functions, unlike the simple functions in the C library.

Since this filesystem is intended for small microcontrollers, I think the best option is to just limit files to 2 GiB and change the APIs to error appropriately. You should be able to split large files into multiple 2GiB files. Unfortunately, pushing this limit past 2GiB complicates several of the APIs, and pushing it past 4GiB would require littlefs pass around 64-bit numbers, which would significantly impact code size. Thoughts?

If there's important info in this disk image let me know. You may be able to extract it by changing the lfs_fuse_read function to not report errors returned from seek (these would be unintentionally casted 32-bit numbers) here. If that doesn't work you may have to write a simple C program to directly read the data without calling advanced APIs (seek/tell/size).

[geky]

Added 2GiB limit over here: littlefs-project/littlefs#112

Thinking about it more, we could add support for 4GiB files with a more complex seek and a lfs_file_stat function. However I'm not sure if it's worth it.

[cgrozemuller]

Thanks for your reply! Unfortunately our problem is more tricky than a single file of 2GiB. We have multiple smaller files adding up to > 2GiB on an sd card formatted with littlefs. When we try to read beyond 2GiB we will get an error. Also, when we try to fill our sd card with smaller files, writing will fail at 2GiB. In our case, support for reading small files beyond 2GiB would be more important than reading big files.

[geky]

Oh that should not be breaking. littlefs should only be limited to 2TiB (or a bit more if you increase the block size). Do you mean something like 4000 x 1MiB files?

[geky]

Look like I can reproduce this. Though I'm not sure what could be causing this to fail after 2GiBs. Interestingly enough littlefs seems to be able to write >2GiB after remounting, so it may be an issue in the allocator or in the fuse layer. I'll see what else I can figure out.

[geky]

Found it: #15

Was a simple mistake in the FUSE disk layer. Let me know if that works for you guys.