alternative filename storage for very long filenames

[vgough]

Long filenames can exceed the filesystem limits after encryption & encoding. An alternative is to store filenames as file contents.

Just to revisit the current design, it is like this:

[ plain text filename ] --- encryption ---> [ encrypted blob ] --- base64 ---> [ encrypted filename ]

• For short filenames (<= 176 bytes), nothing changes
• For longer filenames, I think we should do this:

[ plain text filename ] --- encryption ---> [ encrypted blob ] --- sha256 --- hexdump ---> [ filename hash ]

The file would be called
longfn.[ filename hash ]
and there is a second file,
longfn.[ filename hash ].fullname
that stores the "encrypted blob".

I think this is the way to go. Build it into encfs instead of creating an external filesystem. This would be difficult to test and use and hence error prone and fragile.

The construction using the sha256 hash makes sure file lookup is efficient even for long file names. There should be no performance impact for directories not containing long file names. Each long file name in a folder will hit readdir performance with the penalty of reading the longfn.[ filename hash ].fullname.

Pull requests welcome!

[fulldecent]

-1 this significantly increases the cost of readdir

---

Readme.md:

Fast on classical HDDs

EncFS is typically much faster than ecryptfs for stat()-heavy workloads when the backing device is a classical hard disk. This is because ecryptfs has to to read each file header to determine the file size - EncFS does not. This is one additional seek for each stat. See PERFORMANCE.md for detailed benchmarks on HDD, SSD and ramdisk.

[vgough]

It isn't necessary to store all filenames this way, just the ones that would otherwise be too long. Ideally you would only pay a performance penalty when using long filenames, otherwise you wouldn't see a difference.

[fulldecent]

Got it. Because encrypted files may be moved across different host filesystems, "too long filesystem" must be defined independent of host filesystem. Practically, most filesystems support 255 bytes (https://en.wikipedia.org/wiki/Comparison_of_file_systems). So this issue is more specifically, "alternative filename storage for filenames over 190 characters".

[Aikhjarto]

Not only too long filenames might raise problems but also too long pathnames.
E.g., last year I had some problems with encfs in combination with rsync hosters. They supported only 255 bytes (minus the /home/$USER string) as maximum pathname.

I would really like to see an option for encfs to set maximum length of filenames and pathnames separately.

[lalomartins]

FWIW, there are two reasons I'm looking for an alternative to ecryptfs, and the top one is file name length; ecryptfs' limit (135 characters IIRC) is just not workable for 2015. (The second reason is performance on regular hard disks, which is why I ended up looking at encfs.)

[fulldecent]

For people that are following this issue, I am interested to know if it is because your file names are long or because there is a lot of folder nesting? If it is the latter then storing file name in the file wont help here.

Here is one solution:

Store naming information in a "directory file" in each directory. Example cipher text:

/index.idx
/1.dat
/2.dat
/3.dat
/4/
/4/index.idx
/4/1.dat

This solution creates a locking problem with index.idx.

[lalomartins]

It's because the filenames are long, for me at least. Some programs generate filenames automatically based on some data or another, hashes in particular, and they assume filesystems have a reasonable limit, something like 256. One such program had a bug reported on that topic and closed it with a “wontfix” since, well, if it's just one or two filesystems that have such a small limit, better that people just don't use those, eh? :-)

[vgough]

Most people have probably never had to worry about short names (8+3 anyone?), and hash-based names have become common with programs like Git, as it provides a convenient way to do content-based addressing. So I think that long names are here to stay.

One problem is that encryption expands the names, so those long names become even longer. It seems feasible to hide long names from the underlying FS without big changes to the encfs, and without impacting performance of anyone who isn't using long names. For example, handle filenames as normal unless it has a certain prefix, in which case you have to read the name from elsewhere (like the file header). Older versions of encfs would be able to read everything but the long files, as it wouldn't be able to decode those names. As @fulldecent points out, there isn't any one consistent limit to target, so this would probably be a FS config option.

Producing shorter paths would be a more major overhaul, not something I'd imagine being backward compatible with existing filesystems. So I'm not inclined to worry about path limits.

[lalomartins]

I'd be fine with this solution.

What do you wean with the config option? Minimum length to encode, or maximum to allow? I think the latter isn't necessary at all — it would be kind of ironic if after this change encfs becomes one of the few fs to allow arbitrarily long filenames and then you need to rename your files when you want to move them out 😉 And minimum length to encode can be easily autodetected on mount, based on the host filesystem limit + encoding overhead.

[vgough]

Config would be for the maximum encoded name length. Since Encfs volumes can be moved between systems, an Encfs folder could always contain longer names than the underlying storage system supports, so it's no different than what a user would already face if they tried to move files from one filesystem to one with more restrictive limits.

[dpc]

My problem is encfs produces too long filenames, and due to this I can't sync it with other hosts using tools like syncthing.

Wouldn't it be possible to have an option that would store metadata in a separate file or something? I don't care about the performance. I just want everything to work smoothly.