UTF-8 NFD file name on SMB storage cannot be accessed

[PVince81]

Steps

1. Setup a SMB server on Linux
2. Configure a share "shared"
3. Directly inside the share's folder on disk (not going through the network), create a file "good-ümlaut.txt"
4. Create another file "bad-ümlaut.txt"
5. Convert the file name encoding to NFD: convmv -r -f utf8 -t utf8 --nfd --notest bad-ümlaut.txt (ref: http://laufer.tumblr.com/post/1066670961/solved-weird-problem-with-special-characters-in).
6. Setup an external storage in OC and mount that share as "/smb"
7. Navigate into the share in the web UI

Expected result

File appears in list

Actual result

File not visible

{"reqId":"KAz39vteBSxGCMxr1TEG","remoteAddr":"127.0.0.1","app":"OC\\Files\\Cache\\Scanner","message":"!!! Path 'encoding\/bad-u\u0308mlaut.txt' is not accessible or present !!!","level":0,"time":"2015-12-24T09:58:29+00:00","method":"GET","url":"\/owncloud\/index.php\/apps\/files\/ajax\/list.php?dir=%2Fsmb%2Fencoding&sort=name&sortdirection=asc"}

Versions

Observed in OC 8.2.2

[PVince81]

Had a chat with @icewind1991 and he told me that the file names are normalized by OC.
So if the remote SMB server contains a file with a file with a "mac" umlaut, it will normalize the name and store the normalized name in the file cache. Then later when it wants to download the file, it would use the normalized name. So obviously that name doesn't exist on the SMB storage => not found.

We cannot disable normalization of file names because it will break syncing on Mac OS then.

Ideal would be if the SMB server itself would have a setting that tells it to normalize UTF-8 names...

So far the only known workaround is to use convmv recursively to fix the file names on the remote share.

@icewind1991 @DeepDiver1975 @butonic won't fix?
CC @cmonteroluque

[PVince81]

Another idea would be to have ownCloud detect such files and automatically normalize the file names on the remote storage, but that would be a bit "intrusive" and it doesn't feel like it's OC's job to do that.

[PVince81]

More info about NFD vs NFC:

• https://en.wikipedia.org/wiki/Filename#Unicode_interoperability

[PVince81]

Another possibly, very ugly and with possible performance impact: whenever we have a UTF-8 file name and connect to the remote SMB, try with both the NFC and NFD file name encoding. If one matches, then assume it's the same file.
But then what happens if the SMB storage has two files, one with each encoding ? 💥

[PVince81]

I just tried locally and it is indeed possible to have both files with the same name and different encoding:

-rw-r--r-- 1 vincent nobody 18 Dec 24 10:51 bad-ümlaut.txt
-rw-r--r-- 1 vincent nobody 18 Dec 24 11:24 bad-ümlaut.txt
-rw-r--r-- 1 vincent nobody 32 Dec 24 10:51 good-ümlaut.txt

My font represent them slightly differently:

[PVince81]

We could also reopen the discussion about adding a new column to oc_filecache to store the real file name (maybe call it storage_name).
Let me find the old ticket about it.

[PVince81]

also CC @nickvergessen who worked on this idea back then

[PVince81]

Somehow can't find an exact ticket.
Found some bits here:

• Discussion about case sensitive remote FS: Better handling for case-insensitive file systems (ext storage) #17161 (comment) referring to https://github.com/owncloud/core/wiki/Cross-Platform-File-Handling#platform-name
• similar issue but involving the client: Can't detect existing file #12044

[PVince81]

• the PR that removed the file_mapper table with bits of the extra column topic: Remove file mapper - was only use in Windows and never worked properly #17379 (comment)

Setting to 9.0 for now to keep in view.

[karlitschek]

@DeepDiver1975 FYI For the WND discussions