Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Files and directories containing : or | are ignored #854

Closed
rikvdh opened this issue Aug 11, 2013 · 65 comments
Closed

Files and directories containing : or | are ignored #854

rikvdh opened this issue Aug 11, 2013 · 65 comments

Comments

@rikvdh
Copy link

@rikvdh rikvdh commented Aug 11, 2013

Expected behaviour

Files getting synced to my ownCloud server

Actual behaviour

I see this occurring in the Linux client.
When I create files or directories containing | : > < ? or a few other characters, they are automatically ignored.

I already did some research and found this commit in csync:
http://git.csync.org/users/freitag/csync.git/commit/?id=e0807cba1b164fcf409cb71e3da2f8f7b88637a9

I don't know why this commit is there, creating files on Linux with | : < > * ? is perfectly valid. :)

total 16
drwxr-xr-x 4 rik rik 4096 Aug 11 08:32 .
drwxr-xr-x 10 rik rik 4096 Aug 10 16:11 ..
-rw-r--r-- 1 rik rik 0 Aug 11 08:22 abc*def
-rw-r--r-- 1 rik rik 0 Aug 10 16:02 abc:def
-rw-r--r-- 1 rik rik 0 Aug 11 08:21 abc>def
-rw-r--r-- 1 rik rik 0 Aug 11 08:22 abc?def
drwxr-xr-x 2 rik rik 4096 Aug 11 08:22 abc|def
drwxr-xr-x 2 rik rik 4096 Aug 10 16:08 a|c

Steps to reproduce

  1. Install Linux client
  2. Sync ownCloud
  3. Create a file with: touch 'abc|def' and see it getting ignored :)

Server configuration

not applicable, normal Debian Wheezy with apache2

Client configuration

Client version: latest from git (but also occurs with 1.3.0)
Operating system: Debian Wheezy
OS language: en_US
Installation path of client: /usr/local/bin/owncloud

Log

08-10 16:10:59:661 csync_ftw: Uniq ID from Database: test/a|c ->
08-10 16:10:59:661 csync_walker: directory: /home/rik/ownCloud/test/a|c
08-10 16:10:59:661 _csync_detect_update: test/a|c excluded (1)
08-10 16:10:59:661 _csync_detect_update: ==> file: test/a|c - hash 6527849599191992178, mtime: 1376143708
08-10 16:10:59:662 _csync_detect_update: file: test/a|c, instruction: INSTRUCTION_IGNORE <<=

@rikvdh
Copy link
Author

@rikvdh rikvdh commented Aug 11, 2013

When I apply this patch to csync everything seems to work normal and the files are synced correctly:

diff --git a/src/csync_exclude.c b/src/csync_exclude.c
index 81f828d..42c4f0f 100644
--- a/src/csync_exclude.c
+++ b/src/csync_exclude.c
@@ -147,13 +147,6 @@ int csync_excluded(CSYNC *ctx, const char *path) {
     for (p = path; *p; p++) {
       switch (*p) {
         case '\\':
-        case ':':
-        case '?':
-        case '*':
-        case '"':
-        case '>':
-        case '<':
-        case '|':
           return 1;
         default:
           break;
@ghost
Copy link

@ghost ghost commented Aug 11, 2013

Hi,

AFAIK some of this chars aren't supported on other platforms like windows. Maybe thats just the reason why they are ignored. Keep in mind it doesn't help you if you can sync that files on linux but not on windows in a mixed setup.

@rikvdh
Copy link
Author

@rikvdh rikvdh commented Aug 11, 2013

Hi,

I see what you mean, I've booted a Windows VM and checked and I'm indeed not allowed to create files with : or |, sorry.

@rikvdh rikvdh closed this Aug 11, 2013
@MarcelWaldvogel
Copy link

@MarcelWaldvogel MarcelWaldvogel commented Aug 12, 2013

I nevertheless think this should be changed on Linux/OSX clients; if a directory is only ever used by those clients or the web interface, there is no reason to silently ignore these files.-

My expected behavior would be (in order of decreasing priority):

  1. All files are synced with the server; if a client receives a particular file from the server which contains an illegal character, it is escaped on the client side (i.e., the server file '1:0.png' becomes '1%{3A}0.png' when downloaded to a Windows box only, unchanged in all other views). There are probably better choices for the escape char…
  2. (Fallback) Warn the user on the first sync where this file is found (in either download or upload case), that it will not be synced.

BTW: If you want real Windows file name safety, files with a basename of NUL, CON, AUX, PRN, … should also be treated specially.

@a-schild
Copy link

@a-schild a-schild commented Sep 12, 2013

I also just stumbled over this issue.
A client (with OS-X) has many folder + files with timestamp as filename.
Like: "Rechnung 2013.09.12 08:12.docx"

I think this exclude list should be the list of forbiden characters the OS on which the client is running.
It has then to handle cases correctly, when there are files on the server which are not allowed to be downloaded.

When a file with name "Rechnung 2013.09.12 08:12.docx" is put via Webdav on the server,
The mac client should be able to sync it.
The linux client should be able to sync it.
The windows client has to handle it (In what way still has to be defined)

André

@rikvdh rikvdh reopened this Sep 12, 2013
@rikvdh
Copy link
Author

@rikvdh rikvdh commented Sep 12, 2013

I also think this can be fixed as @a-schild metioned. Dropbox does this the same way,

@dragotin
Copy link
Contributor

@dragotin dragotin commented Sep 27, 2013

@a-schild did not mention a solution but a feature request. What is the solution? Please explain what dropbox does and why that is good if you want us to implement that ;-)

@MarcelWaldvogel
Copy link

@MarcelWaldvogel MarcelWaldvogel commented Sep 27, 2013

I would suggest moving this thread to a different ticket, as it has nothing to do with the title anymore. I do not know how Dropbox does it, but a possible method would be similar to the one used in PeerStore:

  1. To make the method cope with inserts, deletes, and moves in the file, they use Rabin fingerprints to split the file in anchor-based blocks of many kiB each. (E.g. by waiting for the least significant 16 bits to become 0, the file is split into blocks of about 64kiB each; resynchronizing in the first or second block after the change.)
  2. When synchronizing the file, the sequences of the hashes of the updated file are sent to the other side, together with the changed blocks. The remote side can then reconstruct the file from this information.

Of course, there are some options to this basic scheme which can improve worst-case behavior (e.g., storing the hashes locally or bounding the blocks into a given range of lengths).

@a-schild
Copy link

@a-schild a-schild commented Sep 28, 2013

@dragotin Yes, it was not a solution, but it's definitively a unexpected behaviour of the sync client.
One good showcase is for example the sync of a music library.
Here we often have : and ? characters under OS-X & Linux.

I would suggest that the client does sync all files with legal filenames of the platform it is running on.
So a linux + OS-X can sync files with *,? and : characters in file+folder names from/to the server.

On the other hand, a Windows sync client will then have make a descision on what to do with such invalid filenames.
As @MarcelWaldvogel mentioned in the second last post, it could for example escape all "invalid" characters to something like %xx or just replace them with a _

I don't see what the last comment of @MarcelWaldvogel has to do with the concept of invalid file name characters...?

@MarcelWaldvogel
Copy link

@MarcelWaldvogel MarcelWaldvogel commented Sep 28, 2013

Sorry, had confused the thread. Please ignore.

Viele Grüsse,
-Marcel Waldvogel
(kurz&bündig, da mobil)

Am 28.09.2013 um 08:39 schrieb a-schild notifications@github.com:

@dragotin Yes, it was not a solution, but it's definitively a unexpected behaviour of the sync client.
One good showcase is for example the sync of a music library.
Here we often have : and ? characters under OS-X & Linux.

I would suggest that the client does sync all files with legal filenames of the platform it is running on.
So a linux + OS-X can sync files with *,? and : characters in file+folder names from/to the server.

On the other hand, a Windows sync client will then have make a descision on what to do with such invalid filenames.
As @MarcelWaldvogel mentioned in the second last post, it could for example escape all "invalid" characters to something like %xx or just replace them with a _

I don't see what the last comment of @MarcelWaldvogel has to do with the concept of invalid file name characters...?


Reply to this email directly or view it on GitHub.

@ghost
Copy link

@ghost ghost commented Sep 28, 2013

Hi,

just my two cents:

Having different behaviors in the client on different platforms will probably leads to a big mess:

  • What happens if you're syncing a file with a special char from the linux client which allows this to an windos server?
  • What happens if you're replacing the special chars as written above with _? This files gets synced again to the server and will be probably downloaded and duplicated on linux clients because they can't know if this is the real filename or a replaced special char.

I don't think that it is that easy to just replace special chars or escape them but maybe i'm wrong.

@a-schild
Copy link

@a-schild a-schild commented Sep 28, 2013

@RealRancor Files with öäüéàè etc. should not cause problems (At leat for me they work)

@a-schild
Copy link

@a-schild a-schild commented Sep 28, 2013

I know it's not a simple thing to do the sync right, but to NOT sync them is definitely the wrong thing.

Using a escape-style encoding of the special characters should be possible... (Not saying it's easy)

@ghost
Copy link

@ghost ghost commented Sep 28, 2013

Files with öäüéàè etc. should not cause problems (At leat for me they work)

This is just a reference from the other issue where i'm referencing a user to this issue about the ignored : and | chars in filenames.

but to NOT sync them is definitely the wrong thing.

I rather would write:

but to NOT sync them is definitely the wrong thing in my opinion.

:)

@rikvdh
Copy link
Author

@rikvdh rikvdh commented Sep 28, 2013

@RealRancor It is the wrong thing. If the tool doesn't notify users about not syncing files the whole tool isn't reliable and will not be adopted by companies or non-technical people.

I think escaping on server-side is a solution, as I would not prefer Windows as a server it is probably still something you want to support.

Client side you can replace characters if the OS does not support it. Replacing : and | on windows to _ will still make filenames read-able for non-geeks.

From my opinion this is the best solution, and also Dropbox does it this way. (not sure for server-side, but probably they don't run windows because their transfer-system is based on rsync).

@ghost
Copy link

@ghost ghost commented Sep 28, 2013

It is the wrong thing.

Its still your opinion. I think the devs have their reasons why they are ignoring those files. Let them decide how the client is handling this stuff. :)

Client side you can replace characters if the OS does not support it. Replacing : and | on windows to _ will still
make filenames read-able for non-geeks.

Renaming is probably the worst solution for this because if you rename the file it will become a new file. If you're doing a rename you probably need to rename the file in all clients (Mac, Linux, Windows)?

@dragotin
Copy link
Contributor

@dragotin dragotin commented Sep 28, 2013

@RealRancor is absolutely true saying that it is not easy at all. If we change a filename by escaping the next bug report will be that we overwrite files with escaped ones...

It is doable. But it is not "just escape and be happy". The day will come were we open that keg... OTOH, nice that we are now in a state were we discuss this as a real problem, we had other times with other probs ;-)

BTW, the "ignored chars" problem should not be confused with UTF8 problems (à,ò,è,ì and friends) - these often originate from misconfiguration on server- or client side.

@ghost
Copy link

@ghost ghost commented Sep 28, 2013

Hi,

BTW, the "ignored chars" problem should not be confused with UTF8 problems (à,ò,è,ì and friends)

this problem is not confused with the UTF8 problem. :) A user in the referenced issue (not the creator of the issue) had exactly the question why files with : are ignored:

#1048 (comment)

@muetze-online
Copy link

@muetze-online muetze-online commented Nov 7, 2013

I see the problem to find a good way with these files unter Windows while downloading them.

But the situation NOW is, that any Mac- or Linux-user will only see, that their files are NOT uploaded, when they look into the sync-details. The ownCloud-client is a sync tool that doesn't sync my files - not the expected behaviour. I have these files for years and I don't think, a program should tell me, how my files are named - only the operating system should do such things.

Do it the dropbox-way:
You can delegate the problem to the Windows users: A Windows-User will see in the sync-details, that some files are not downloaded. If they use Windows and Mac/Linux they probably know about the filename incoherences. I tried this with dropbox. Files with " or ? in its name will be uploaded from the Mac and NOT downloaded on a Windows-system. This is no perfect way, but I will sync my files using ownCloud - not dropbox.

Please make it possible to sync files with " and ? etc.

muetze

@KasumiNinja
Copy link

@KasumiNinja KasumiNinja commented Nov 11, 2013

I also think that ignoring files is the wrong thing to do. This leads to unexpected behaviour and could lead to users losing files. I think it's best to rename files with underscore. Is there any eta when this problem will be addressed?

@muetze-online
Copy link

@muetze-online muetze-online commented Nov 12, 2013

Renaming is no good idea, because it doubles all files with these characters, because the renamed file will be synced either. Instead the renaming has to be done in a way, that the new file will not be synced - like filename_renamed_by_ownCloud.ext.
But there is the next problem - how will changes in this file be resented to the original with the ignored characters. You can do it with a journal of all renaming - but I don't know if this is an solution.

muetze

@MarcelWaldvogel
Copy link

@MarcelWaldvogel MarcelWaldvogel commented Nov 12, 2013

The solutions in order of increasing implementation complexity:

  1. Do not sync files with characters which cannot be represented by the local file system, but add it to the list of "ignored files" in the sync log (1)
  2. Use escape sequences on the local file system only. Instead of (ab)using an ASCII character again, which is likely used (e.g. percent-encoding), use a Unicode character that is unlikely to be used in file names, such as U+2707 (tape drive). Then replace ":" with (TAPE DRIVE)COLON(TAPE DRIVE) etc., which still allows the user to use the tape drive symbol in file names, if she really wants to.
  3. Also record this translation in the csyncdb; however, this makes it hard to unencode the translated filename on copied/moved files.

(1) It would be good to list a reason for the file being ignored during sync: filename in ignore pattern list, illegal character, or hard link (even though I strongly believe that a hard link should not be a reason for ignoring the synchronization, but it currently is the case)

@muetze-online
Copy link

@muetze-online muetze-online commented Nov 12, 2013

Thank you for this clearly and evident post.

muetze

@a-schild
Copy link

@a-schild a-schild commented Nov 26, 2013

MS SkyDrive has similar problems, and this it's not well accepted by the users

http://community.office365.com/en-us/forums/154/t/165638.aspx

@uvesten
Copy link

@uvesten uvesten commented Oct 15, 2014

Agree with futurePilot, this needs to be more visible for the user!

@dragotin
Copy link
Contributor

@dragotin dragotin commented Oct 15, 2014

We had a more visible notification of that earlier and people complained, so we changed it to the way it is now, which shows the list in the activity tab in the setup page.

@muetze-online
Copy link

@muetze-online muetze-online commented Oct 16, 2014

I turned my back to Windows years ago - and now its got me by the balls :-). I don't want my system to be restricted by the problems of Windows and its filesystem.

@axlblue
Copy link

@axlblue axlblue commented Oct 27, 2014

I'm not a programmer.
My issue is now 'closed' but how do I fix my problem?
Can't the Mac client deal with this?

@dragotin
Copy link
Contributor

@dragotin dragotin commented Dec 4, 2014

Closing this with a link to https://github.com/owncloud/core/wiki/Cross-Platform-File-Handling which describes how we will tackle this problem once we're at it.

@dragotin dragotin closed this Dec 4, 2014
@Bazon
Copy link

@Bazon Bazon commented Dec 30, 2014

Just for the protocol another case where this leads to trouble:
Pictures created by the Android Hangouts App get names like "IMG_20141230_123250:nopm:.jpg".
The start comes from the Camera-App, Hangouts adds ":nopm:".
--> These files can't be shared via OwnCloud in an easy way.

@unamedplayer
Copy link

@unamedplayer unamedplayer commented Mar 6, 2015

Hello, just adding my voice to this. Out of curiosity I've done some testing with Google Drive and they handle all kinds of odd characters via translation. The file names may not be consistent across filesystems but no data is left unsynced which is very important IMHO.

Thank you for all the comments and pointing out the path forward.

@danimo
Copy link
Contributor

@danimo danimo commented Mar 6, 2015

fyi: This is being worked on for ownCloud Server 8.1 on a broader scope.

@unamedplayer
Copy link

@unamedplayer unamedplayer commented Mar 6, 2015

Yay :) Thank you!

@unamedplayer
Copy link

@unamedplayer unamedplayer commented Jun 24, 2015

@danimo , do you have any fresh info on this by any chance? I don't mean to pester but I've been keeping a close eye on all beta releases, we are now well into 8.1 and I don't see any indication that this is being addressed at the moment. Which is fine I'm not complaining :) but I'm wondering about current state of things is and also what you meant by "broader scope". Thank you!

@AshleyYakeley
Copy link

@AshleyYakeley AshleyYakeley commented Nov 20, 2015

What's the status of this? The issue is closed, but it still appears to be a problem in 8.1?

@brevilo
Copy link

@brevilo brevilo commented Nov 20, 2015

If I'm not mistaken it takes client 2.1 (soon to be released) for this to work with the 8.1 server...

@DCKcode
Copy link

@DCKcode DCKcode commented May 29, 2016

This bug still affects me. I'm on OwnCloud 9.0.2 server (ZFS on FreeBSD) with client 2.1.1 on OS X. Files with colons in their name don't sync, rejected with a 400 error by the server. This bug has been present in all earlier versions I've used too. Has this really been resolved?

If possible I'd like to help debugging the problem.

@dragotin
Copy link
Contributor

@dragotin dragotin commented May 30, 2016

@DeepDiver1975 isn't that supposed to be fixed with server 9.0.x? The client ignores these filenames only on win32.

@DeepDiver1975
Copy link
Member

@DeepDiver1975 DeepDiver1975 commented May 30, 2016

@DeepDiver1975 isn't that supposed to be fixed with server 9.0.x? The client ignores these filenames only on win32.

yes - the server is accepting file names depending on the server side filesystem.
On a posix server system only \ and / are not allowed - see https://github.com/owncloud/core/blob/master/lib/private/Files/Storage/Common.php#L519

@DeepDiver1975
Copy link
Member

@DeepDiver1975 DeepDiver1975 commented May 30, 2016

and this is implemented since 8.1

@erhan-
Copy link

@erhan- erhan- commented Jun 1, 2016

I have the same issue on a debian jessie server with ubuntu 12.04 lts client:

|0|path/file:name.pdf|INST_NEW|Up|145795xxxx||3235xxx||6|The item is not synced because of previous errors: Error downloading https://web/remote.php/webdav/path/file:name.pdf - server replied: Internal Server Error (Invalid request for Invalid path, ":" is not allowed (InvalidPathException))|0|0|0|||INST_NONE|

@a-schild
Copy link

@a-schild a-schild commented Jun 1, 2016

@erhan- what owncloud server & client versions are you using?

@erhan-
Copy link

@erhan- erhan- commented Jun 3, 2016

Server: 9.0.2
Client: 2.2.0

Had multiple illogical problems with one specific client after client update to 2.2.0 but it all resolved after removing the account from the client and readding it again, resyncing in a new folder. Consider my problem as solved.

@mckaygerhard
Copy link

@mckaygerhard mckaygerhard commented Jan 10, 2018

i used and have that problem, but moved to nextcloud and not any more. puff oc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet