Files and directories containing : or | are ignored #854

Closed
rikvdh opened this Issue Aug 11, 2013 · 65 comments

Comments

Projects
None yet
@rikvdh

rikvdh commented Aug 11, 2013

Expected behaviour

Files getting synced to my ownCloud server

Actual behaviour

I see this occurring in the Linux client.
When I create files or directories containing | : > < ? or a few other characters, they are automatically ignored.

I already did some research and found this commit in csync:
http://git.csync.org/users/freitag/csync.git/commit/?id=e0807cba1b164fcf409cb71e3da2f8f7b88637a9

I don't know why this commit is there, creating files on Linux with | : < > * ? is perfectly valid. :)

total 16
drwxr-xr-x 4 rik rik 4096 Aug 11 08:32 .
drwxr-xr-x 10 rik rik 4096 Aug 10 16:11 ..
-rw-r--r-- 1 rik rik 0 Aug 11 08:22 abc*def
-rw-r--r-- 1 rik rik 0 Aug 10 16:02 abc:def
-rw-r--r-- 1 rik rik 0 Aug 11 08:21 abc>def
-rw-r--r-- 1 rik rik 0 Aug 11 08:22 abc?def
drwxr-xr-x 2 rik rik 4096 Aug 11 08:22 abc|def
drwxr-xr-x 2 rik rik 4096 Aug 10 16:08 a|c

Steps to reproduce

  1. Install Linux client
  2. Sync ownCloud
  3. Create a file with: touch 'abc|def' and see it getting ignored :)

Server configuration

not applicable, normal Debian Wheezy with apache2

Client configuration

Client version: latest from git (but also occurs with 1.3.0)
Operating system: Debian Wheezy
OS language: en_US
Installation path of client: /usr/local/bin/owncloud

Log

08-10 16:10:59:661 csync_ftw: Uniq ID from Database: test/a|c ->
08-10 16:10:59:661 csync_walker: directory: /home/rik/ownCloud/test/a|c
08-10 16:10:59:661 _csync_detect_update: test/a|c excluded (1)
08-10 16:10:59:661 _csync_detect_update: ==> file: test/a|c - hash 6527849599191992178, mtime: 1376143708
08-10 16:10:59:662 _csync_detect_update: file: test/a|c, instruction: INSTRUCTION_IGNORE <<=

@rikvdh

This comment has been minimized.

Show comment
Hide comment
@rikvdh

rikvdh Aug 11, 2013

When I apply this patch to csync everything seems to work normal and the files are synced correctly:

diff --git a/src/csync_exclude.c b/src/csync_exclude.c
index 81f828d..42c4f0f 100644
--- a/src/csync_exclude.c
+++ b/src/csync_exclude.c
@@ -147,13 +147,6 @@ int csync_excluded(CSYNC *ctx, const char *path) {
     for (p = path; *p; p++) {
       switch (*p) {
         case '\\':
-        case ':':
-        case '?':
-        case '*':
-        case '"':
-        case '>':
-        case '<':
-        case '|':
           return 1;
         default:
           break;

rikvdh commented Aug 11, 2013

When I apply this patch to csync everything seems to work normal and the files are synced correctly:

diff --git a/src/csync_exclude.c b/src/csync_exclude.c
index 81f828d..42c4f0f 100644
--- a/src/csync_exclude.c
+++ b/src/csync_exclude.c
@@ -147,13 +147,6 @@ int csync_excluded(CSYNC *ctx, const char *path) {
     for (p = path; *p; p++) {
       switch (*p) {
         case '\\':
-        case ':':
-        case '?':
-        case '*':
-        case '"':
-        case '>':
-        case '<':
-        case '|':
           return 1;
         default:
           break;
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Aug 11, 2013

Hi,

AFAIK some of this chars aren't supported on other platforms like windows. Maybe thats just the reason why they are ignored. Keep in mind it doesn't help you if you can sync that files on linux but not on windows in a mixed setup.

ghost commented Aug 11, 2013

Hi,

AFAIK some of this chars aren't supported on other platforms like windows. Maybe thats just the reason why they are ignored. Keep in mind it doesn't help you if you can sync that files on linux but not on windows in a mixed setup.

@rikvdh

This comment has been minimized.

Show comment
Hide comment
@rikvdh

rikvdh Aug 11, 2013

Hi,

I see what you mean, I've booted a Windows VM and checked and I'm indeed not allowed to create files with : or |, sorry.

rikvdh commented Aug 11, 2013

Hi,

I see what you mean, I've booted a Windows VM and checked and I'm indeed not allowed to create files with : or |, sorry.

@rikvdh rikvdh closed this Aug 11, 2013

@MarcelWaldvogel

This comment has been minimized.

Show comment
Hide comment
@MarcelWaldvogel

MarcelWaldvogel Aug 12, 2013

I nevertheless think this should be changed on Linux/OSX clients; if a directory is only ever used by those clients or the web interface, there is no reason to silently ignore these files.-

My expected behavior would be (in order of decreasing priority):

  1. All files are synced with the server; if a client receives a particular file from the server which contains an illegal character, it is escaped on the client side (i.e., the server file '1:0.png' becomes '1%{3A}0.png' when downloaded to a Windows box only, unchanged in all other views). There are probably better choices for the escape char…
  2. (Fallback) Warn the user on the first sync where this file is found (in either download or upload case), that it will not be synced.

BTW: If you want real Windows file name safety, files with a basename of NUL, CON, AUX, PRN, … should also be treated specially.

I nevertheless think this should be changed on Linux/OSX clients; if a directory is only ever used by those clients or the web interface, there is no reason to silently ignore these files.-

My expected behavior would be (in order of decreasing priority):

  1. All files are synced with the server; if a client receives a particular file from the server which contains an illegal character, it is escaped on the client side (i.e., the server file '1:0.png' becomes '1%{3A}0.png' when downloaded to a Windows box only, unchanged in all other views). There are probably better choices for the escape char…
  2. (Fallback) Warn the user on the first sync where this file is found (in either download or upload case), that it will not be synced.

BTW: If you want real Windows file name safety, files with a basename of NUL, CON, AUX, PRN, … should also be treated specially.

@a-schild

This comment has been minimized.

Show comment
Hide comment
@a-schild

a-schild Sep 12, 2013

I also just stumbled over this issue.
A client (with OS-X) has many folder + files with timestamp as filename.
Like: "Rechnung 2013.09.12 08:12.docx"

I think this exclude list should be the list of forbiden characters the OS on which the client is running.
It has then to handle cases correctly, when there are files on the server which are not allowed to be downloaded.

When a file with name "Rechnung 2013.09.12 08:12.docx" is put via Webdav on the server,
The mac client should be able to sync it.
The linux client should be able to sync it.
The windows client has to handle it (In what way still has to be defined)

André

I also just stumbled over this issue.
A client (with OS-X) has many folder + files with timestamp as filename.
Like: "Rechnung 2013.09.12 08:12.docx"

I think this exclude list should be the list of forbiden characters the OS on which the client is running.
It has then to handle cases correctly, when there are files on the server which are not allowed to be downloaded.

When a file with name "Rechnung 2013.09.12 08:12.docx" is put via Webdav on the server,
The mac client should be able to sync it.
The linux client should be able to sync it.
The windows client has to handle it (In what way still has to be defined)

André

@rikvdh rikvdh reopened this Sep 12, 2013

@rikvdh

This comment has been minimized.

Show comment
Hide comment
@rikvdh

rikvdh Sep 12, 2013

I also think this can be fixed as @a-schild metioned. Dropbox does this the same way,

rikvdh commented Sep 12, 2013

I also think this can be fixed as @a-schild metioned. Dropbox does this the same way,

@dragotin

This comment has been minimized.

Show comment
Hide comment
@dragotin

dragotin Sep 27, 2013

Contributor

@a-schild did not mention a solution but a feature request. What is the solution? Please explain what dropbox does and why that is good if you want us to implement that ;-)

Contributor

dragotin commented Sep 27, 2013

@a-schild did not mention a solution but a feature request. What is the solution? Please explain what dropbox does and why that is good if you want us to implement that ;-)

@MarcelWaldvogel

This comment has been minimized.

Show comment
Hide comment
@MarcelWaldvogel

MarcelWaldvogel Sep 27, 2013

I would suggest moving this thread to a different ticket, as it has nothing to do with the title anymore. I do not know how Dropbox does it, but a possible method would be similar to the one used in PeerStore:

  1. To make the method cope with inserts, deletes, and moves in the file, they use Rabin fingerprints to split the file in anchor-based blocks of many kiB each. (E.g. by waiting for the least significant 16 bits to become 0, the file is split into blocks of about 64kiB each; resynchronizing in the first or second block after the change.)
  2. When synchronizing the file, the sequences of the hashes of the updated file are sent to the other side, together with the changed blocks. The remote side can then reconstruct the file from this information.

Of course, there are some options to this basic scheme which can improve worst-case behavior (e.g., storing the hashes locally or bounding the blocks into a given range of lengths).

I would suggest moving this thread to a different ticket, as it has nothing to do with the title anymore. I do not know how Dropbox does it, but a possible method would be similar to the one used in PeerStore:

  1. To make the method cope with inserts, deletes, and moves in the file, they use Rabin fingerprints to split the file in anchor-based blocks of many kiB each. (E.g. by waiting for the least significant 16 bits to become 0, the file is split into blocks of about 64kiB each; resynchronizing in the first or second block after the change.)
  2. When synchronizing the file, the sequences of the hashes of the updated file are sent to the other side, together with the changed blocks. The remote side can then reconstruct the file from this information.

Of course, there are some options to this basic scheme which can improve worst-case behavior (e.g., storing the hashes locally or bounding the blocks into a given range of lengths).

@a-schild

This comment has been minimized.

Show comment
Hide comment
@a-schild

a-schild Sep 28, 2013

@dragotin Yes, it was not a solution, but it's definitively a unexpected behaviour of the sync client.
One good showcase is for example the sync of a music library.
Here we often have : and ? characters under OS-X & Linux.

I would suggest that the client does sync all files with legal filenames of the platform it is running on.
So a linux + OS-X can sync files with *,? and : characters in file+folder names from/to the server.

On the other hand, a Windows sync client will then have make a descision on what to do with such invalid filenames.
As @MarcelWaldvogel mentioned in the second last post, it could for example escape all "invalid" characters to something like %xx or just replace them with a _

I don't see what the last comment of @MarcelWaldvogel has to do with the concept of invalid file name characters...?

@dragotin Yes, it was not a solution, but it's definitively a unexpected behaviour of the sync client.
One good showcase is for example the sync of a music library.
Here we often have : and ? characters under OS-X & Linux.

I would suggest that the client does sync all files with legal filenames of the platform it is running on.
So a linux + OS-X can sync files with *,? and : characters in file+folder names from/to the server.

On the other hand, a Windows sync client will then have make a descision on what to do with such invalid filenames.
As @MarcelWaldvogel mentioned in the second last post, it could for example escape all "invalid" characters to something like %xx or just replace them with a _

I don't see what the last comment of @MarcelWaldvogel has to do with the concept of invalid file name characters...?

@MarcelWaldvogel

This comment has been minimized.

Show comment
Hide comment
@MarcelWaldvogel

MarcelWaldvogel Sep 28, 2013

Sorry, had confused the thread. Please ignore.

Viele Grüsse,
-Marcel Waldvogel
(kurz&bündig, da mobil)

Am 28.09.2013 um 08:39 schrieb a-schild notifications@github.com:

@dragotin Yes, it was not a solution, but it's definitively a unexpected behaviour of the sync client.
One good showcase is for example the sync of a music library.
Here we often have : and ? characters under OS-X & Linux.

I would suggest that the client does sync all files with legal filenames of the platform it is running on.
So a linux + OS-X can sync files with *,? and : characters in file+folder names from/to the server.

On the other hand, a Windows sync client will then have make a descision on what to do with such invalid filenames.
As @MarcelWaldvogel mentioned in the second last post, it could for example escape all "invalid" characters to something like %xx or just replace them with a _

I don't see what the last comment of @MarcelWaldvogel has to do with the concept of invalid file name characters...?


Reply to this email directly or view it on GitHub.

Sorry, had confused the thread. Please ignore.

Viele Grüsse,
-Marcel Waldvogel
(kurz&bündig, da mobil)

Am 28.09.2013 um 08:39 schrieb a-schild notifications@github.com:

@dragotin Yes, it was not a solution, but it's definitively a unexpected behaviour of the sync client.
One good showcase is for example the sync of a music library.
Here we often have : and ? characters under OS-X & Linux.

I would suggest that the client does sync all files with legal filenames of the platform it is running on.
So a linux + OS-X can sync files with *,? and : characters in file+folder names from/to the server.

On the other hand, a Windows sync client will then have make a descision on what to do with such invalid filenames.
As @MarcelWaldvogel mentioned in the second last post, it could for example escape all "invalid" characters to something like %xx or just replace them with a _

I don't see what the last comment of @MarcelWaldvogel has to do with the concept of invalid file name characters...?


Reply to this email directly or view it on GitHub.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Sep 28, 2013

Hi,

just my two cents:

Having different behaviors in the client on different platforms will probably leads to a big mess:

  • What happens if you're syncing a file with a special char from the linux client which allows this to an windos server?
  • What happens if you're replacing the special chars as written above with _? This files gets synced again to the server and will be probably downloaded and duplicated on linux clients because they can't know if this is the real filename or a replaced special char.

I don't think that it is that easy to just replace special chars or escape them but maybe i'm wrong.

ghost commented Sep 28, 2013

Hi,

just my two cents:

Having different behaviors in the client on different platforms will probably leads to a big mess:

  • What happens if you're syncing a file with a special char from the linux client which allows this to an windos server?
  • What happens if you're replacing the special chars as written above with _? This files gets synced again to the server and will be probably downloaded and duplicated on linux clients because they can't know if this is the real filename or a replaced special char.

I don't think that it is that easy to just replace special chars or escape them but maybe i'm wrong.

@a-schild

This comment has been minimized.

Show comment
Hide comment
@a-schild

a-schild Sep 28, 2013

@RealRancor Files with öäüéàè etc. should not cause problems (At leat for me they work)

@RealRancor Files with öäüéàè etc. should not cause problems (At leat for me they work)

@a-schild

This comment has been minimized.

Show comment
Hide comment
@a-schild

a-schild Sep 28, 2013

I know it's not a simple thing to do the sync right, but to NOT sync them is definitely the wrong thing.

Using a escape-style encoding of the special characters should be possible... (Not saying it's easy)

I know it's not a simple thing to do the sync right, but to NOT sync them is definitely the wrong thing.

Using a escape-style encoding of the special characters should be possible... (Not saying it's easy)

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Sep 28, 2013

Files with öäüéàè etc. should not cause problems (At leat for me they work)

This is just a reference from the other issue where i'm referencing a user to this issue about the ignored : and | chars in filenames.

but to NOT sync them is definitely the wrong thing.

I rather would write:

but to NOT sync them is definitely the wrong thing in my opinion.

:)

ghost commented Sep 28, 2013

Files with öäüéàè etc. should not cause problems (At leat for me they work)

This is just a reference from the other issue where i'm referencing a user to this issue about the ignored : and | chars in filenames.

but to NOT sync them is definitely the wrong thing.

I rather would write:

but to NOT sync them is definitely the wrong thing in my opinion.

:)

@rikvdh

This comment has been minimized.

Show comment
Hide comment
@rikvdh

rikvdh Sep 28, 2013

@RealRancor It is the wrong thing. If the tool doesn't notify users about not syncing files the whole tool isn't reliable and will not be adopted by companies or non-technical people.

I think escaping on server-side is a solution, as I would not prefer Windows as a server it is probably still something you want to support.

Client side you can replace characters if the OS does not support it. Replacing : and | on windows to _ will still make filenames read-able for non-geeks.

From my opinion this is the best solution, and also Dropbox does it this way. (not sure for server-side, but probably they don't run windows because their transfer-system is based on rsync).

rikvdh commented Sep 28, 2013

@RealRancor It is the wrong thing. If the tool doesn't notify users about not syncing files the whole tool isn't reliable and will not be adopted by companies or non-technical people.

I think escaping on server-side is a solution, as I would not prefer Windows as a server it is probably still something you want to support.

Client side you can replace characters if the OS does not support it. Replacing : and | on windows to _ will still make filenames read-able for non-geeks.

From my opinion this is the best solution, and also Dropbox does it this way. (not sure for server-side, but probably they don't run windows because their transfer-system is based on rsync).

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Sep 28, 2013

It is the wrong thing.

Its still your opinion. I think the devs have their reasons why they are ignoring those files. Let them decide how the client is handling this stuff. :)

Client side you can replace characters if the OS does not support it. Replacing : and | on windows to _ will still
make filenames read-able for non-geeks.

Renaming is probably the worst solution for this because if you rename the file it will become a new file. If you're doing a rename you probably need to rename the file in all clients (Mac, Linux, Windows)?

ghost commented Sep 28, 2013

It is the wrong thing.

Its still your opinion. I think the devs have their reasons why they are ignoring those files. Let them decide how the client is handling this stuff. :)

Client side you can replace characters if the OS does not support it. Replacing : and | on windows to _ will still
make filenames read-able for non-geeks.

Renaming is probably the worst solution for this because if you rename the file it will become a new file. If you're doing a rename you probably need to rename the file in all clients (Mac, Linux, Windows)?

@dragotin

This comment has been minimized.

Show comment
Hide comment
@dragotin

dragotin Sep 28, 2013

Contributor

@RealRancor is absolutely true saying that it is not easy at all. If we change a filename by escaping the next bug report will be that we overwrite files with escaped ones...

It is doable. But it is not "just escape and be happy". The day will come were we open that keg... OTOH, nice that we are now in a state were we discuss this as a real problem, we had other times with other probs ;-)

BTW, the "ignored chars" problem should not be confused with UTF8 problems (à,ò,è,ì and friends) - these often originate from misconfiguration on server- or client side.

Contributor

dragotin commented Sep 28, 2013

@RealRancor is absolutely true saying that it is not easy at all. If we change a filename by escaping the next bug report will be that we overwrite files with escaped ones...

It is doable. But it is not "just escape and be happy". The day will come were we open that keg... OTOH, nice that we are now in a state were we discuss this as a real problem, we had other times with other probs ;-)

BTW, the "ignored chars" problem should not be confused with UTF8 problems (à,ò,è,ì and friends) - these often originate from misconfiguration on server- or client side.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Sep 28, 2013

Hi,

BTW, the "ignored chars" problem should not be confused with UTF8 problems (à,ò,è,ì and friends)

this problem is not confused with the UTF8 problem. :) A user in the referenced issue (not the creator of the issue) had exactly the question why files with : are ignored:

#1048 (comment)

ghost commented Sep 28, 2013

Hi,

BTW, the "ignored chars" problem should not be confused with UTF8 problems (à,ò,è,ì and friends)

this problem is not confused with the UTF8 problem. :) A user in the referenced issue (not the creator of the issue) had exactly the question why files with : are ignored:

#1048 (comment)

@ghost ghost referenced this issue Oct 2, 2013

Closed

Client doesn't sync UTF8 chars #1058

@muetze-online

This comment has been minimized.

Show comment
Hide comment
@muetze-online

muetze-online Nov 7, 2013

I see the problem to find a good way with these files unter Windows while downloading them.

But the situation NOW is, that any Mac- or Linux-user will only see, that their files are NOT uploaded, when they look into the sync-details. The ownCloud-client is a sync tool that doesn't sync my files - not the expected behaviour. I have these files for years and I don't think, a program should tell me, how my files are named - only the operating system should do such things.

Do it the dropbox-way:
You can delegate the problem to the Windows users: A Windows-User will see in the sync-details, that some files are not downloaded. If they use Windows and Mac/Linux they probably know about the filename incoherences. I tried this with dropbox. Files with " or ? in its name will be uploaded from the Mac and NOT downloaded on a Windows-system. This is no perfect way, but I will sync my files using ownCloud - not dropbox.

Please make it possible to sync files with " and ? etc.

muetze

I see the problem to find a good way with these files unter Windows while downloading them.

But the situation NOW is, that any Mac- or Linux-user will only see, that their files are NOT uploaded, when they look into the sync-details. The ownCloud-client is a sync tool that doesn't sync my files - not the expected behaviour. I have these files for years and I don't think, a program should tell me, how my files are named - only the operating system should do such things.

Do it the dropbox-way:
You can delegate the problem to the Windows users: A Windows-User will see in the sync-details, that some files are not downloaded. If they use Windows and Mac/Linux they probably know about the filename incoherences. I tried this with dropbox. Files with " or ? in its name will be uploaded from the Mac and NOT downloaded on a Windows-system. This is no perfect way, but I will sync my files using ownCloud - not dropbox.

Please make it possible to sync files with " and ? etc.

muetze

@KasumiNinja

This comment has been minimized.

Show comment
Hide comment
@KasumiNinja

KasumiNinja Nov 11, 2013

I also think that ignoring files is the wrong thing to do. This leads to unexpected behaviour and could lead to users losing files. I think it's best to rename files with underscore. Is there any eta when this problem will be addressed?

I also think that ignoring files is the wrong thing to do. This leads to unexpected behaviour and could lead to users losing files. I think it's best to rename files with underscore. Is there any eta when this problem will be addressed?

@muetze-online

This comment has been minimized.

Show comment
Hide comment
@muetze-online

muetze-online Nov 12, 2013

Renaming is no good idea, because it doubles all files with these characters, because the renamed file will be synced either. Instead the renaming has to be done in a way, that the new file will not be synced - like filename_renamed_by_ownCloud.ext.
But there is the next problem - how will changes in this file be resented to the original with the ignored characters. You can do it with a journal of all renaming - but I don't know if this is an solution.

muetze

Renaming is no good idea, because it doubles all files with these characters, because the renamed file will be synced either. Instead the renaming has to be done in a way, that the new file will not be synced - like filename_renamed_by_ownCloud.ext.
But there is the next problem - how will changes in this file be resented to the original with the ignored characters. You can do it with a journal of all renaming - but I don't know if this is an solution.

muetze

@MarcelWaldvogel

This comment has been minimized.

Show comment
Hide comment
@MarcelWaldvogel

MarcelWaldvogel Nov 12, 2013

The solutions in order of increasing implementation complexity:

  1. Do not sync files with characters which cannot be represented by the local file system, but add it to the list of "ignored files" in the sync log (1)
  2. Use escape sequences on the local file system only. Instead of (ab)using an ASCII character again, which is likely used (e.g. percent-encoding), use a Unicode character that is unlikely to be used in file names, such as U+2707 (tape drive). Then replace ":" with (TAPE DRIVE)COLON(TAPE DRIVE) etc., which still allows the user to use the tape drive symbol in file names, if she really wants to.
  3. Also record this translation in the csyncdb; however, this makes it hard to unencode the translated filename on copied/moved files.

(1) It would be good to list a reason for the file being ignored during sync: filename in ignore pattern list, illegal character, or hard link (even though I strongly believe that a hard link should not be a reason for ignoring the synchronization, but it currently is the case)

The solutions in order of increasing implementation complexity:

  1. Do not sync files with characters which cannot be represented by the local file system, but add it to the list of "ignored files" in the sync log (1)
  2. Use escape sequences on the local file system only. Instead of (ab)using an ASCII character again, which is likely used (e.g. percent-encoding), use a Unicode character that is unlikely to be used in file names, such as U+2707 (tape drive). Then replace ":" with (TAPE DRIVE)COLON(TAPE DRIVE) etc., which still allows the user to use the tape drive symbol in file names, if she really wants to.
  3. Also record this translation in the csyncdb; however, this makes it hard to unencode the translated filename on copied/moved files.

(1) It would be good to list a reason for the file being ignored during sync: filename in ignore pattern list, illegal character, or hard link (even though I strongly believe that a hard link should not be a reason for ignoring the synchronization, but it currently is the case)

@muetze-online

This comment has been minimized.

Show comment
Hide comment
@muetze-online

muetze-online Nov 12, 2013

Thank you for this clearly and evident post.

muetze

Thank you for this clearly and evident post.

muetze

@a-schild

This comment has been minimized.

Show comment
Hide comment
@a-schild

a-schild Nov 26, 2013

MS SkyDrive has similar problems, and this it's not well accepted by the users

http://community.office365.com/en-us/forums/154/t/165638.aspx

MS SkyDrive has similar problems, and this it's not well accepted by the users

http://community.office365.com/en-us/forums/154/t/165638.aspx

@danimo

This comment has been minimized.

Show comment
Hide comment
@danimo

danimo Nov 28, 2013

Contributor

Just an update on this: We can't decide this on the client side. The problem is that we need to ensure that the server can always handle the characters that we allow to upload. I have created owncloud/core#6102 to track the server-part discussion.

Contributor

danimo commented Nov 28, 2013

Just an update on this: We can't decide this on the client side. The problem is that we need to ensure that the server can always handle the characters that we allow to upload. I have created owncloud/core#6102 to track the server-part discussion.

@muetze-online

This comment has been minimized.

Show comment
Hide comment
@muetze-online

muetze-online Jan 19, 2014

Is it not possible to let the user decide, which characters are synced. I am a fan of transparency. It took me a time to find the problem ignored characters in filenames. There could be an option for every synced folder like: "sync folder only on Mac, forbidden chars: ':'" and "folder sync with windows, forbidden chars: ":,<>/|?'" etc.

This lets the user decide, which characters will be synced and tells him. A help text can explain the problems with some chars under windows - Mac OS X can use all characters except ":". I decided this operating system because of its wide possibilities and the OwnCloud-Client make me not to use this chars - this su***.

I think, transparency AND user decision can be a solution, until a server-sided solution is found, which will be difficult.

Is it not possible to let the user decide, which characters are synced. I am a fan of transparency. It took me a time to find the problem ignored characters in filenames. There could be an option for every synced folder like: "sync folder only on Mac, forbidden chars: ':'" and "folder sync with windows, forbidden chars: ":,<>/|?'" etc.

This lets the user decide, which characters will be synced and tells him. A help text can explain the problems with some chars under windows - Mac OS X can use all characters except ":". I decided this operating system because of its wide possibilities and the OwnCloud-Client make me not to use this chars - this su***.

I think, transparency AND user decision can be a solution, until a server-sided solution is found, which will be difficult.

@p-bro

This comment has been minimized.

Show comment
Hide comment
@p-bro

p-bro Feb 10, 2014

Just to weigh in: Any limitations on the file names imposed by ownCloud (and not by the OS of the client) will reduce the user acceptance of the program. I don't want to rethink my file naming because of my cloud service, and actually renaming files is error-prone and an (in the user's perspective) unnecessary hassle.

I'd be fully satisfied with not downloading files with unsupported characters to a client and giving a notice to that effect. The file could then be retrieved from the web interface, if need be. But if the local file system supports a file name, it should be included in the sync.

I am not familiar how the files are stored on the server. If it's really unfeasible to deposit files there with characters not allowed on the server file system, a first workaround would be to also exclude those file names. But IMHO that would still only be a kludge: From a user perspective all legal file names should be synced, if the users wants to.

p-bro commented Feb 10, 2014

Just to weigh in: Any limitations on the file names imposed by ownCloud (and not by the OS of the client) will reduce the user acceptance of the program. I don't want to rethink my file naming because of my cloud service, and actually renaming files is error-prone and an (in the user's perspective) unnecessary hassle.

I'd be fully satisfied with not downloading files with unsupported characters to a client and giving a notice to that effect. The file could then be retrieved from the web interface, if need be. But if the local file system supports a file name, it should be included in the sync.

I am not familiar how the files are stored on the server. If it's really unfeasible to deposit files there with characters not allowed on the server file system, a first workaround would be to also exclude those file names. But IMHO that would still only be a kludge: From a user perspective all legal file names should be synced, if the users wants to.

@matiasbarone

This comment has been minimized.

Show comment
Hide comment
@matiasbarone

matiasbarone Apr 4, 2014

In my case I made a little script that make a copy of files with : and ? replacing thats characters for _ and allow sync the copy. Put it in a cronjob, and problem "solved".

# Load files that contain the folders
FILES=/home/myhome/.local/share/data/ownCloud/folders/*
# For each folder
for f in $FILES
do
 # obtain the path from the config file 
 . $f >/dev/null 2>&1
  path=$(echo -e "${localPath/\x/\u00}")
  echo "Processing $path Folder..."
 
 # Search in the path files with special characters
 find $path -type f -name '*[:?"]*' |
    while IFS= read -r; do
      #make the copy or overwrite if the original is newer than the copy
      cp --verbose -p -u "$REPLY" "${REPLY//[:?\"]/_}"
    done
done

In my case I made a little script that make a copy of files with : and ? replacing thats characters for _ and allow sync the copy. Put it in a cronjob, and problem "solved".

# Load files that contain the folders
FILES=/home/myhome/.local/share/data/ownCloud/folders/*
# For each folder
for f in $FILES
do
 # obtain the path from the config file 
 . $f >/dev/null 2>&1
  path=$(echo -e "${localPath/\x/\u00}")
  echo "Processing $path Folder..."
 
 # Search in the path files with special characters
 find $path -type f -name '*[:?"]*' |
    while IFS= read -r; do
      #make the copy or overwrite if the original is newer than the copy
      cp --verbose -p -u "$REPLY" "${REPLY//[:?\"]/_}"
    done
done
@xopxe

This comment has been minimized.

Show comment
Hide comment
@xopxe

xopxe May 15, 2014

I agree that he client should sync-up everything as is, and only download what it can. This is simple, while limiting the damage to the minimum. This can be easily explained to the final user (this os can not store files with the name you choosed), and seems legitimate. But saying "we won't handle this file because there is a system out there (whether you use it or not) that can not handle it" is just... Meh. I'm synching Linux to Linux, don't even have a Windows installation available, and this behavior is annoying as hell.
I certainly will not rename my files just for this, specially knowing that there are other options that don't force me to.

xopxe commented May 15, 2014

I agree that he client should sync-up everything as is, and only download what it can. This is simple, while limiting the damage to the minimum. This can be easily explained to the final user (this os can not store files with the name you choosed), and seems legitimate. But saying "we won't handle this file because there is a system out there (whether you use it or not) that can not handle it" is just... Meh. I'm synching Linux to Linux, don't even have a Windows installation available, and this behavior is annoying as hell.
I certainly will not rename my files just for this, specially knowing that there are other options that don't force me to.

@Bazon

This comment has been minimized.

Show comment
Hide comment
@Bazon

Bazon Jun 17, 2014

Another reason why this is bad:
If you save a page with Firefox, it automatically includes a"|", at least with Linux:
The name of the saved file and folder is "pagetitel | filename[.html]" by default.
So each and every saved webpage causes this trouble with owncloud. Works with Dropbox by the way.

Bazon commented Jun 17, 2014

Another reason why this is bad:
If you save a page with Firefox, it automatically includes a"|", at least with Linux:
The name of the saved file and folder is "pagetitel | filename[.html]" by default.
So each and every saved webpage causes this trouble with owncloud. Works with Dropbox by the way.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Oct 8, 2014

@sarusso
See the comment here: #854 (comment)

If this is for high priority for you feel free to contribute code to the core issue linked in the comment above.

ghost commented Oct 8, 2014

@sarusso
See the comment here: #854 (comment)

If this is for high priority for you feel free to contribute code to the core issue linked in the comment above.

@moscicki

This comment has been minimized.

Show comment
Hide comment
@moscicki

moscicki Oct 8, 2014

Contributor

There is lots of noise about this issue already but let me say that I completely agreed with the point of view of @p-bro (#854 (comment)).

We have a server running on unix and users who only use unix and the issue of not syncing ":" character is completely irrelevant for them (and not appreciated).

For the owncloud servers running on unix is this a problem? Or is the problem only because of owncloud servers running on windows (as the server's filesystem cannot support this)? If this is the only limitation then it may be overcome very easily: you query the server with status.php already so you can know if it supports a restricted character set or not. With this knowledge you either restrict the character set on the client (as it is done know = windows server) or lift up any restrictions and implement the simple logic proposed by @p-bro.

No renaming of filenames please or any such mess.

Contributor

moscicki commented Oct 8, 2014

There is lots of noise about this issue already but let me say that I completely agreed with the point of view of @p-bro (#854 (comment)).

We have a server running on unix and users who only use unix and the issue of not syncing ":" character is completely irrelevant for them (and not appreciated).

For the owncloud servers running on unix is this a problem? Or is the problem only because of owncloud servers running on windows (as the server's filesystem cannot support this)? If this is the only limitation then it may be overcome very easily: you query the server with status.php already so you can know if it supports a restricted character set or not. With this knowledge you either restrict the character set on the client (as it is done know = windows server) or lift up any restrictions and implement the simple logic proposed by @p-bro.

No renaming of filenames please or any such mess.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Oct 8, 2014

@moscicki

As the client can't handle this descision at the moment according to the linked comment above you really should discuss this in the core issue: owncloud/core#6102

ghost commented Oct 8, 2014

@moscicki

As the client can't handle this descision at the moment according to the linked comment above you really should discuss this in the core issue: owncloud/core#6102

@danimo

This comment has been minimized.

Show comment
Hide comment
@danimo

danimo Oct 8, 2014

Contributor

#2288 is a potential fix. Things to consider before merging:

  • We are treating : now special on mac (due to its special meaning, displayed as /in Finder)
  • We need to make sure that the server can handle it even on Windows
  • We need to check with different environments (Linux, Mac, Win), and see how it behaves.

Note: We will most probably not merge this into 1.7(.0), even though this merge request is against 1.7.

Contributor

danimo commented Oct 8, 2014

#2288 is a potential fix. Things to consider before merging:

  • We are treating : now special on mac (due to its special meaning, displayed as /in Finder)
  • We need to make sure that the server can handle it even on Windows
  • We need to check with different environments (Linux, Mac, Win), and see how it behaves.

Note: We will most probably not merge this into 1.7(.0), even though this merge request is against 1.7.

@sarusso

This comment has been minimized.

Show comment
Hide comment
@sarusso

sarusso Oct 8, 2014

I would love to contribute, but I am not a php developer and I cannot.

Please do not take me wrong, I am not asking that someone starts to work on this "for me". I am just pointing out an inconsistence: this software just does not do what it claims to do - sync your files. So either you guys fix this or you change the homepage from "With ownCloud you can sync & share your files" to "With ownCloud you can sync & share SOME of your files".

Come on, this behaviour is just ridiculus. People using this software is relying on it also for keeping a backup copy of their files, and then one discovers that it almost silently ignored some of them.

It has been discussed several times also involving @danimo, the server should accept any file name allowed on the clients native filesystems. Then maybe it will not send them to a client not capable of handling them but the server should always accept.

If this cannot be done for technical reasons, and the best thing possible is to accept only file names supported on the server filesystem, I think that it should be stated crystal clear.

Somethign like:

  • ownCloud server for windows fully supports only windows clients.
  • ownCloud server for OSX fully supports windows ans OSX clients.
  • ownCloud server for Linux fully supports any client

My 2 cents,
Stefano.

sarusso commented Oct 8, 2014

I would love to contribute, but I am not a php developer and I cannot.

Please do not take me wrong, I am not asking that someone starts to work on this "for me". I am just pointing out an inconsistence: this software just does not do what it claims to do - sync your files. So either you guys fix this or you change the homepage from "With ownCloud you can sync & share your files" to "With ownCloud you can sync & share SOME of your files".

Come on, this behaviour is just ridiculus. People using this software is relying on it also for keeping a backup copy of their files, and then one discovers that it almost silently ignored some of them.

It has been discussed several times also involving @danimo, the server should accept any file name allowed on the clients native filesystems. Then maybe it will not send them to a client not capable of handling them but the server should always accept.

If this cannot be done for technical reasons, and the best thing possible is to accept only file names supported on the server filesystem, I think that it should be stated crystal clear.

Somethign like:

  • ownCloud server for windows fully supports only windows clients.
  • ownCloud server for OSX fully supports windows ans OSX clients.
  • ownCloud server for Linux fully supports any client

My 2 cents,
Stefano.

danimo added a commit that referenced this issue Oct 9, 2014

@danimo

This comment has been minimized.

Show comment
Hide comment
@danimo

danimo Oct 9, 2014

Contributor

After checking with http://en.wikipedia.org/wiki/Filename#Reserved_characters_and_words and verifying the results on Linux and OS X, I've changed the patch to allow all these characters.

As I said, the server should be accepting all file names already (pending: check on Windows). That said, we have to be very sure not to break anything. That's why we have been rather conservative about the topic.

Contributor

danimo commented Oct 9, 2014

After checking with http://en.wikipedia.org/wiki/Filename#Reserved_characters_and_words and verifying the results on Linux and OS X, I've changed the patch to allow all these characters.

As I said, the server should be accepting all file names already (pending: check on Windows). That said, we have to be very sure not to break anything. That's why we have been rather conservative about the topic.

@FuturePilot

This comment has been minimized.

Show comment
Hide comment
@FuturePilot

FuturePilot Oct 12, 2014

If this isn't going to be fixed any time soon, the client needs to give some kind of error message that not all files have been synced. Yes I know you can check the Activity tab in the client but unless you're specifically looking, there is no way to tell something wasn't synced. (almost) Silently ignoring files is unacceptable as the user thinks everything is working correctly.

On a side note, my entire Owncloud environment is Linux so I'm irked that I am being restricted by Windows' limitations.

If this isn't going to be fixed any time soon, the client needs to give some kind of error message that not all files have been synced. Yes I know you can check the Activity tab in the client but unless you're specifically looking, there is no way to tell something wasn't synced. (almost) Silently ignoring files is unacceptable as the user thinks everything is working correctly.

On a side note, my entire Owncloud environment is Linux so I'm irked that I am being restricted by Windows' limitations.

@uvesten

This comment has been minimized.

Show comment
Hide comment
@uvesten

uvesten Oct 15, 2014

Agree with futurePilot, this needs to be more visible for the user!

uvesten commented Oct 15, 2014

Agree with futurePilot, this needs to be more visible for the user!

@dragotin

This comment has been minimized.

Show comment
Hide comment
@dragotin

dragotin Oct 15, 2014

Contributor

We had a more visible notification of that earlier and people complained, so we changed it to the way it is now, which shows the list in the activity tab in the setup page.

Contributor

dragotin commented Oct 15, 2014

We had a more visible notification of that earlier and people complained, so we changed it to the way it is now, which shows the list in the activity tab in the setup page.

@muetze-online

This comment has been minimized.

Show comment
Hide comment
@muetze-online

muetze-online Oct 16, 2014

I turned my back to Windows years ago - and now its got me by the balls :-). I don't want my system to be restricted by the problems of Windows and its filesystem.

I turned my back to Windows years ago - and now its got me by the balls :-). I don't want my system to be restricted by the problems of Windows and its filesystem.

@ghost ghost referenced this issue in owncloud/core Oct 27, 2014

Closed

colon or slash in filenames #11781

@axlblue

This comment has been minimized.

Show comment
Hide comment
@axlblue

axlblue Oct 27, 2014

I'm not a programmer.
My issue is now 'closed' but how do I fix my problem?
Can't the Mac client deal with this?

axlblue commented Oct 27, 2014

I'm not a programmer.
My issue is now 'closed' but how do I fix my problem?
Can't the Mac client deal with this?

@ghost ghost referenced this issue in owncloud/core Dec 4, 2014

Closed

Syncing issue`s from webinterface to Windows client #12590

@dragotin

This comment has been minimized.

Show comment
Hide comment
@dragotin

dragotin Dec 4, 2014

Contributor

Closing this with a link to https://github.com/owncloud/core/wiki/Cross-Platform-File-Handling which describes how we will tackle this problem once we're at it.

Contributor

dragotin commented Dec 4, 2014

Closing this with a link to https://github.com/owncloud/core/wiki/Cross-Platform-File-Handling which describes how we will tackle this problem once we're at it.

@dragotin dragotin closed this Dec 4, 2014

@Bazon

This comment has been minimized.

Show comment
Hide comment
@Bazon

Bazon Dec 30, 2014

Just for the protocol another case where this leads to trouble:
Pictures created by the Android Hangouts App get names like "IMG_20141230_123250:nopm:.jpg".
The start comes from the Camera-App, Hangouts adds ":nopm:".
--> These files can't be shared via OwnCloud in an easy way.

Bazon commented Dec 30, 2014

Just for the protocol another case where this leads to trouble:
Pictures created by the Android Hangouts App get names like "IMG_20141230_123250:nopm:.jpg".
The start comes from the Camera-App, Hangouts adds ":nopm:".
--> These files can't be shared via OwnCloud in an easy way.

@unamedplayer

This comment has been minimized.

Show comment
Hide comment
@unamedplayer

unamedplayer Mar 6, 2015

Hello, just adding my voice to this. Out of curiosity I've done some testing with Google Drive and they handle all kinds of odd characters via translation. The file names may not be consistent across filesystems but no data is left unsynced which is very important IMHO.

Thank you for all the comments and pointing out the path forward.

Hello, just adding my voice to this. Out of curiosity I've done some testing with Google Drive and they handle all kinds of odd characters via translation. The file names may not be consistent across filesystems but no data is left unsynced which is very important IMHO.

Thank you for all the comments and pointing out the path forward.

@danimo

This comment has been minimized.

Show comment
Hide comment
@danimo

danimo Mar 6, 2015

Contributor

fyi: This is being worked on for ownCloud Server 8.1 on a broader scope.

Contributor

danimo commented Mar 6, 2015

fyi: This is being worked on for ownCloud Server 8.1 on a broader scope.

@unamedplayer

This comment has been minimized.

Show comment
Hide comment
@unamedplayer

unamedplayer Mar 6, 2015

Yay :) Thank you!

Yay :) Thank you!

@unamedplayer

This comment has been minimized.

Show comment
Hide comment
@unamedplayer

unamedplayer Jun 24, 2015

@danimo , do you have any fresh info on this by any chance? I don't mean to pester but I've been keeping a close eye on all beta releases, we are now well into 8.1 and I don't see any indication that this is being addressed at the moment. Which is fine I'm not complaining :) but I'm wondering about current state of things is and also what you meant by "broader scope". Thank you!

@danimo , do you have any fresh info on this by any chance? I don't mean to pester but I've been keeping a close eye on all beta releases, we are now well into 8.1 and I don't see any indication that this is being addressed at the moment. Which is fine I'm not complaining :) but I'm wondering about current state of things is and also what you meant by "broader scope". Thank you!

@AshleyYakeley

This comment has been minimized.

Show comment
Hide comment
@AshleyYakeley

AshleyYakeley Nov 20, 2015

What's the status of this? The issue is closed, but it still appears to be a problem in 8.1?

What's the status of this? The issue is closed, but it still appears to be a problem in 8.1?

@brevilo

This comment has been minimized.

Show comment
Hide comment
@brevilo

brevilo Nov 20, 2015

If I'm not mistaken it takes client 2.1 (soon to be released) for this to work with the 8.1 server...

brevilo commented Nov 20, 2015

If I'm not mistaken it takes client 2.1 (soon to be released) for this to work with the 8.1 server...

@DCKcode

This comment has been minimized.

Show comment
Hide comment
@DCKcode

DCKcode May 29, 2016

This bug still affects me. I'm on OwnCloud 9.0.2 server (ZFS on FreeBSD) with client 2.1.1 on OS X. Files with colons in their name don't sync, rejected with a 400 error by the server. This bug has been present in all earlier versions I've used too. Has this really been resolved?

If possible I'd like to help debugging the problem.

DCKcode commented May 29, 2016

This bug still affects me. I'm on OwnCloud 9.0.2 server (ZFS on FreeBSD) with client 2.1.1 on OS X. Files with colons in their name don't sync, rejected with a 400 error by the server. This bug has been present in all earlier versions I've used too. Has this really been resolved?

If possible I'd like to help debugging the problem.

@dragotin

This comment has been minimized.

Show comment
Hide comment
@dragotin

dragotin May 30, 2016

Contributor

@DeepDiver1975 isn't that supposed to be fixed with server 9.0.x? The client ignores these filenames only on win32.

Contributor

dragotin commented May 30, 2016

@DeepDiver1975 isn't that supposed to be fixed with server 9.0.x? The client ignores these filenames only on win32.

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 May 30, 2016

Member

@DeepDiver1975 isn't that supposed to be fixed with server 9.0.x? The client ignores these filenames only on win32.

yes - the server is accepting file names depending on the server side filesystem.
On a posix server system only \ and / are not allowed - see https://github.com/owncloud/core/blob/master/lib/private/Files/Storage/Common.php#L519

Member

DeepDiver1975 commented May 30, 2016

@DeepDiver1975 isn't that supposed to be fixed with server 9.0.x? The client ignores these filenames only on win32.

yes - the server is accepting file names depending on the server side filesystem.
On a posix server system only \ and / are not allowed - see https://github.com/owncloud/core/blob/master/lib/private/Files/Storage/Common.php#L519

@DeepDiver1975

This comment has been minimized.

Show comment
Hide comment
@DeepDiver1975

DeepDiver1975 May 30, 2016

Member

and this is implemented since 8.1

Member

DeepDiver1975 commented May 30, 2016

and this is implemented since 8.1

@erhan-

This comment has been minimized.

Show comment
Hide comment
@erhan-

erhan- Jun 1, 2016

I have the same issue on a debian jessie server with ubuntu 12.04 lts client:

|0|path/file:name.pdf|INST_NEW|Up|145795xxxx||3235xxx||6|The item is not synced because of previous errors: Error downloading https://web/remote.php/webdav/path/file:name.pdf - server replied: Internal Server Error (Invalid request for Invalid path, ":" is not allowed (InvalidPathException))|0|0|0|||INST_NONE|

erhan- commented Jun 1, 2016

I have the same issue on a debian jessie server with ubuntu 12.04 lts client:

|0|path/file:name.pdf|INST_NEW|Up|145795xxxx||3235xxx||6|The item is not synced because of previous errors: Error downloading https://web/remote.php/webdav/path/file:name.pdf - server replied: Internal Server Error (Invalid request for Invalid path, ":" is not allowed (InvalidPathException))|0|0|0|||INST_NONE|

@a-schild

This comment has been minimized.

Show comment
Hide comment
@a-schild

a-schild Jun 1, 2016

@erhan- what owncloud server & client versions are you using?

a-schild commented Jun 1, 2016

@erhan- what owncloud server & client versions are you using?

@erhan-

This comment has been minimized.

Show comment
Hide comment
@erhan-

erhan- Jun 3, 2016

Server: 9.0.2
Client: 2.2.0

Had multiple illogical problems with one specific client after client update to 2.2.0 but it all resolved after removing the account from the client and readding it again, resyncing in a new folder. Consider my problem as solved.

erhan- commented Jun 3, 2016

Server: 9.0.2
Client: 2.2.0

Had multiple illogical problems with one specific client after client update to 2.2.0 but it all resolved after removing the account from the client and readding it again, resyncing in a new folder. Consider my problem as solved.

@mckaygerhard

This comment has been minimized.

Show comment
Hide comment
@mckaygerhard

mckaygerhard Jan 10, 2018

i used and have that problem, but moved to nextcloud and not any more. puff oc

i used and have that problem, but moved to nextcloud and not any more. puff oc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment