Upload after download when using FAT32 or exFAT storage.

[VincentvgNn]

Description

I have to connect clients that are quit inexperienced computer users. To prevent that they give away the OC shared data when they change their PC, I have decided to have the shared data stored on USB memory sticks. For interchangeability reasons this type of memory is by default formatted in FAT32.
In FAT32 the file modification time (mtime) is unfortunately stored as a local time with a time resolution of 2 secs.

This causes 2 problems:

1. Twice a year there is a one hour DST change. This causes OC detecting an mtime difference of precisely +3600 or -3600 secs. The result is that all clients with the FAT32 storage will start uploading all files again.
   This issue has meanwhile been solved in issue Files from FAT32 volume are uploaded again after Daylight Saving Time (DST) change #2438 by ignoring a local mtime change of not only 0 secs, but also ignoring a difference of exactly +3600 and -3600 secs.
   It was decided not to have this +3600 and -3600 secs ignorance optionally set by a check box in the client's "General Settings" menu.
2. The mtime resolution on the server is 1 sec. So when a client is uploading a file e.g. from an NTFS file system, it can happen that files are having an odd number of mtime secs. Another connected client, using FAT32 and getting these files downloaded, will store them with an even number of secs by rounding up by 1 sec. Next the change of the mtime by +1 sec will be detected resulting in re-uploading of these files with the new mtime in even seconds.

Possible solutions for the "even seconds" problem

1. Personally I do not worry about all files getting even seconds for the mtime. So I could first store new files on a FAT32 storage before putting them in the folders to be synced by OC.
   However, users not knowing about this FAT32 problem will sooner or later be confronted with the re-uploading of files. Therefore the OC client could be modified such that mtimes with odd seconds will always be rounded up to even seconds when storing the files.
   But is this acceptable for all OC users?
2. Second solution: let the OC client detect in a smart way that the mtime has been rounded up by +1 sec into an even value and then ignore this as a change.
   This comes on top of the accepted +3600 or -3600 seconds offset needed for the DST.
   So the local mtime change is detected as follows:
   a. mtime change 0 secs --> file did not change (solved in issue 2438)
   b. mtime change +3600 secs --> file did not change (solved in issue 2438)
   c. mtime change -3600 secs --> file did not change (solved in issue 2438)
   d. mtime change +1 sec and the new mtime is even --> file did not change
e. mtime change +3601 secs and the new mtime is even --> file did not change
f. mtime change -3599 secs and the new mtime is even --> file did not change
g. mtime change has another value --> the file has been changed
3. I tested exFAT as a possible alternative that supports Windows, Mac OS X (since 2009) and Linux (via FUSE or non-mainline kernel driver). It solves the timezone problem of FAT32.
   Unfortunately the time stamp seconds for the mtime are still put into 4 bits (http://ntfs.com/exfat-time-stamp.htm), so the time resolution is still in even seconds.

Change request

1. Does the OC team agree that this re-upload after download problem needs to be solved?
   If yes, can solution 1 (always only even seconds) or solution 2d, 2e, 2f, 2g be implemented?
   Or are there other options?
2. Can the ignored values for the mtime change related to FAT32, exFAT, DST and timezone changes briefly be documented in the client's manual?

Steps to reproduce

1. Have client 1, using NTFS, synchronized with client 2, using FAT32 or exFAT.
2. Copy a file with odd secs for the mtime into a shared folder at client 1.
   The file will be uploaded to the server and downloaded from the server to client 2.
   At client 2 the mtime will increase by 1 sec and the file will synced back to client 1.
3. Thew effect of DST changes can be simulated by changing the timezone setting at client 2 one or more hours forward or backward.
   On FAT32 +/- 1 hour will be neglected. More hours will result in re-uploading all files.
   On exFAT changing over more timezones will not give any re-uploading.

[guruz]

CC @ogoffart

[VincentvgNn]

@ogoffart @guruz
The new software modification seems to be quite smart. Not giving away a +1 sec additional play on the local mtime and preventing an unwanted re-upload of the files that got +1 sec on their mtime.
But, .... what happens if the client get's files copied via another path before syncing them via the cloud or if the client resets (removes) the local database to get a resync of all. Then the files have locally no more etag and will be compared again with the files on the server based on mtime!
I tested this and saw the conflict between local files that got the mtime changed by +1 sec. All these files were re-uploaded and via the cloud copied back to the source client that uses NTFS. Above, from all these files, statistically 50% of them, a conflict version was stored on the server.
Files originally having an even number of seconds do not have this problem.
Is my reasoning correct?

What's the opinion of the OC design team?
Should the FAT32/exFAT option be 100% OK or should it be abandoned?
I still prefer having a check box to switch the +/- 3600 sec play and the _conditional_ +1 sec play on/off such that most users are never bothered by the FAT32/exFAT problem and just get the original 1 sec resolution without any exception.

[moscicki]

What about checking the signature of the partition and enable these particular mtime tricks only for sync folders sitting on FAT32? The only caveat of this approach in general are cross-mounted partitions (e.g. accessing a windows partition from linux).

[ogoffart]

what happens if the client get's files copied via another path before syncing them via the cloud

I don't see how this is a problem.

or if the client resets (removes) the local database to get a resync of all.

This is not supported.

a conflict version was stored on the server.

That's not possible.

[VincentvgNn]

Most synchronizing programs are facing the same issue.Examples:

Second Copy:
http://www.secondcopy.com/kb/article.aspx?id=10169 (By design Second Copy ignores the 2-second time difference if the two file systems are not the same.)
http://www.secondcopy.com/kb/article.aspx?id=13299 (IgnoreDST=1)

Beyond Compare has the following options to fine-tune the timestamp criteria:

[VincentvgNn]

@ogoffart

Closed? Thanks a lot for solving so far.

I don't see how this is a problem.

In case of Syncing "manually" a large portion of files like discussed at: https://forum.owncloud.org/viewtopic.php?f=24&t=20656