Speed up sync by keeping etag history

[icewind1991]

As a way to speed up the sync client getting all the changes made on the server it might be an idea to keep the history of etags for the root folder of each user and the time that etag was generated.

With such a history in place the sync client could send a request to the server with it's own last known etag of the root folder which the server can use to easily get a list of all changed files and folder on the server since the last sync. If the server sends a fill of all changed files with their new etags and the new etags of all parent folders a download-only sync can be done with 0 propfinds, leaving only a single request overhead instead of having to run a propfind against all changed folders.

The query for listing all changes made only takes, after creating a new index, around 1 ms with 15000 files in the filecache in my test DB.

cc @dragotin @DeepDiver1975 @karlitschek @bartv2

[PVince81]

So if I understand well each "revision" of that history is like a snapshot of the file list tree or its changes from the last one (not the file contents) ?
Interesting idea.

[icewind1991]

My idea was to have a table of fileid, time and etag (which can be null for deleted files) and add a row every time a file changes.

Then if the client send the last root etag that it has locally the server has enough information to calculate all changed, new and deleted files since the last sync

[DeepDiver1975]

This would allow us to perform change tracking and should deliver enough information to implement the webdav sync protocol https://tools.ietf.org/html/rfc6578

But this requires major rework on the client - no idea if we should walk this path.

@dragotin @danimo @MTRichards @karlitschek @craigpg

[PVince81]

This logic looks similar to how the Dropbox API works, it also accepts a hash as input. But I believe it only does it for a single directory listing.

[karlitschek]

Interesting idea. We should look into this in the long run to improve performance. Let's look into this for 8

[DeepDiver1975]

@dragotin @danimo something to think about these days?

[PVince81]

This sounds like the way how Webdav sync works.

[PVince81]

If we have the change history in one place it could also be used as replacement for the activity table (or at least the file-based ones)