Update and re-sort pull queue when new index updates arrive

[AudriusButkevicius]

Or send a specific message IndexUpdateFinished to notify that there won't be any more index updates coming.

The core issue is the fact that even though we predefine some pull order, we send small indexes of 1-2 files as we scan and discover stuff, making the pull order obsolete.

This should also tie into the scanner, essentially ideally not sending any updates until the scan is actually finished, so that we'd send the whole set of changes discovered in one go.

[calmh]

While I agree that this limits the usefulness of the sorting in some cases I think most users prefer starting the sync quicker instead. But if a dive did know when the other side is "done scanning" it could of course decide to wait for that based on some config.

Doesn't tie in very cleanly with inotify.

[calmh]

Another option would be to "just" live resort and reprioritize when updates come in. Potentially cancelling sync of one file, leaving the temp file in place, and moving to another. I think this could be done fairly cleanly in my new puller whenever that sees the light of day.

[AudriusButkevicius]

Ok, let's do that instead.

[scienmind]

Slightly off-topic:

Regarding "leaving temp files in place" , few times I had unfinished temps littering the target folder, and not cleaning themselves:

1 - source node connects & starts sync
2 - temp created on destination device
3 - source disconnects
4 - source renames/removes the file
5 - now ~temp files are left in the target folder forever
(somehow now ST treats those temps on the destination device as actual files to be preserved and synced)

I'm not sure it it's a bug on its own, or a corner case of a desired behavior, in case it's the latter, leaving longer time frame for temps to lay around will potentially give birth to more such "obsolete" temps.

[AudriusButkevicius]

They get removed after 24 hours by default. I can't imagine syncthing treating them as normal files if they have the syncthing specific patterns in them. Perhaps it can happen if you dual boot (as the pattern is different, but I am not sure).

[scienmind]

That actually makes sense. I guess each time I spotted such a "leftover", I just never waited long enough for 24 hours to pass, and always removed them manually after a shorter period.

But if that is the way it cleans up, I don't see any real problem there.

[calmh]

We leave them around for 24 hours because they presumably contain some data we wanted, and we can reuse those blocks from the temp file instead of downloading them again.