New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Devices with ignored files stay "syncing" forever #623

Closed
calmh opened this Issue Sep 4, 2014 · 20 comments

Comments

Projects
None yet
10 participants
@calmh
Member

calmh commented Sep 4, 2014

To handle the case where a node ignores a bunch of files, we should not assume a node is "syncing" just because it doesn't have all files we think it should have. It should be based on something else, like whether we've seen a file request from that node in the last x seconds or similar.

@calmh calmh added the bug label Sep 4, 2014

@hadogenes

This comment has been minimized.

Show comment
Hide comment
@hadogenes

hadogenes Sep 10, 2014

Maybe it would be better if the other node inform us about his syncing state.

hadogenes commented Sep 10, 2014

Maybe it would be better if the other node inform us about his syncing state.

@calmh calmh modified the milestone: v1.0-maybe Jan 15, 2015

@vext01

This comment has been minimized.

Show comment
Hide comment
@vext01

vext01 Apr 1, 2015

Just hit the problem where stignore screws up the sync status. It references this bug:
https://github.com/syncthing/syncthing/wiki/Ignoring-Files

Indeed, it would be nice if it were fixed.

vext01 commented Apr 1, 2015

Just hit the problem where stignore screws up the sync status. It references this bug:
https://github.com/syncthing/syncthing/wiki/Ignoring-Files

Indeed, it would be nice if it were fixed.

@withinboredom

This comment has been minimized.

Show comment
Hide comment
@withinboredom

withinboredom Jun 10, 2015

Is this really a bug? If a node ignores 75% of another node's files, it would be correct that it was only 25% synced from one reference while 100% from another.

withinboredom commented Jun 10, 2015

Is this really a bug? If a node ignores 75% of another node's files, it would be correct that it was only 25% synced from one reference while 100% from another.

@calmh

This comment has been minimized.

Show comment
Hide comment
@calmh

calmh Jun 11, 2015

Member

Yeah that's fine. We just shouldn't claim that it's "syncing" when in reality it's just sitting there rolling it's thumbs, happy in it's own little view of the world.

Member

calmh commented Jun 11, 2015

Yeah that's fine. We just shouldn't claim that it's "syncing" when in reality it's just sitting there rolling it's thumbs, happy in it's own little view of the world.

@calmh

This comment has been minimized.

Show comment
Hide comment
@calmh

calmh Sep 3, 2016

Member

Given that we don't keep track of other devices' activity, and can't do so in the general case, would it be a valid bandaid over this to simply rename the state? Instead of "Syncing" (that implies we know something about what the device is doing), call it "Partially in sync" or something equivalent (that only says something that we know)?

Member

calmh commented Sep 3, 2016

Given that we don't keep track of other devices' activity, and can't do so in the general case, would it be a valid bandaid over this to simply rename the state? Instead of "Syncing" (that implies we know something about what the device is doing), call it "Partially in sync" or something equivalent (that only says something that we know)?

@AudriusButkevicius

This comment has been minimized.

Show comment
Hide comment
@AudriusButkevicius

AudriusButkevicius Sep 3, 2016

Member

I wonder if we should add a new message type and publish someof the events to the rest of the peers, just like the events api.

Member

AudriusButkevicius commented Sep 3, 2016

I wonder if we should add a new message type and publish someof the events to the rest of the peers, just like the events api.

@calmh calmh modified the milestones: Planned, Unplanned (Contributions Welcome) Dec 29, 2016

@calmh calmh self-assigned this Dec 29, 2016

@dgnuff

This comment has been minimized.

Show comment
Hide comment
@dgnuff

dgnuff Mar 22, 2017

Forgive me if this cannot be implemented for other reasons, but I believe there is a possible solution to this problem.

Maybe it should be optional because the case that withinboredom points out may be valid some of the time.

In my instance I have three machines, call them Alice, Bob and Charles. They all share a folder, but Alice ignores some files that Bob and Charles don't. As reported, this leads to Bob and Charles showing Alice as permanently syncing.

If Bob and Charles knew about Alice's ignore list, then they could figure out the list of files that are actually shared. They would evaluate a combined ignore list from the union of Alice's ignore list and theirs; from this the shared file list would be the full set of files minus any that are in the combined ignore list.

The problem with this is that right now, a node only has to maintain one ignore list per shared folder. If this were implemented, it would have to maintain one ignore list for each (shared folder, machine) pair; this could mean a lot more data stored, especially if the number of machines in the cluster is large, and depending on the data requirements of an ignore list.

dgnuff commented Mar 22, 2017

Forgive me if this cannot be implemented for other reasons, but I believe there is a possible solution to this problem.

Maybe it should be optional because the case that withinboredom points out may be valid some of the time.

In my instance I have three machines, call them Alice, Bob and Charles. They all share a folder, but Alice ignores some files that Bob and Charles don't. As reported, this leads to Bob and Charles showing Alice as permanently syncing.

If Bob and Charles knew about Alice's ignore list, then they could figure out the list of files that are actually shared. They would evaluate a combined ignore list from the union of Alice's ignore list and theirs; from this the shared file list would be the full set of files minus any that are in the combined ignore list.

The problem with this is that right now, a node only has to maintain one ignore list per shared folder. If this were implemented, it would have to maintain one ignore list for each (shared folder, machine) pair; this could mean a lot more data stored, especially if the number of machines in the cluster is large, and depending on the data requirements of an ignore list.

@calmh

This comment has been minimized.

Show comment
Hide comment
@calmh

calmh Mar 23, 2017

Member
Member

calmh commented Mar 23, 2017

@dgnuff

This comment has been minimized.

Show comment
Hide comment
@dgnuff

dgnuff Mar 23, 2017

Thank you for your quick response.

Great to hear that you've done this. I'll be watching for an update.

dgnuff commented Mar 23, 2017

Thank you for your quick response.

Great to hear that you've done this. I'll be watching for an update.

@mannp

This comment has been minimized.

Show comment
Hide comment
@mannp

mannp Apr 6, 2017

Excellent news, thanks for this :) will it come out in a release candidate first?

mannp commented Apr 6, 2017

Excellent news, thanks for this :) will it come out in a release candidate first?

@calmh

This comment has been minimized.

Show comment
Hide comment
@calmh

calmh Apr 7, 2017

Member
Member

calmh commented Apr 7, 2017

@mannp

This comment has been minimized.

Show comment
Hide comment
@mannp

mannp Apr 7, 2017

Thats great news there is a better way, though thanks for your efforts, but is it still on the same release time frame ;-)

mannp commented Apr 7, 2017

Thats great news there is a better way, though thanks for your efforts, but is it still on the same release time frame ;-)

@calmh

This comment has been minimized.

Show comment
Hide comment
@calmh

calmh Apr 7, 2017

Member
Member

calmh commented Apr 7, 2017

@mannp

This comment has been minimized.

Show comment
Hide comment
@mannp

mannp Apr 7, 2017

Oh thats a shame. Could your way be incorporated as an option and updated later, rather than having the confusing scenario there is now?

Loving the product, but as a noob the x% complete scenario is still confusing :)

mannp commented Apr 7, 2017

Oh thats a shame. Could your way be incorporated as an option and updated later, rather than having the confusing scenario there is now?

Loving the product, but as a noob the x% complete scenario is still confusing :)

@calmh calmh removed their assignment Apr 8, 2017

@jsacrist

This comment has been minimized.

Show comment
Hide comment
@jsacrist

jsacrist Oct 17, 2017

The better way is not written yet, so it is not planned for any release.

Is it not written because of lack of developers? I've never contributed to syncthing but I'd happily volunteer for this issue.

jsacrist commented Oct 17, 2017

The better way is not written yet, so it is not planned for any release.

Is it not written because of lack of developers? I've never contributed to syncthing but I'd happily volunteer for this issue.

@imsodin

This comment has been minimized.

Show comment
Hide comment
@imsodin

imsodin Oct 17, 2017

Member

I have something in the works, that solves this. It isn't yet a PR, see this discussion: https://forum.syncthing.net/t/device-completion-and-ignores/10765
If you beat me to it, then kudos to you ;)
Also there is plenty of other good stuff yet to be implemented just waiting for you, just ask on the forum if you want pointers.

Member

imsodin commented Oct 17, 2017

I have something in the works, that solves this. It isn't yet a PR, see this discussion: https://forum.syncthing.net/t/device-completion-and-ignores/10765
If you beat me to it, then kudos to you ;)
Also there is plenty of other good stuff yet to be implemented just waiting for you, just ask on the forum if you want pointers.

@AudriusButkevicius

This comment has been minimized.

Show comment
Hide comment
@AudriusButkevicius

AudriusButkevicius Oct 17, 2017

Member

There is a ton of easy tickets under the first issue or something like that label in the tracker if you have the appetite.

Member

AudriusButkevicius commented Oct 17, 2017

There is a ton of easy tickets under the first issue or something like that label in the tracker if you have the appetite.

imsodin added a commit to imsodin/syncthing that referenced this issue Oct 24, 2017

all: add invalid/ignored files to global list (fixes syncthing#623)
This allows to discern ignored files from missing files on remote devices to
determine accurate completion status.

@st-review st-review closed this in c080f67 Nov 11, 2017

calmh added a commit to calmh/syncthing that referenced this issue Nov 11, 2017

Merge branch 'master' into noreplace
* master:
  all: Add invalid/ignored files to global list, announce to peers (fixes syncthing#623)
  lib/watchaggregator: Don't care about timings during testing on darwin
  lib/model: Incremental block stats usage reporting
@AudriusButkevicius

This comment has been minimized.

Show comment
Hide comment
@AudriusButkevicius

AudriusButkevicius Nov 11, 2017

Member

Is this actually solved? The ticket talks about something else.

Member

AudriusButkevicius commented Nov 11, 2017

Is this actually solved? The ticket talks about something else.

@calmh

This comment has been minimized.

Show comment
Hide comment
@calmh

calmh Nov 11, 2017

Member

Yeah I know and I thought the same... And technically it's unrelated, but the only example of the issue that is mentioned in this ticket is that devices with ignores look like they are "99% syncing" forever while they are not syncing... And that is solved by the PR.

Member

calmh commented Nov 11, 2017

Yeah I know and I thought the same... And technically it's unrelated, but the only example of the issue that is mentioned in this ticket is that devices with ignores look like they are "99% syncing" forever while they are not syncing... And that is solved by the PR.

@Xeenych

This comment has been minimized.

Show comment
Hide comment
@Xeenych

Xeenych Nov 21, 2017

I have a shared folder A with some stuff and subfolder B with hundred thousand of small files (tile cache of web map). Syncthing takes enormous amount of memory and time to scan and rescan A with B. So I add B to ignore and all is ok, because ST does not index ignored files (as opposite to BTSync).

Looks like adding "global ignore list" or sending ignored files to other peers would need ST to scan ignored files and take large amounts of memory and time in my case.

Xeenych commented Nov 21, 2017

I have a shared folder A with some stuff and subfolder B with hundred thousand of small files (tile cache of web map). Syncthing takes enormous amount of memory and time to scan and rescan A with B. So I add B to ignore and all is ok, because ST does not index ignored files (as opposite to BTSync).

Looks like adding "global ignore list" or sending ignored files to other peers would need ST to scan ignored files and take large amounts of memory and time in my case.

@calmh calmh modified the milestones: Planned, v0.14.41 Dec 3, 2017

@calmh calmh changed the title from "Syncing" status should be based on activity, not completion percentage to Devices with ignored files stay "syncing" forever Dec 3, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment