Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Next Gen Ignores: Requirements #2491

Open
calmh opened this issue Nov 19, 2015 · 71 comments

Comments

Projects
None yet
@calmh
Copy link
Member

commented Nov 19, 2015

This issue replaces the ones that relate to fundamental issues with how our ignores work. The following is the up to date list of requirements:

Certain Requirements

  • Must be understandable by someone who understands the interface in use by DropBox, OneDrive, etc.
  • Must be a simple tree view with checkboxes
    • Root of tree (=folder root) checked by default
    • Can uncheck root and then check just the branches that we want synced
  • Must be able to black list extensions
  • Must live in config (not .stignore files)
  • Must "live update". That is, if I ignore a file that was going to be pulled, we should not pull it. If I unignore a file that we are missing, it should be pulled shortly thereafter.
  • Must be able to specify if ignored files should be preserved or removed on directory delete.
    • Per what though? File extension? Directory?
  • Power user access to flexible, underlying patterns
  • Be able to to remove files for folders that are not picked, as it remove things that are now ignored.
  • Have inclusion and exclusion mode, ignore everything, include X, versus include everything, ignore X.

Maybe?

  • Per directory file extension black lists ("never sync *.tmp")
  • Per directory file extension while lists ("only sync *.doc")
  • Should sync between devices
  • Ability to ignore a directory by the presence of a special file in it (cachedir.tag).

Comments and discussion below. A project maintainer will keep the list above in sync as the requirements change or are clarified. Note that this is is not a list of problems with the current system, but requirements for a new system.

@crccheck

This comment has been minimized.

Copy link

commented Nov 19, 2015

Why not also keep .stignore, but make it sync between devices as .gitignore and .hgignore do? That way, there's a developer friendly way to keep files out. As someone who had to delete 100k node_modules files yesterday (not to mention .DS_Store, packages, build, .tox, etc), that would have save me 3 gigs of bandwidth and time.

There's two distinct user stories:

  1. As a user to Syncthing who also knows Dropbox, I want a familiar interface for selectively syncing files so I only sync what I care about
  2. As a developer using Syncthing, I want a familiar interface for selectively syncing files globally so I only sync what I care about without additional set up every new install.
@calmh

This comment has been minimized.

Copy link
Member Author

commented Nov 19, 2015

A .fooignore file might come naturally to developers, but that's really the only demographic that expects a hidden file with a bizarre extension (think Windows). I think it's a developer accident that it exists at all in it's current form today - it's really a piece of folder configuration as anything else. Not having it in the folder itself also works better with read only folders - perhaps you want to share the contents of a DVD-ROM... (This also requires handling the non-existence of .stfolder, but there's a ticket for that.)

Whether it syncs or not is really somewhat beside the point of the existence of the file, but I do think there should be at least an option to sync the patterns, yes.

(You can do that today with a little #include hackery as well.)


To bring it more in line with the actual ticket; what's the actual requirement that having an .stignore on disk fulfills?

@cdhowie

This comment has been minimized.

Copy link
Contributor

commented Nov 19, 2015

I don't care where the ignore data is stored, but patterns allow me to express ignores in a concise manner and with more flexibility than a tree option. Please at least retain this functionality, if not in a file called .stignore then somewhere else. None of the options provided in this ticket summary will meet my needs.

If the current pattern-based ignore option is removed I'm afraid I may have to stop upgrading Syncthing, as versions without pattern-based ignore wouldn't meet my needs.

@calmh

This comment has been minimized.

Copy link
Member Author

commented Nov 19, 2015

@cdhowie What are the needs, that are not covered by the tree options and the extension white/black lists?

@cdhowie

This comment has been minimized.

Copy link
Contributor

commented Nov 19, 2015

There is no accommodation for prefixes in the current proposal. I can't be the only one with a ._* ignore.

Neither is there a way to express a pattern along the lines of "ignore all objects named bin under directory foo" whereas we can do that today with

/foo/bin
/foo/**/bin

I'm not asking you not to implement a nicer interface for new users, I'm asking you not to sacrifice the power users on the altar of the masses.

@calmh

This comment has been minimized.

Copy link
Member Author

commented Nov 19, 2015

Well, currently the masses are being lead to the slaughter en masse as they try to work the ignore system, so some sort of balance there is required. :D

The prefix thing is a good point, and should probably be supported somehow. I need to think about the last one...

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Nov 19, 2015

So the two could coexist, but power users will have to grasp their concepts around about the order in which it gets executed, etc.

@calmh

This comment has been minimized.

Copy link
Member Author

commented Nov 19, 2015

Well, yes. At some point whatever the user is configuring, graphically or text based or however, is going to to get compiled into some list of match objects of some kind. Today that's a list of regexps, unfortunately that's a bit inefficient to evaluate. But something like that; whatever it ends up being could of course be exposed in some way so that they can be input directly into the config or something.

@cdhowie

This comment has been minimized.

Copy link
Contributor

commented Nov 19, 2015

whatever it ends up being could of course be exposed in some way so that they can be input directly into the config or something.

I would absolutely be happy with that resolution, if there is a (somewhat) easy way to edit this in the GUI. Like an "advanced" button at the bottom of the simple UI that opens the list of compiled patterns, letting me fiddle with them.

(I'd be willing to contribute dev time to making that happen, as this is an important feature for me.)

@calmh

This comment has been minimized.

Copy link
Member Author

commented Nov 19, 2015

Possibly. The way I could envision it in that case is that you'll probably have a simple default layout with a tree view and a list of extensions or something. Switch to advanced mode and you get the compiled patterns or something and can edit them at will. But once you've done that, you probably won't be able to switch back, as what you created in the advanced mode might not be expressible in the simple mode.

@cdhowie

This comment has been minimized.

Copy link
Contributor

commented Nov 19, 2015

But once you've done that, you probably won't be able to switch back, as what you created in the advanced mode might not be expressible in the simple mode.

Same thing I was thinking. Or, you can switch back, but you'll get a warning that your manually-modified patterns will get destroyed if you do.

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Nov 19, 2015

I think we can keep what we have today and keep it the same way it is today, just add a thing before ignores which is the dir selector.

@calmh

This comment has been minimized.

Copy link
Member Author

commented Nov 19, 2015

Okay I added that to the requirements

@cdhowie

This comment has been minimized.

Copy link
Contributor

commented Nov 19, 2015

Thanks @calmh. If you are in need of dev power to implement this feature let me know. I don't have a lot of time but I'm willing to spare some for ST.

@mkroehnert

This comment has been minimized.

Copy link

commented Nov 19, 2015

@crccheck keep in mind that storing .stignore files inside the directories to be synced leaks metadata if you copy the directory to a USB stick for example.
You might not want other people to know which program you are using for synchronization and which files are ignored.

@Schroedingers-Cat

This comment has been minimized.

Copy link

commented Nov 20, 2015

I think it is important to differentiate between ignoring as part of setting up a new repository and ignoring as part of setting up a new node.
When you create a new repository and want to exclude f.e. all binaries prefix_*.extension from the subdirectory /foo/bin, you don't want to see these files on any new node you sync this folder to. That's why exclusions expressed at this state as part of the folder setup should be considered an essential information and thus be synced automatically to any new node. What is excluded at this stage should never get any attention by syncthing (for that repository).
In contrast to that is ignoring as part of setting up a new node. Users likely have a device to which they don't want to sync the entire repository but only a specific subfolder. These information are device specific and should only be cared about on that node.
To summarize, ignoring at the repository level is some global filter that every device applies while ignoring at the node level is a filter that is specific to that device only.

Ignoring at the node level is probably what most users want as this is what Dropbox/Onedrive etc. offer. You have a tree list and uncheck a folder/file you don't want to download to that node. A simple interface with a folder tree, maybe even a file list, with checkboxes should suffice.
Ignoring at the repository level is what advanced users do. They care about not syncing Thumbs.db or some config.dat files of a subdirectory containing device-specific information like it's screen resolution. Even something simple like excluding files based on their extension means that they activated the option "show file extensions" in the explorer (which is what most users don't do). I don't think that these users need a more user friendly way to deal with exclusions because if they can express their needs in a form like "filter out all docx-files" they should be able to put it like *.docx. So instead of switching between simple mode and editable compiled patterns (you called it "advanced mode"), start with the simple folder tree list that configures at the node level and add a button called something like global ignore patterns or advanced mode where power users can input ignore patterns (like we have now) for the repository level.
This way, power users have fine grained control about what sort of data a syncthing repository contains and both most users and power users have a simple method to control what portion of that syncthing repository a certain node downloads/uploads to other nodes.

That is why I like @AudriusButkevicius suggestion:

I think we can keep what we have today and keep it the same way it is today, just add a thing before ignores which is the dir selector.

This is good for all users. Plus it doesn't involve rewriting the entire system, just adding the "simple node-level" sync system and it's GUI. (Not that the system would be simple to implement, but simple to use).

Also, repository-level ignore patterns don't need to reside in a .stignore file, they could also be saved within the syncthing configuration folder and synced between nodes as metadata.

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Nov 20, 2015

Also, another thing people asked for is synced ignores and global ignores (applied to all folders)

@sneak

This comment has been minimized.

Copy link

commented Nov 28, 2015

If set to a non-random sync priority (e.g. newestFirst) , the puller getting hung up on a directory delete with ignored files inside it (e.g. .DS_Store) means the folder stops syncing until the logs are checked for why (and the directory manually deleted). The default of 'random' seems to mitigate the impact of this, if I'm understanding it correctly.

@katrinleinweber

This comment has been minimized.

Copy link

commented Nov 30, 2015

I'd like to chip in a pretty good example of visualising exclusions from a folder tree: Crashplan. The greyed-out & red-marked files are excluded via filters:

crashplan file list with exclusion rule results

What Syncthing could improve over this design is the ability to right-click a file or folder & quickly access the menu to change the rules. E.g. the context menu could have the option in-/exclude files/folders like this one, depending on whether an already excluded or included file is right-clicked. This would open an edit filters panel on the side, in which changes trigger a preview in the folder tree. The preview could highlight & explain what would happen to the file/folder set (in colorblind-safe manner).

Including a specific file or sub-folder that is excluded by a rule should be as simple as clicking the greyed-out check-box.

@capi

This comment has been minimized.

Copy link
Member

commented Nov 30, 2015

What would also be good would be a policy per directory (as advanced setting), if new sub-directories are to be added automatically or not. For some directories it's desirable to automatically add new children, for others, it's not.

@ibizaman

This comment has been minimized.

Copy link

commented Jan 23, 2018

@madumlao I agree completely with your whole comment.

About parsing the file, I would go one step further. When you go in the ignore patterns menu, you would see two tabs. One Basic and one Advanced. The Advanced would be the one we have now: you edit the .stignore file directly. The Basic one would be the tree view, and you could only switch to the basic view if the .stignore has rules parseable by the tree view.

The idea is to copy the behavior of some tools I use regularly, like Jira for example. It has a JQL language allowing you to query and filter the tickets in complex ways. You can either write the JQL directly or use the dropdown buttons to generate a simple JQL behind the scenes. And you can only use the dropdown buttons if the JQL is simple enough, otherwise the basic mode is unaccessible.

I think it's a good idea to proceed like this because that way we can start with a simple set of rules for the basic view (the tree view), while not restraining advanced users and while allowing us to add features later on, when users had a try.

Anyway, there's no reason we should implement everything at once. What about a small but functional proof of concept? I would be happy to give it a go. What do you think of this first requirement list for the PoC:

  • Basic / Advanced tab
  • Advanced tab is what we have now
  • Basic tab is the tree view
  • Tree view lists the whole global folder and file tree, it won't show the .stignore file as it's never synced anyway
    • next to each file/folder, there's a checkbox
    • if the checkbox is checked, the file/folder is synced
    • if checkbox is unchecked, the file/folder is ignored
    • by default all checkbox are checked, generated .stignore is empty in this case (make it compatible with current behavior)
    • if a checkbox get unchecked, all checkboxes inside that folder get unchecked too (1)
    • when an unchecked checkbox get checked:
      • if its parent folder was unchecked, check it (do that recursively)
      • check all the children folders
  • When clicking on save in the Basic tab, a new .stignore is generated and replaced
  • Basic tab can't be opened (greyed out) if .stignore has too complicated rules. If it has, the user is an advanced user anyway.
  • The generation/parsing of the .stignore file would simply work like this: for all unchecked file/folder, the .stignore would have a rule with the absolute path of the file/folder. With the special case that it won't generate any rule to exclude children of an excluded folder, because it's not needed thanks to (1).
    That's all the parser would support, if there's another rule not matching the absolute path of a file or folder in the global folder and file tree, the Basic tab is greyed out.

This PoC is not that small, but it's the smallest set of requirements I could come up with which are consistent and would IMO make for a good user experience.

(1) This is because of the following rule, quoting the doc (empahsis mine):

A pattern beginning with a ! prefix negates the pattern: matching files are included (that is, not ignored). This can be used to override more general patterns that follow. Note that files in ignored directories can not be re-included this way. This is due to the fact that Syncthing stops scanning when it reaches an ignored folder, so doesn’t know what files it might contain.

What do you think?

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Jan 23, 2018

This needs more than that, as this assumes a default behaviour of excluding. It essence, any new directory added would automatically be included, which might not be what users want, so we need a way to invert the whole selection process.

I think this needs to be split into two separate matching engines, one for tree view, second one for patterns.

@madumlao

This comment has been minimized.

Copy link

commented Jan 24, 2018

This needs more than that, as this assumes a default behaviour of excluding. It essence, any new directory added would automatically be included, which might not be what users want, so we need a way to invert the whole selection process.

Isn't this already accomplished by ** followed by more specific !foo !bar rules?

@ibizaman

This comment has been minimized.

Copy link

commented Jan 24, 2018

Taking into account it would be a PoC for a tree view version of a simple .stignore file, I don't think we should tackle the inverse parser here. Not saying it wouldn't be a good feature to have, but I don't see any reason why the PoC requirements should include that.

Also what @madumlao said. Which wouldn't work in the Basic tab of the PoC as it would be deemed too complicated for the parser.

That being said, I don't understand what you mean by "this" assumes a default behavior of excluding. Is "this" the PoC requirements? The PoC would only be a parser of .stignore. When a new folder or file is created, the current behavior will kick in: because it's not added in the .stignore file, it will show up checked in the tree view. So I would've said the opposite: the PoC assumes the default behavior of including.

Edit: what's more, we could easily do both. The Advanced tab would just stay as-is while the Basic tab would have an additional checkbox at the bottom "include new directories by default". Depending if it's checked, we would generate a .stignore like I proposed or like @madumlao proposed. Again I wouldn't do this in the PoC, but the cool thing with this IMO is that we let the user choose what he wants, while not changing other parts of syncthing since both behaviors are already possible right now.

@tengwar

This comment has been minimized.

Copy link

commented Jan 24, 2018

@calmh calmh removed this from the v1.0 milestone Jan 31, 2018

@Hoeze

This comment has been minimized.

Copy link

commented May 19, 2018

What is the current state of this?

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented May 19, 2018

None, nobody is working on this.

@erdnaxeli

This comment has been minimized.

Copy link

commented Jun 29, 2018

Isn't this already accomplished by ** followed by more specific !foo !bar rules?

Actually it's the reverse, the first rule that matches is used. To only sync folders foo and bar:

!/foor
!/bar
**

This is the way of selecting files to sync as of today.

@ibizaman

This comment has been minimized.

Copy link

commented Jun 30, 2018

Actually, I am, very very slowly though. I have the UI working with the tree view coming from syncthing's global state. What's left on my side is the correct parsing/formatting of the rules.

I'm actually so slow on this that I have some merge conflicts. No real blockers, just need to find a big enough chunk of time to power through this.

@paulwalko

This comment has been minimized.

Copy link

commented Jul 26, 2018

I was looking around at things a found the 'selective' branch and although it's over 3 years old I figured it might be somewhat useful since nobody's mentioned it yet.

@grahamPegNetwork

This comment has been minimized.

Copy link

commented Jul 27, 2018

@ibizaman
Awesome to hear, this is a high value feature for me. I tend to run file sync software on desktop, laptop, server and mobile. One highly wanted feature is to have repositories for music, audio books, ebooks etc, and be able to easily "check out" a folder from the directory tree (a particular album/book) on mobile. Currently I'm stuck with streaming from plex which is less than ideal. I previously used Seafile for this and believe Owncloud/Nextcloud had similar functionality.

I'd be glad to offer a modest tip via ETH/PayPal if this feature makes it to my phone. Ping me if this gets added. :-)

@ibizaman

This comment has been minimized.

Copy link

commented Aug 12, 2018

@smartlaw-jenkins that's very appreciated. Actually my incentive is to make my wife use syncthing. 😅 For her to use something it must be pretty - I can't blame her - and this a missing feature that is needed for the adoption of syncthing.

@ibizaman

This comment has been minimized.

Copy link

commented Aug 16, 2018

I fixed the merge conflicts, code is nearly good for an alpha PR.

There's one thing I'm struggling with which is testing the parser that would read the rules in the textarea and tick the boxes in the tree.

By struggling for testing I mean manual tests are super slow I don't know how to add automated tests on the js code.

So, how can I test the js code? I don't need any browser or user interaction, just some input/output testing on several js functions.

If easier I can post the PR as it is now.

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Aug 16, 2018

Sadly we have literally nothing for JS testing.
An early PR can point out flaws before you sink more time into a potentially suboptimal implementation.

ibizaman added a commit to ibizaman/syncthing that referenced this issue Aug 17, 2018

gui/ignore: add skeleton tree view to pick files to sync (fixes synct…
…hing#2491)

This commit changes the editFolderModalView/folder-ignores tab by
adding two nested tabs under it. The Advanced tab holds the currently
existing text rules. The new Basic tab holds a tree view.

The tree view shows the existing files and folders that exist under
the given root folder configured in syncthing. This tree view is
populated from the root folder's global state found at the /db/browse
endpoint.

ibizaman added a commit to ibizaman/syncthing that referenced this issue Aug 17, 2018

gui/ignore: sync folder ignores tree view with text rules (fixes sync…
…thing#2491)

This commit syncs the treeview and text rules used to ignore
files.

The logic here is to share a common object call
$scope.syncedFiles. The tree view and text rules get synced when we
switch from tab to tab.

When we go from the Advanced (text rules) tab to the Basic (tree view)
tab, we parse the text rules, produce the syncedFiles object and
populate the checkboxes in the tree as needed. A special case is when
the text rules are not parseable (because too complicated), then we
deactivate the tree view. This step is complicated.

When we go from the Basic to the Advanced tab, we produce the shared
syncedFiles object and use that to rewrite the text rules. This step
is simple.
@Avamander

This comment has been minimized.

Copy link

commented Sep 15, 2018

I followed the chain of issues to here and it seems that one of the original ideas has gotten lost. Namely, on-demand file download. This would mean everything is in a blacklist until the user selects the file to be downloaded and displayed. The use-case of this would be basically serving files to clients, be it music, pictures, movies or something else, clients that lack the space and bandwidth to sync entire media libraries to.

A few more specific examples:

  • I would love it if I could instruct the Android app to download certain song from my music library with just one tap, I don't want to manually pick songs I want on my device especially if symlinks aren't supported, the solution at the moment is a waste of disk space, and my time and effort
  • My laptop to sync one folder of the project I'm currently working on instead my entire collection of projects that takes hundreds of gigabytes
@ibizaman

This comment has been minimized.

Copy link

commented Sep 16, 2018

@Avamander With the latest idea, you would need to:

  1. add a rule with * to ignore all
  2. tick the checkboxes in the tree for the songs you want to download

Could you explain the following more? I would say these contradict themselves.

instruct the Android app to download certain song from my music library with just one tap, I don't want to manually pick songs I want on my device

Also, from what I understood, you can already edit the .stignore file to do what you want, although it's something manual and very tedious at the moment.

EDIT: by new ideas I'm talking about #5132

@Avamander

This comment has been minimized.

Copy link

commented Sep 16, 2018

@ibizaman

instruct the Android app to download certain song from my music library with just one tap, I don't want to manually pick songs I want on my device

The most important part here is the with just one tap, the 1. add a rule with * to ignore all makes syncthing ignore everything downloaded (right?) and that would mean unchecking a box only has the effect of not syncing changes of the file - it doesn't delete it from the device. The closest example of the functionality I have in mind is GDrive's "Available offline" button and if I have understood everything correctly the latest idea doesn't behave exactly like that.

@kolcon

This comment has been minimized.

Copy link

commented Sep 16, 2018

MS's OneDrive does the same thing with W10... "Save space and download things as needed", or some similar name.

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Sep 16, 2018

@Avamander there is syncthing-lite for android that probably suits better what you want, I don't think we'll have that part of main syncthing. Also, W10 one-drive placeholder functionality is out of the question as it depends on some undocumented win32 apis.

@Hoeze

This comment has been minimized.

Copy link

commented Sep 16, 2018

I'd love a solution which shows all the files but only keeps a portion of them locally.

What about some FUSE solution?
I thought FUSE has already been ported to Windows.
Android should not be that of a problem, I'm pretty sure it already has appropriate API's.

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

commented Sep 16, 2018

There is syncthing-fuse already a thing, yet I am not sure how much it works, and what the performance is like, anyways, both of these are definately not canidates to become part of syncthing.

@ibizaman

This comment has been minimized.

Copy link

commented Sep 17, 2018

One could always use https://rclone.org/cache/ in the meantime. I didn't try it myself though.

@rptb1

This comment has been minimized.

Copy link

commented Sep 30, 2018

I reached here from #193, which I find by searching for "selective sync". I'll just state what I personally want: Resilio style selective sync as on Android. I don't want to have to edit text files with patterns, or even have such files. I want to (for example) point Syncthing at my entire music collection on my NAS, then be able to pull down tracks (files) or albums (folders) by picking them from a searchable tree. Once synced, they stay in sync if changed. If I delete them on the Android end, they stay deleted until I ask for them again. In this way I can use any number of source machines as libraries of things I might need quickly, without fuss, without having to remember where they are, on any number of other devices. Thanks!

@grahamPegNetwork

This comment has been minimized.

Copy link

commented Jan 3, 2019

Further discussion regarding implementation has moved to #5132

@nekr0z

This comment has been minimized.

Copy link
Contributor

commented Jan 31, 2019

Not that my opinion should really matter, but I'm all for "ability to ignore a directory by the presence of a special file in it", provided that it can be user-tuned, too. What I mean is not only cachedir.tag triggering this behaviour, but rather anything explicitly defined by user in .stignore (or whatever new model for expressing user's wishes on this subject).

Let's say a user can put some condition in .stignore that would tell Syncthing to ignore any directory containing some file named, say: ".stdearsyncthingdontevertouchanythinginthisdirectoryprettyplease" - that would be an ideal solution for the issue of ignoring git-enabled projects that seems to pop up in forums now and again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.