Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include/Exclude mechanism #535

Closed
memeplex opened this Issue Dec 28, 2015 · 24 comments

Comments

Projects
None yet
2 participants
@memeplex
Copy link

memeplex commented Dec 28, 2015

As go regexps don't support negative lookahead is difficult to exclude some set defined by a barely selective pattern (like ^\. in a home directory) and yet include some small subset of it (say some dotfiles I want to sync).

@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Dec 28, 2015

Hello @memeplex, thank you for reporting this and welcome to drive!

Would you mind giving an example for your use case because I don't yet understand the deficiency?

@memeplex

This comment has been minimized.

Copy link
Author

memeplex commented Dec 28, 2015

Sure. Say I want to sync the part of my home directory with contents generated by me. This will include some dotfiles and most directories and files that don't start with a dot. But there is a large number of dotfiles that I don't want to sync at all. So the rules will be something like:

-^\.*$
+^\.mydofile1$'
+^\.mydotfile2$'

I could do that with a negative lookahead as follows:

exclude=^\.(?!(mydotfile1|mydotfile2))$

But (?! is not supported by go.

@memeplex

This comment has been minimized.

Copy link
Author

memeplex commented Dec 28, 2015

See for example rclone, which implements rule based filtering: http://rclone.org/filtering/

@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Dec 28, 2015

Alright, thanks for the example, am gonna take a look at this and make a couple of comparisons then get back to you.

@memeplex

This comment has been minimized.

Copy link
Author

memeplex commented Dec 28, 2015

I thought drive push .mydotfile1 .mydotfile2 could be a workaround but it doesn't push anything. Am I doing something wrong?

@memeplex

This comment has been minimized.

Copy link
Author

memeplex commented Dec 28, 2015

It only seems to work when .mydofile1 and .mydotfile2 are already in the server, then changes are pushed. Is it expected to work like that?

@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Dec 28, 2015

So I've ran your tests to compare with other languages e.g Python and none has matched, maybe a revision of the regex might change things. Please see https://gist.github.com/odeke-em/c29ae8bc475fd0fb9eda.

In regards to:

I thought drive push .mydotfile1 .mydotfile2 could be a workaround but it doesn't push anything. Am I doing something wrong?

drive won't push anything that you've already synced and needs no changes unless you request an explicit push in this case with a --force flag. Please see https://github.com/odeke-em/drive#pushing. Otherwise if the content changes then sure it will push it up or if modTime changes occur.

Would you mind giving me your drive version, ie after running

$ go get github.com/odeke-em/drive/drive-gen && drive-gen
$ driver version

that should give you information kind of similar to this

$ go get github.com/odeke-em/drive/drive-gen && drive-gen && drive version
drive version: 0.3.4
Commit Hash: '6992e27390b35ec51db59ae1d03e0cb439bbb38b'
Go Version: devel +337c5541 Sun Dec 20 20:08:59 2015 -0700
OS: darwin/amd64
BuildTime: 2015-12-27 20:28:07.005553354 -0700 MST
@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Dec 28, 2015

Oops, I forgot to mention that hidden files by default are ignored, please use the --hidden flag during a pull or push.

@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Dec 28, 2015

I think I can also reproduce the .dot files not being pushed. I'll file a bug about .dot files not being pushed, probably a regression. Thanks for that catch!

@memeplex

This comment has been minimized.

Copy link
Author

memeplex commented Dec 28, 2015

Regarding the regexp: just remove the ending $.

Regarding the push thing: I'm not testing it with hidden files. What I'm doing is:

  1. Create a new file or directory xxx.
  2. Run drive push xxx.

Then nothing happens. Now if I push the parent directory of xxx, then xxx is added to the server. After that I:

  1. Modify the contents of xxx.
  2. Run drive push xxx.

Then xxx is updated in the server.

@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Dec 28, 2015

Roger that. I can confirm that I broke this functionality with PR #528 and here is a breakdown of what I screwed up:
Before a raw lookup was done for the first file, if none was returned, an appropriate error and nil file was passed back so a need to sync would be made.
Now a channel of files ie get all the matches is returned and if no match was made, that channel is closed and no files are registered as registered whether empty or not hence no syncing ever occurs since matches are within a loop ie

for rem := range remotesChan {
   ....
}

// Will never be run if the file didn't exist upstream.

Am patching this up and thank you very much for bringing this to my attention. Fix coming up soon.

@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Dec 28, 2015

@memeplex please get the latest from master and your work around should be working. The bug in your push/pull workaround that you found has been addressed by PR #537.

@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Dec 28, 2015

You can do so if you don't already know by

$ go get -u -v github.com/odeke-em/drive/drive-gen && drive-gen
@memeplex

This comment has been minimized.

Copy link
Author

memeplex commented Dec 28, 2015

Thank you very much. Despite include/exclude support would be nice lacking regexp lookahead, I think I can workaround that with some shell glob matching and your fix.

@memeplex

This comment has been minimized.

Copy link
Author

memeplex commented Dec 28, 2015

See for example extglob for bash:
https://www.gnu.org/software/bash/manual/html_node/Pattern-Matching.html#Pattern-Matching
It allows for powerful selective file expansion in a simple way. Coupled with the ability to push/pull only some paths, this renders the ignore file more or less obsolete.

@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Dec 28, 2015

Ah I see, thanks for the references. I'll keep digging and then I'll see if we can get this in, but I do agree the ability to match not all files in a match set is definitely useful.

If you don't mind and can give me feedback, I could hack up for you a solution but basically inside your .driveignore
for each clause that you'd like to exclude from ignoring, you'd have to use the #exclude: tag

^\. #this one gets ignored
#exclude:(mydotfile|only|others)

It is 10:07PM and except for going out to lunches and etc, I don't have to be busy till the 4th of January 2016 so I think I could rig up something for you in this time. Please let me know if you are interested then we can proceed.

@memeplex

This comment has been minimized.

Copy link
Author

memeplex commented Dec 28, 2015

Looks fine to me. I would avoid the word exclude here, I understand you're using it as a "double negation" in "exclude from ignore", but it usually means the same than just ignore. Also some escaping mechanism will be in order as the tag could be part of a valid regexp itself. A leading ! could be just fine, escaped with !.

@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Dec 28, 2015

Good point right there, a '!' actually is simpler and is related to the original deficiency ?!.
Aye aye, I'll ping you soon with a PR.

odeke-em added a commit that referenced this issue Dec 28, 2015

ignorer: added include + exclude handling
Fixes #535.

As of time:"1451307614.785091", Go apparently doesn't have a
"negative lookahead" mechanism ie `ignore all but`. This commit
adds an `exclude but for` clause that mitigates this shortcoming.
@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Dec 28, 2015

@memeplex I got sometime before bed and I've spun out a PR for you #541. Please help me test it out to see if it matches your use cases and it looks alright. Please don't hesitate to reach out in case of any questions or any ideas on this.
Thank you.

@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Dec 29, 2015

Hey there @memeplex, I just wanted to let you know that now you can do the inclusion/exclusion you had wanted.

# My .driveignore file
# Note that it can be put in ~/ or ever single directory for which you want one customized
# e.g mySchoolDrive/.driveignore, myWorkDrive/a/b/.driveignore myWorkDrive/a/c/.driveignore
^\.
!^\.dockercfg
!^\.bashrc|\.info

Please get the latest from master and thank you for requesting for this and for the discussion!

@odeke-em odeke-em added this to the v0.3.5 milestone Dec 29, 2015

@odeke-em odeke-em self-assigned this Dec 29, 2015

@memeplex

This comment has been minimized.

Copy link
Author

memeplex commented Jan 3, 2016

Thank you very much, you're the best.

@ghost

This comment has been minimized.

Copy link

ghost commented Mar 5, 2017

Is it possible in .driveignore do something similar to .gitignore:

//when you want to commit just the content of magic1 and magic2 
/*
!src/
src/*
!src/someBigFolder/
src/someBigFolder/*
!src/someBigFolder/magic1/
!src/someBigFolder/magic2/
@odeke-em

This comment has been minimized.

Copy link
Owner

odeke-em commented Mar 5, 2017

Hello there @antigone2, thanks for the question and welcome to drive.

I don't know, never tried it, please try and let us know if it works :)

@ghost

This comment has been minimized.

Copy link

ghost commented Mar 5, 2017

Well, @odeke-em the thing is I have failed to figure it out, that is why I am writing :)

The goal is this:

Situation:

  1. You have established gDrive folder
  2. You have created the a symlink ln -s /from/your/wokring/direcotory /to/gDrive/

Now you want to set up .driveignore to ignore all but some sub/sub folders in your working direcotory - typically you want to backup to drive just source codes and config files, no compiled results etc.

Only thing what was working for me was:

example situation

let's say you have in gDrive folder container in which you have test1, test 2, test3 folders which all contain some files, and you want only test1 and all its content to sync.

to ignore all what f.e. starts with /container/, but not what starst with !^/container/test1. That worked, but it is not a perfect solution and it is not working on next levels.

So my question really is if is it just something simple I am too dumb to figure out, or is it impossible to do by design. Thank you very much for your answer, btw nice project, up-to date ppa, thank you and @rakyll for doing it - exactly what you need on Ubuntu.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.