Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move convenient path selections in push/pull #6

Closed
2 of 3 tasks
MischaPanch opened this issue Jan 7, 2022 · 2 comments
Closed
2 of 3 tasks

Move convenient path selections in push/pull #6

MischaPanch opened this issue Jan 7, 2022 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@MischaPanch
Copy link
Collaborator

MischaPanch commented Jan 7, 2022

We should make it easier to push/pull a bunch of paths based on patterns. For that we should add

  • Permit passing glob-patterns to push
  • Permit passing glob-patterns to pull
  • Add the possibility to pass a regex as except_matches kwarg to permit simple exclusion of files. The current regex kwarg should be renamed to if_matches.

This would permit things like

storage.push("data/**/*.jpg", except_matches=r".*test.*") 

We could additionally allow passing a except_condition: Callable[[str]], bool] = None (or do you think if_condition is more natural?), in which case the above can be rewritten

storage.push("data/**/*.jpg", except_condition=lambda n: "test" in n) 

The condition could be even made more general, mapping the metadata-object to a bool (thereby e.g. allowing filtering by size), at the cost of a more complicated interface for callables. @fariedabuzaid @AnesBenmerzoug what do you think?

@MischaPanch
Copy link
Collaborator Author

MischaPanch commented Feb 28, 2023

Condition is not really needed for now, can be added later. Since the transaction is based on summaries, the condition could be more involved (e.g. including file size and other things). Passing globs to push was implemented, as well as exclude_pattern. Passing glob to pull was not implemented and might not be needed, it is a bit more complicated to do

@MischaPanch
Copy link
Collaborator Author

Closing this, libcloud does not support glob-patterns for listing and implementing this by ourselves would be messy

@MischaPanch MischaPanch self-assigned this Feb 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant