dvc pull with target ignoring specific targets inside

[HectorAnadon]

Is it possible to run dvc pull a folder, ignoring certain folders (.dvc files). If not, would it be possible to implement? Example of a use case below as it is a bit hard to explain.

Inside of folder A, I have folders 10, 11, 20, 21, 31, 41 and I want everything except 20 and 21. Would be something like this possible?
dvc pull -R A --ignore 2*

Thanks!

[karajan1001]

Dos dvc support File and directory patterns now?

[HectorAnadon]

It worked in Ubuntu for me for pulling and pushing

[efiop]

Pattern support here is provided by your shell expansion. You could exclude with it too https://stackoverflow.com/questions/216995/how-can-i-use-inverse-or-negative-wildcards-when-pattern-matching-in-a-unix-linu . Not sure it is worth supporting on dvc's end like that though 🤔

[HectorAnadon]

I would say this doesn't contemplate this use case where the files are more nested.
Folder A contains folder B and C. I want to pull all the files from C and from B I want to ignore a certain pattern. Of course I could run two pull commands but when there a lot of folders this becomes a bit annoying.

[efiop]

@HectorAnadon I'm pretty sure you could do that with shell expansion too. Sorry, but I don't think this is worth implementing on dvc side. --ignore would be constantly confused with shell expansion (e.g. if you forget single-quote it - it expands by your shell, so it will look like a bug in dvc). Plus, it is not clear where that matching should start or stop, which will again bring confusion as some people will think that it works inside of dvc-tracked dirs as well. This is way too much complication. Running two commands or using dvc pull -R target1 target2 ... is a much simpler solution. Closing for now.

[HectorAnadon]

For example, how would could you do dvc pull from a whole repository ignoring certain folders? Is there a way? I would be really interested on this.

[efiop]

@HectorAnadon I don't have a tested command to share, but from the link that I've shared above it will probably be something like:

$ shopt -s extglob
$ dvc pull -R !(Unwanted)

I see that you are indeed very interested in this and that it would be a major convenience feature for you. Reopening. Maybe you would be interested in implementing it? If not, I can't really guarantee that we will get to this any time soon, unless it is requested by a lot of users.

[efiop]

@HectorAnadon You might find dvc/dvcignore.py implementation helpful for this new feature, since it does pretty much that but you need to set ignored things in .dvcignore first.

[HectorAnadon]

I was unaware of .dvcignore. (still a lot of things to learn). I think that will do the trick and there is no need to extend pull functionality.
Related to this, I've found when doing dvc pull, sometimes the ignored files are deleted. Why is that? However, if it is a folder that has been modified, the folder is not deleted.
I can give an example:
It behaves differently if I ignore **/mesh/* the folder is not deleted. If I ignore **/mesh.dvc when doing dvc pull the folder gets deleted.

Could you explain why does that happen? Thank you very much