dvc: wildcard outputs?

[jorgeorpinel]

It's a well known limitation of DVC that 2 stages (or stage and .dvc file, etc.) can't have overlapping output paths (maybe also deps in some cases?) — this applies to directories, or course. For example

$ dvc add data/
$ dvc run -n clean -d data -o data python cleanup.py data
ERROR...

In the case above the dependency and the output are the same because maybe there's multiple raw data files in data/ and you don't want to use -d for each one — it may even be impossible if it's a variable number of raw data files coming from a previous, non-deterministic stage.
Similarly, the output may be hundreds of files (or a non-deterministic variable number) so you just want to indicate the whole directory.
For some external reason, maybe you need to avoid splitting the raw and clean data directories — we've had support cases like this e.g. this one.

Solution: Wildcards? E.g.

$ dvc add data/raw*
$ dvc run -n clean -d data/raw* -o data/**/clean* python cleanup.py data

[jorgeorpinel]

Cc @dmpetrov

[efiop]

$ dvc add data/raw*
$ dvc run -n clean -d data/raw* -o data/**/clean* python cleanup.py data

These commands won't get you what you want. First of all, they are evaluated by your shell and not by dvc. To make your shell pass them to dvc, you'll need to escape it somehow (e.g. wrap it in single-quotes), which makes it quite error-prone.

But this is discussed in #1462 , so we can probably close this as a duplicate.

[jorgeorpinel]

To make your shell pass them to dvc, you'll need to escape it somehow

Sure, it's just a quick example.

But this is discussed in #1462

I see it there indeed, thanks for the reference. It's a big discussion issue from December 2018 though... Probably the best way to make it actionable is to break it into smaller ones? Like this one!

[efiop]

@jorgeorpinel I wouldn't call this one simple. It is not only about supporting wildcards, but rather supporting some scenarios that might need wildcards(or there might be a better way to do it). So I would keep the discussion there instead of also having this one.

[jorgeorpinel]

Yeah I didn't say it's simple. And sure, up to you. I already commented over there too. Thanks