Make `dvc run` handle files with same name but different path

[sotte]

Let's say I have two scripts: code/gen_features_v1.py and code/gen_features_v2.py. They generate feature files data/v1/features.pkl and data/v2/features.pkl respectively.

I run the scripts like so:

dvc run -o data/v1/features.pkl python code/gen_features_v1.py
dvc run -o data/v2/features.pkl python code/gen_features_v2.py

But now the features.pkl.dvc only tracks the features.pkl from the second run even though I might need both.

cmd: python code/gen_features_v2.py
md5: some_md5
outs:
- cache: true
  md5: some_md5
  path: data/v2/features.pkl

Quite often I have have files with the same name. So how do I handle this? Or should dvc run be extended?

[efiop]

Hi @sotte !

We should definitely add some confirmation prompt for such cases(added #974 to track that), but in the mean time you could use -f option for dvc run to specify the name of the produced dvc file. Another way to handle this, would be for you to use git branches instead of creating v1,v2 files. The advantage of using git branches is that you could also utilize dvc metrics functionality to compare your results. You can find more info on it at https://dvc.org/doc/get-started/compare-experiments and https://dvc.org/doc/commands-reference/metrics .

Thanks,
Ruslan

[sotte]

Hey @efiop

thanks for the feedback!

Regarding the explicit -f: I see how that would solve the problem. But I know that I'll forget to do that, especially if there is no warning. So #974 is crucial.

Regarding git branches: yes, that works in many cases. If there are are two datasets that have to be explored (same name but different path) two branches don't really make sense.

Regarding the metrics: I already have some machinery in place to analyze, filter, sort, whatnot metrics. DVC does not offer anything here for my case. However, I really like the feature! :)

I really think the .dvc file should be stored next to the actual data then this problem would be nil. dvc add already stores the .dvc file next to the data. Why does dvc run have to be different?
I admit that this might lead to some backward compatibility problems. Maybe another optional argument for dvc run makes sense here, similar to -f but it automatically sets the path of the .dvc file: dvc run -F -o data/v1/features.pkl python gen.py --> data/v1/faetures.pkl.dvc.

Cheers,
Stefan

[efiop]

Regarding the explicit -f: I see how that would solve the problem. But I know that I'll forget to do that, especially if there is no warning. So #974 is crucial.

Sure, I am about to merge it into master. I'll release new version later today.

Regarding git branches: yes, that works in many cases. If there are are two datasets that have to be explored (same name but different path) two branches don't really make sense.

Maybe I'm missing something in your scenario, but you can have different data sets added in different branches. Just create git branch for your new data set, put it in place of the old one and call dvc repro, it will handle everything for you.

I really think the .dvc file should be stored next to the actual data then this problem would be nil. dvc add already stores the .dvc file next to the data. Why does dvc run have to be different? I admit that this might lead to some backward compatibility problems.

In case of the dvc add you simply add your data to your project, not running any scripts/code to process anything, so dvc places stage file right next to it, so even when there is no cache available you can still see where your data is located(i.e. when browsing a repository on github). On the other hand with dvc run you actually process your data, creating a meaningful stage in your pipeline and here dvc file acts like something similar to a Makefile, being an orchestration code for your main code. The reason dvc places stage file for dvc run in the directory that your script is being executed from is because when reproducing your stage, dvc uses the directory that the stage is located in as a working directory from which your script is going to be executed.

Maybe another optional argument for dvc run makes sense here, similar to -f but it automatically sets the path of the .dvc file: dvc run -F -o data/v1/features.pkl python gen.py --> data/v1/faetures.pkl.dvc.

Unfortunately I don't think it is a good idea, because you would still have to specify -o and paths in your command relative to the data/v1, which with -F option only creates more confusion. Currently you could use -c option for dvc run to specify the directory that dvc will chdir to before executing your command. For example:

dvc run -c data/v1 -o features.pkl python ../../code/v1/gen.py  --> data/v1/features.pkl.dvc

Notice that -o featuers.pkl as well as ../../code/v1/gen.py are specified relative to the data/v1, because it is the directory that this command is going to be executed in.

Thanks,
Ruslan

[Hong-Xiang]

I'm searching for solution for dealing with grid search with dvc, and found this issue maybe related to that.

The problem I'm facing is grid search for hyper parameters for model. In most case I'd like to run multiple program/pipe line in parallel to fully utilize computing power and reduce computation time.
However this is difficult when using branches to change hyper parameters. IMO allowing file with same name in different folder maybe a good solution for this.

[sotte]

@Hong-Xiang that's a good example, thanks!

[efiop]

Hi @Hong-Xiang !

Could you please shed more light on the issue you are experiencing? We do not prohibit same filenames in different folders, and our latest version asks for confirmation before overwriting it if that does happen. Maybe I misunderstood your scenario, but in your case you would need parallel dvc run, which is currently forbidden, because of the nature of how we deal with our global lock. However, we are indeed working on allowing running multiple dvc run instances simultaneously right now, please see #755 to track the progress.

Thanks,
Ruslan

[Hong-Xiang]

@efiop , thanks for your explanation. Now I understand that currently multiple dvc run is not ready yet. However in my project, parallel training is necessary and I think it could "partially" done with current version of dvc. Detailed explanation is shown below.

The issue I'm facing should be separated into two

1. prepare multiple work directory and some processed data for grid search (in this step, there are multiple steps of pre-processing, not time consuming comparing to training)
2. train them (in this step, parallel is desired, time consuming)

Usually I have multiple CLI tools to perform data processing, exactly what dvc pipe line is really helpful. But when I have multiple work directory, it become hard to use dvc files. For example:

File structure of my project is something like:

.
├── code
│   ├── src
│   │   ├── myproj
│   │   │   ├── cli
│   │   │   │   ├── prog1.py
│   │   │   │   ├── prog2.py
│   │   │   │   ├── __init__.py
│   │   │   │   ├── train.py
│   │   │   │   ├── eval.py
│   │   │   ├── train
│   │   │   │   ├── some files ...
├── data
│   ├── data_raw.npy
├── runs
│   ├── gs1
│   │   ├── config.json
│   │   ├── train.json
│   ├── gs2
│   │   ├── config.json
│   │   ├── train.json
│   ├── gs3
│   │   ├── config.json
│   │   ├── train.json
├── debug
│   ├── proc1.npy.dvc
│   ├── proc2.npy.dvc
│   ├── train.npy.dvc
│   ├── test.npy.dvc
│   ├── config.json
│   ├── train.json
└── setup.py

In debug directory, multiple dvc run were performed to construct the pipe line like

dvc run -d ./data/data_raw.npy -d config.json -o proc1.npy myproj proc1 -c config.json ./data/data_raw.npy proc1.npy
dvc run -d proc1.npy -d config.json -o train.npy myproj proc2 -c config.json proc1.npy train.npy
dvc run -d proc1.npy -d config.json -o test.npy myproj proc2 -c config.json proc1.npy test.npy
dvc run -d train.json -d train.npy -d test.npy -o model myproj train -c train.json train.npy test.npy

dvc is an awesome tool when I modified config.json and train.json, I can finish all the work by simply using dvc repro.

But after developing the pipe line in debug directory, I'd like to run it on runs/gs1, runs/gs2 and runs/gs3. Each of them contains different train.json and config.json. Then I have the problem, how can I use pipe line I defined in debug directory in runs/gs* directories? It's easy to modify the file structure to make sure they got same raw file by relative path, but how can I "run" those .dvc files located debug file in runs/gs* files? Should I copy them to all those files?

If I understand the doc of dvc correctly, solution for this seems to be using git branches and work exactly in debug directory for different hyper parameters (*.json files), and dvc provides some good tools like metrics for analysis results. But I just can't train multiple models simultaneously in this approach.

If we can't run dvc run in parallel, I still want to use dvc repro sequentially in runs/gs* files for pre-processing steps, and then just run the train step in parallel without using dvc system, something like:

cd runs/gs1
dvc repro train.npy.dvc
dvc repro test.npy.dvc
myproj train -c train.json train.npy test.npy

Is it possible to achieve this in current version of dvc？Or is there other approaches I missed?

[efiop]

Hi @Hong-Xiang !

Thank you for sharing! Very interesting scenario indeed. It is now clear that git branch is not the one and only proper way to perform experiments, especially in the grid search scenario. I think we should consider introducing something similar to "execution matrix" to allow defining the pipeline once and then making it easy to run on a range of data sets/configurations in parallel. Additional great thing about such potential approach is that you will still be able to use dvc metrics to view the results for each runs/gsN and then choose the best performing one. I've created #1018 to track the progress on that. In the meantime, I think you could workaround it by reformatting your project a bit like so:

├── runs
│   ├── gs1
│   │   ├── config.json
│   │   ├── train.json
│   ├── gs2
│   │   ├── config.json
│   │   ├── train.json
│   ├── gs3
│   │   ├── config.json
│   │   ├── train.json
│   ├── debug
│   │   ├── proc1.npy.dvc
│   │   ├── proc2.npy.dvc
│   │   ├── train.npy.dvc
│   │   ├── test.npy.dvc
│   │   ├── config.json
│   │   ├── train.json

Notice how debug directory is now under runs/ and looks like a regular gsN. This allows for your debug pipeline to have more symmetry with other gsNs. Now you can work on your debug pipeline and when you are ready to run it on your gsN, you can simply copy your pipeline like so: cp debug/*.dvc gsN/. Now you will be able to run your gsN pipelines with dvc repro gsN/model.dvc. Also, to make your project more convenient, I would suggest to:

1. Add -f Dvcfile to your last stage dvc run -d train.json -d train.npy -d test.npy -o model myproj train -c train.json train.npy test.npy, so that you will be able to trigger reproduction with a more convenient dvc repro -c runs/gsN command;
2. In the runs/ directory run

$ dvc run -d gs1/model \
             -d gs2/model \
             ...
             -d gsN/model \
             -f Dvcfile

to neatly tie down all your gsNs to be able to simply use dvc repro(from the runs/ directory) or dvc repro -c runs(from the project's root) to automatically run all of your gsNs. Notice that since we still don't support parallel run execution, your gsNs will be run sequentially.

If we can't run dvc run in parallel, I still want to use dvc repro sequentially in runs/gs* files for pre-processing steps, and then just run the train step in parallel without using dvc system, something like:

cd runs/gs1
dvc repro train.npy.dvc
dvc repro test.npy.dvc
myproj train -c train.json train.npy test.npy

Is it possible to achieve this in current version of dvc？Or is there other approaches I missed?

Sure, to do that, you can just replace the last

dvc run -d train.json -d train.npy -d test.npy -o model myproj train -c train.json train.npy test.npy

with

dvc run -d train.json -d train.npy -d test.npy -f Dvcfile

to have a convenient way to get everything prepared in a single dvc repro -c runs/gsN/(or simply dvc repro if run from the runs/gsN directory) command.

Thanks,
Ruslan

[Hong-Xiang]

@efiop , thanks a lot for your advice, it helps a lot and significantly reduced time cost and complexity for grid search. "execution matrix" looks very cool and would make grid search even more easier, looking forward for it.

[efiop]

Hi @Hong-Xiang !

We would love to hear more about how you use dvc, which features you like/dislike and which features you would like to see implemented. Would you be interested in a quick video meeting with us(@dmpetrov , @shcheklein and I) over the skype/hangouts? How about 17th of August from 10:30AM to 11AM GMT+8? If you don't like video meetings, we would be happy to talk to you using any method of communication you like.

Thanks,
Ruslan

@efiop , could we close this issue?

[efiop]

@MrOutis Yep, let's close it in favor of #1018 and #1462 . Thanks for noticing! :)