Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dvc run: pass dependencies and outputs as params #995

Open
dmpetrov opened this Issue Aug 10, 2018 · 4 comments

Comments

Projects
None yet
2 participants
@dmpetrov
Copy link
Member

dmpetrov commented Aug 10, 2018

From xiang0x48: https://discuss.dvc.org/t/is-there-any-elegant-way-of-passing-argument-d-and-o-to-command-run-by-dvc-run/66

"When using dvc run, argument of dvc run and true command to be run are always similar, thus
dvc run -d source.npy -o target.npy python some_process.py source.npy target.npy . As there are repeated file names, it might be a “smell of code”."

Solution: introduce --pass-params to add all the dependencies and output params in the same order.

Example:
Commnd dvc run -d source.npy -d proc.py -o target.npy --pass-params python proc.py myparam
DVC adds all the params in addition to the original one (myparam): myparam source.npy proc.py target.npy

@efiop

This comment has been minimized.

Copy link
Member

efiop commented Aug 10, 2018

This is really hacky and not at all flexible. You only build your pipeline once, so it is not that inconvenient to introduce such a hack. Plus it defeats the purpose of our explicit dvc run syntax making your command really cryptic. I would rather not add this.

@dmpetrov

This comment has been minimized.

Copy link
Member Author

dmpetrov commented Aug 10, 2018

Something is needed to avoid the duplication which is a good source of mistakes.

This is the way to avoid the duplication with a minimum overhead - a single option instead of "flexible" set of options. Another alternative is to do this by param: @1 @3 or @dvcparam1 @dvcparam3. The first approach just the simplest way of doing this.

Any of these can work.

Yes, you build a pipeline once. But it still requires a way to avoid mistakes.
I agree that this is a quite hacky option and we should not use it in tutorials.

@efiop

This comment has been minimized.

Copy link
Member

efiop commented Feb 25, 2019

Related #1462

@efiop

This comment has been minimized.

Copy link
Member

efiop commented Apr 16, 2019

Related discussion on discord https://discordapp.com/channels/485586884165107732/563406153334128681/567801713948360725 proposing --parse option that would parse command by itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.