Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Short flag for persisted outputs #1884

Open
guysmoilov opened this issue Apr 14, 2019 · 3 comments
Open

Short flag for persisted outputs #1884

guysmoilov opened this issue Apr 14, 2019 · 3 comments

Comments

@guysmoilov
Copy link
Contributor

@guysmoilov guysmoilov commented Apr 14, 2019

From: #1214
Since this is will probably be an often-used feature, I believe it warrants its own single-letter command line flag. Maybe -p -P?

@efiop

This comment has been minimized.

Copy link
Member

@efiop efiop commented Apr 14, 2019

@guysmoilov We are still debating whether or not persist should be an allowed hack or not. Here is another question that we are thinking about right now while working on #1877 with @mroutis . Imagine you have persist outputs for some stage in your pipeline. Should dvc repro consider them as changed every time and re-run them or should it not? If we are talking about continuing from a checkpoint, it feels like persist should only be used until you are happy with the results, after which you should unset that and maybe even re-run your pipeline to ensure that it was indeed produced by your script, and so dvc repro should always consider such stages as changed and re-run them. On the other hand, if we are using such persistent output as a kind of a log file, then it shouldn't trigger dvc repro. What are your thoughts about that as a user?

@guysmoilov

This comment has been minimized.

Copy link
Contributor Author

@guysmoilov guysmoilov commented Apr 14, 2019

I think it definitely shouldn't be "always changed".
Changed implies that the dependencies are different, not that I want to resume from a checkpoint.
When I want to resume from a checkpoint, I just force the re-running of the stage.
tbh I also am not a huge fan of the "stage with command and no deps is always considered changed" approach, it's annoying when I want a stage to import raw data using wget from a site that has no etag. I think such a "callback stage" should be a user-controlled decision. But maybe I'm outdated here? I remember there were feature requests regarding callback stages.

@efiop

This comment has been minimized.

Copy link
Member

@efiop efiop commented Apr 14, 2019

@guysmoilov Great point! I agree with that.

We have an issue for callback stages already https://github.com/iterative/dvc/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+callback . We'll try to permanently fix it in 1.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.