run/repro: add option to not remove outputs before reproduction

[efiop]

https://discordapp.com/channels/485586884165107732/485596304961962003/540232443769258014
https://discordapp.com/channels/485586884165107732/485586884165107734/547430257821483008
https://discordapp.com/channels/485586884165107732/485596304961962003/557823591379369994

What about corrupting the cache, @efiop ? Is this for reflink only?

[efiop]

@MrOutis Great point! We should dvc unprotect the outputs when this option is specified, so that user has safe copies(or reflinks).

[AlJohri]

Adding my +1 here. This would be helpful for doing a warm start during parameter search. Some of our stages are very long running and starting from the previous results would help it complete much faster.

[guysmoilov]

@AlJohri WDYT of this (hacky) possible solution?
Add a stage before the training stage, that takes the latest cached checkpoint from the output of the training stage, moves or copies it to a different path, and then that resume checkpoint is a cached dependency of the training stage?

This should allow each git commit to describe an accurate snapshot of a training run - the checkpoint you start from, and the checkpoint that resulted.

[efiop]

Probably should look something like:

$ dvc run --outs-no-remove ckpt ...

which would add something like

remove: False

to the dvc file for ckpt output, so that dvc repro knows about it.