Direct stage dependencies on other stages specifiable by the stages's name

[ChristophJud]

TL;DR

Proposing the -D <stage> option in the run command to specify dependencies on other stages.

One Possible Scenario

In order to reproduce results dependent on the whole software and OS environment, I'm executing my ML software in a Docker container. In this way, I can control not only the library versions of the used python packages but also the python version, the OS version, the CUDA/cuDNN versions etc. Thus, in my opinion, it makes sense to build the Docker container using dvc run for example:
dvc run -n build_docker -d docker/Dockerfile -d src/requirements.txt docker build -f docker/Dockerfile -t my_image . --network=host --no-cache=True

which depends on the Dockerfile and the requirements.txt file. The output of this command is the Docker image at some location which depends on the OS and the docker installation. Therefore, it is difficult to specify it as an output, and usually, one doesn't want to track the docker image but rather the Dockerfile.

The next stage could be e.g. a preprocessing step of the ML pipeline. It is executed within the previously built docker container and thus depends on the build_docker stage
dvc run -n preprocess -p config.yaml:data --external -o /tmp/preprocessed_data -d /data/ -d src/preprocess docker run -t --init --entrypoint= --runtime=nvidia --ipc=host --volume=/home/user/src:/app my_image:latest python3 -B src/main.py

There are following options to make the preprocessing stage dependent on the build_docker stage:

1. With the current dependency option, files and directories can be specified exclusively as dependencies. In this case, one could add the same dependencies to the preprocess stage which have been specified in the build_docker stage. However, this would be error-prone as things have to be specified multiple times and one could not ensure that the image has been actually build before running preprocess.
2. Because it is difficult and probably not useful to define the docker image as an output one can wrap the build_docker in a separate script which outputs a dummy file. This file can be defined as an output in build_docker and as a dependency for the preprocess stage. However, this is not that handy and somewhat confusing.

Proposed Solution

I'm proposing a third more elegant way to define this dependency namely by specifying a stage as a dependency using e.g. a -D option. Thus we could just add -D build_docker to the preprocess stage without specifying the dependency indirectly over a dummy file.

[shcheklein]

Discord context: https://discordapp.com/channels/485586884165107732/563406153334128681/760468302857895997

@ChristophJud I'm curious if this one:

the simplest solution that comes to my mind is for the build stage to output a dummy file "build.out" (e.g. with a timestamp of the latest build in it) and for the next stage depend on this file.

can be a workaround for this?

[ChristophJud]

https://discord.com/channels/485586884165107732/563406153334128681/760748507060895815

I think, that would be an option. However, as discussed above, dvc would not be able to figure out if the docker image has changed. in the docker scenario, there is docker image inspect which allows you to gather detailed information about your container. e.g. a timestamp. Maybe, in addition to the proposed option -D <stage> one could say, that one has to provide a hook which determines for dvc if something has changed. Such as: -D <stage> <hook-command>. Instead a file content, it would be a cmd-output which can be analyzed by dvc. In the docker example it could be docker image inspect my_image | grep LastTagTime. If this cmd-output changes, dvc knows, something has changed. This would add a lot of flexibility.

[ChristophJud]

I've just realized that actually, the stage would not matter anymore. It would just be: -D <command> where dvc gathers the stdout and compares it to the stdout of the last execution of command e.g. with the md5 hash of the output or so.

[shcheklein]

I think this is very closely related to the #4363 , #3439 (I recommend reading it from the bottom : ) )

[efiop]

Closing as stale.