consider introducing dvc hooks

[efiop]

These would work similarly to git hooks. For example, one could configure post-repro hook that would run dvc push. Nice thing about these is that they would be tracked by git, so you won't have to re-install your hooks after each clone as you have to do with git hooks. The scenarios for which this feature could be useful are not yet that clear to me, so I would love to hear if any users have any scenarios in mind in which they would find such functionality handy.

Also, as @shcheklein noted #2359 (comment) if we actually decide to implement this, we'll need to consider using something better than single executables for hooks (e.g. yaml config like pre-commit does).

#2359 (comment)

[dashohoxha]

The scenarios for which this feature could be useful are not yet that clear to me, so I would love to hear if any users have any scenarios in mind in which they would find such functionality handy.

This is a scenario where DVC hooks may be useful: #2330
Let me explain.

DVC hooks can be placed on .dvc/hooks/ and may be named like pre-run, post-run, pre-fetch, post-fetch, pre-push, post-push etc. In general, pre-<command> and post-<command> (maybe not for all the commands).

Before running the command fetch for example, DVC will check for the existence of the hooks. Then it will call pre-fetch passing it all the arguments of the command fetch. From the hook it will get DONE or a possibly modified list of arguments. If the return status is DONE, then DVC will stop, because this indicates that the hook has already done the job of fetching the data and there is nothing else left to be done. If the pre-hook returns a list of arguments, then the fetch command itself will be called with this (possibly-modified) list of arguments. Then the post-fetch will be called (if available), passing it the return status of the fetch command.

How is this related to the issue #2330? We can imagine a couple of hooks pre-fetch and pre-push which actually do the backup/restore of the data themselves, bypassing/replacing the commands fetch and pull. These hooks may use rclone for example, to do the job. While it is true that rclone may be suboptimal compared to the DVC commands push and fetch, it supports more storage providers than DVC does. So, this may provide a not-so-good solution in a case where DVC provides no solution.

Another example of using hooks may be with the command dvc run, to avoid duplication of the deps/outs. If the command is: dvc run -d <inputfile> -o <outputfile> <command>, then the hook pre-run will be called, which will get the arguments -d <inputfile> -o <outputfile> <command>, and will return the arguments -d <inputfile> -o <outputfile> <command> <inputfile> <outputfile>. Then the command run itself will be called with these (augmented) arguments.

Another scenario is the one described here: #2359 (comment)
When you call the command dvc repro it will also find and call the hook post-repro, which will automatically call dvc commit and dvc push.

[efiop]

Closing for now, as there doesn't seem to be enough interest. The feature still seems useful, so maybe we'll reopen in the future, once there are more use cases.

[evstratbg]

+1 for this feature
or, any other possibilities to get notification of new files/version of files available at dvc repo?

[efiop]

@evstratbg Could you share more details about your scenario, please? 🙂

[evstratbg]

@efiop sure
We currently store very large ML models in dvc. We are starting to switch to k8s and plan to build the models in a separate docker image, because the code changes more often than the models. We discussed several options for building models from dvc and it would be very convenient to build them using curl query from dvc hook. It may not be the most popular way, but we are still in the process :) If you suddenly have recommendations or experience "how to deploy an application with very large ML models to k8s", I will be happy to absorb the experience

[efiop]

@evstratbg Hm, might be missing something, but have you considered regular git hooks? E.g. when a .dvc(or dvc.yaml/lock) file is being committed to git - your git pre-commit hook will send a curl query.