Support running hook (scripts) during different parts of the deployment process

[andir]

It is probably a good place to have a discussion about this kind of feature here.

I'd like to be able to hook into the deployment of my nodes with some custom scripts. Depending on the server I'd like to unlock the rootfs via SSH when I reboot it. e.g. morph deploy ... reboot should execute some custom script that is executed as soon as the machine is reachable again.

Ideas for hooks:

• before-build: before the actual machines build is executed this command shall be run. Potentially with some meaningful arguments (path to network, target name, flags?).

• after-build: once the build of a machine has finished this would be executed. Useful if you want to upload all your build outputs to some (remote) binary cache before actually deploying something.

• before-deploy: Before running a deployment on a single machine (fires once for each?) this script would be executed.

• after-deploy: Pretty much at the time where you would push secrets I guess.

• before-reboot: Before the reboot is triggered. Could help users to remove a server from a LB or to gracefully kill some process.

• after-reboot: After the reboot when the servers become available again.

I would expect the configuration to look like this:

{
network.hooks = {
   pre-build = pkgs.writeScript "pre-build" "exit 0";
};

some-host.deployment = {
   hooks = {
    pre-build = pkgs.writeScript "yet-another-script-pre-build" "exit 1";
    after-build = pkgs.writeScript "push-out-path" "nix copy $1 --to file:///some/local/binary/cache";
  };
};
}

By no means I am set on any of the interfaces or specific hooks outlines above. I just want to throw my (brief) idea in here and start a discussion about this.

[johanot]

see also #84

[andir]

see also #84

Oh! I must have missed that when looking for "hooks" in the issue tracker :)

[adamtulinius]

Hi there,

Thanks for the suggestion. The request is not entirely new to us, but we've talked about experimenting with HTTP based hooks first, since the security implications are easier to reason about from the morph side of things.

That aside, could you expand a bit on your suggestions? For instance..

• Do you imagine these hooks being run on the local host or the remote host? Or should that be configurable?
• What should happen when a hook fails for some reason?
• Should morph support multiple scripts per hook, and if so, what about ordering of them?

Hooks would be awesome to implement, but it can also get pretty hairy if we try to do everything at once. I think it might be worth just throwing something together as a start (maybe even only or two hook points), and add it to master behind some --experimental flags, so that we can iterate on it without making people think that it's a done feature.

[adamtulinius]

Oh! I must have missed that when looking for "hooks" in the issue tracker :)

Don't worry, the draft is really rough, and it has pretty much gathered dust since nixcon. :-(

[johanot]

Oh! I must have missed that when looking for "hooks" in the issue tracker :)

@andir probably because it's a PR, not an issue :)

[andir]

Thanks for the suggestion. The request is not entirely new to us, but we've talked about experimenting with HTTP based hooks first, since the security implications are easier to reason about from the morph side of things.

That is probably a valid concern but I fear it will end up becoming too big of a monster for most/many users. I do not know what the distribution of morph users currently looks like (how many machines, private, single/multi-admin, …). In my personal circle of friends we are mostly a few people that run hobbyist projects using morph. Having to run a webserver locally just to get some hooks done might not be feasible.

Edit: Remember the thing that many people probably like about morph (besides not being legacy python code…) is that it is stateless and lightweight. That would no longer be the case if you have to run a webserver to get some hooks going.

If you are talking about web hook that also means that we need the full range of:

• specifying the HTTP verb
• specifying arbitrary headers (potentially depending on some secret, local session file, …)
• specify some way to describe the payload. You probably do not want to send the same content over and over again. So this will probably get us into templating hell.

All of that could probably avoided when just inventing our own protocol to send events to some foreign system.

That aside, could you expand a bit on your suggestions? For instance..
* Do you imagine these hooks being run on the local host or the remote host? Or should that be configurable?

I would currently only support local commands. Remote commands are more properly encapsulated in systemd (one-shot) services on the remote target. We shouldn't try to reinvent the activation scripts.

* What should happen when a hook fails for some reason?

I would fail (or even better rollback) the deployment. If a build fails, no harm is done. Execute a build failed hook (or similar), if set.

* Should morph support multiple scripts per hook, and if so, what about ordering of them?

My initial idea would be to have the same mechnasim as in nixpkgs and allow people to mkForce an option to remove prior values if they intend to set a new set of complete hooks (of a specific kind).

e.g. { server.deployment.hooks.postBuild = mkForce [ ./script.sh ]; } if you only want script.sh to run after the build.

Hooks would be awesome to implement, but it can also get pretty hairy if we try to do everything at once. I think it might be worth just throwing something together as a start (maybe even only or two hook points), and add it to master behind some --experimental flags, so that we can iterate on it without making people think that it's a done feature.

Agreed, I would only start small. Maybe we need to figure out how we can cut the morph deployments into more logical phases and then follow the intuition one gathers from stdenv where there is post$Phase, pre$Phase etc.. hooks.

[adamtulinius]

Maybe we need to figure out how we can cut the morph deployments into more logical phases and then follow the intuition one gathers from stdenv where there is post$Phase, pre$Phase etc.. hooks.

We've been thinking about "morph 2.x" for some time, basically turning morph from nested if-then-else spaghetti, to features made from composable primitives. Maybe those two are related?

(not actually ignoring the rest you wrote, but I wanted to reply to this specifically)

[andir]

Maybe we need to figure out how we can cut the morph deployments into more logical phases and then follow the intuition one gathers from stdenv where there is post$Phase, pre$Phase etc.. hooks.

We've been thinking about "morph 2.x" for some time, basically turning morph from nested if-then-else spaghetti, to features made from composable primitives. Maybe those two are related?

(not actually ignoring the rest you wrote, but I wanted to reply to this specifically)

I thik so. A fast paced development and actually getting rid of some legacy rather then trying to stick with some old design forever sounds good.

[amerocu]

I'm also looking into more tightly integrating morph into our deployment's workflow, and this seems what could work for us.
Our need is to just keep track of what and when the different phases of the deployments happens and orchestrate some small setup/clean-up of the systems.

Is there still some interest to move forward this solution, or better ideas have come up in the meantime?