From adb335f39f48aa98b29848b36bdd660a5cf12fd7 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 4 May 2023 11:07:25 +0200 Subject: [PATCH] add guide on post build hook originally written under LGPL 2.1 for the Nix reference manual. authors as recorded in the Nix repository: git log --pretty="Co-authored-by: %an <%ae>" -- doc/manual/advanced-topics/post-build-hook.xml doc/manual/src/advanced-topics/post-build-hook.md | sort | uniq relicenced with authors' permission to CC-BY-SA 4.0: https://github.com/NixOS/nix.dev/issues/463 Co-authored-by: Eelco Dolstra Co-authored-by: Graham Christensen Co-authored-by: Joe Hermaszewski Co-authored-by: Kevin Stock Co-authored-by: endgame Co-authored-by: regnat --- source/recipes/post-build-hook.md | 123 ++++++++++++++++++++++++++++++ 1 file changed, 123 insertions(+) create mode 100644 source/recipes/post-build-hook.md diff --git a/source/recipes/post-build-hook.md b/source/recipes/post-build-hook.md new file mode 100644 index 000000000..2d33cbdac --- /dev/null +++ b/source/recipes/post-build-hook.md @@ -0,0 +1,123 @@ +# Using the `post-build-hook` + +# Implementation Caveats + +Here we use the post-build hook to upload to a binary cache. This is a +simple and working example, but it is not suitable for all use cases. + +The post build hook program runs after each executed build, and blocks +the build loop. The build loop exits if the hook program fails. + +Concretely, this implementation will make Nix slow or unusable when the +internet is slow or unreliable. + +A more advanced implementation might pass the store paths to a +user-supplied daemon or queue for processing the store paths outside of +the build loop. + +# Prerequisites + +This tutorial assumes you have [configured an S3-compatible binary +cache](../package-management/s3-substituter.md), and that the `root` +user's default AWS profile can upload to the bucket. + +# Set up a Signing Key + +Use `nix-store --generate-binary-cache-key` to create our public and +private signing keys. We will sign paths with the private key, and +distribute the public key for verifying the authenticity of the paths. + +```console +# nix-store --generate-binary-cache-key example-nix-cache-1 /etc/nix/key.private /etc/nix/key.public +# cat /etc/nix/key.public +example-nix-cache-1:1/cKDz3QCCOmwcztD2eV6Coggp6rqc9DGjWv7C0G+rM= +``` + +Then update [`nix.conf`](../command-ref/conf-file.md) on any machine that will access the cache. +Add the cache URL to [`substituters`](../command-ref/conf-file.md#conf-substituters) and the public key to [`trusted-public-keys`](../command-ref/conf-file.md#conf-trusted-public-keys): + + substituters = https://cache.nixos.org/ s3://example-nix-cache + trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= example-nix-cache-1:1/cKDz3QCCOmwcztD2eV6Coggp6rqc9DGjWv7C0G+rM= + +Machines that build for the cache must sign derivations using the private key. +On those machines, add the path to the key file to the [`secret-key-files`](../command-ref/conf-file.md#conf-secret-key-files) field in their [`nix.conf`](../command-ref/conf-file.md): + + secret-key-files = /etc/nix/key.private + +We will restart the Nix daemon in a later step. + +# Implementing the build hook + +Write the following script to `/etc/nix/upload-to-cache.sh`: + +```bash +#!/bin/sh +set -eu +set -f # disable globbing +export IFS=' ' +echo "Uploading paths" $OUT_PATHS +exec nix copy --to "s3://example-nix-cache" $OUT_PATHS +``` + +> **Note** +> +> The `$OUT_PATHS` variable is a space-separated list of Nix store +> paths. In this case, we expect and want the shell to perform word +> splitting to make each output path its own argument to `nix +> store sign`. Nix guarantees the paths will not contain any spaces, +> however a store path might contain glob characters. The `set -f` +> disables globbing in the shell. +Then make sure the hook program is executable by the `root` user: + +```console +# chmod +x /etc/nix/upload-to-cache.sh +``` + +# Updating Nix Configuration + +Edit `/etc/nix/nix.conf` to run our hook, by adding the following +configuration snippet at the end: + + post-build-hook = /etc/nix/upload-to-cache.sh + +Then, restart the `nix-daemon`. + +# Testing + +Build any derivation, for example: + +```console +$ nix-build -E '(import {}).writeText "example" (builtins.toString builtins.currentTime)' +this derivation will be built: + /nix/store/s4pnfbkalzy5qz57qs6yybna8wylkig6-example.drv +building '/nix/store/s4pnfbkalzy5qz57qs6yybna8wylkig6-example.drv'... +running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'... +post-build-hook: Signing paths /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example +post-build-hook: Uploading paths /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example +/nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example +``` + +Then delete the path from the store, and try substituting it from the +binary cache: + +```console +$ rm ./result +$ nix-store --delete /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example +``` + +Now, copy the path back from the cache: + +```console +$ nix-store --realise /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example +copying path '/nix/store/m8bmqwrch6l3h8s0k3d673xpmipcdpsa-example from 's3://example-nix-cache'... +warning: you did not specify '--add-root'; the result might be removed by the garbage collector +/nix/store/m8bmqwrch6l3h8s0k3d673xpmipcdpsa-example +``` + +# Conclusion + +We now have a Nix installation configured to automatically sign and +upload every local build to a remote binary cache. + +Before deploying this to production, be sure to consider the +[implementation caveats](#implementation-caveats).