docs: add host-custom-binary-cache-with-s3.md #140
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
| ## Introduction {#introduction} | ||
|
|
||
| In [Adding Custom Cache Servers](../nixos-with-flakes/add-custom-cache-servers.md), we | ||
| learned how to add custom binary cache servers to speed up the build process. In this |
There was a problem hiding this comment.
It's unnecessary for most of the readers of this book, add custom binary cache servers to speed up the build processis a bit obscure that it will be difficult for readers to understand what this is for and whether they need it.
So I would like to explain the benefits of this Custom Binary Cache with S3 here, who will need it, what kind of prerequisites for reading this.
An example for reference: https://nixos-and-flakes.thiscute.world/best-practices/remote-deployment
There was a problem hiding this comment.
Resolved in 7996ef3
|
Thanks for your contribution. I added some comments on the content to make it more reader-friendly and easier to follow. |
I've resolved changes requested per suggestion. Feel free to review it again. Cheers. |
| multiple packages that depend on the same third package, but on different versions of it. | ||
| To achieve this, all packages are installed in the global nix store under `/nix/store/` | ||
| and are then symlinked to the respective locations. To verify a package's uniqueness, its | ||
| whole directory is hashed, and the hash put into the name of the package's main folder. |
There was a problem hiding this comment.
Not only its whole directory, but the whole derivation's data-structure is used to derive its hash.
This is so called the “input-addressed” model, since the output paths are computed from all the input of the build.
Details: https://github.com/NixOS/rfcs/blob/master/rfcs/0062-content-addressed-paths.md#detailed-design
There was a problem hiding this comment.
BTW, adding an example of the store paths here will make it more clear, e.g.
# List some store paths in /nix/store
$ ls /nix/store/ | head
001gp43bjqzx60cg345n2slzg7131za8-nix-nss-open-files.patch
001im7qm8achbyh0ywil6hif6rqf284z-bootstrap-stage0-binutils-wrapper-boot.drv
0029mr9vdfdkd3wzh6785w578j179m0l-pkg-config-wrapper-0.29.2.drv
0039201phg77q8n7djd530f46qq259yh-Librpcsvc-26.tar.gz.drv
0039fczca4mmdvs16v278q09njzibz9n-python3.11-gevent-22.10.2.drv
003bv9i3pipypw9zwcfg1s7wqa5g0ggw-fmtutil.sed
003ri07b1gfl0v2svxlapk5yjdrr0zhn-System.Reflection.Emit.4.0.1.nupkg.drv
003wkhdk1ph54r6gx6lxax53bpf0bf4s-aws-c-sdkutils-0.1.12.drv
004z9fdf9jqg29pgwh17a0qkqv7xwb3j-utf8-string-1.0.2.tar.gz.drv
0053frndrzncxwzyxffr9lnwx4kngvl5-texlive-fullwidth-0.1.drvThere was a problem hiding this comment.
Not only
its whole directory, but the whole derivation's data-structure is used to derive its hash. This is so called the “input-addressed” model, since the output paths are computed from all the input of the build.Details: https://github.com/NixOS/rfcs/blob/master/rfcs/0062-content-addressed-paths.md#detailed-design
I don't quite get what you mean here. Could you please propose your changes directly?
There was a problem hiding this comment.
BTW, adding an example of the store paths here will make it more clear, e.g.
# List some store paths in /nix/store $ ls /nix/store/ | head 001gp43bjqzx60cg345n2slzg7131za8-nix-nss-open-files.patch 001im7qm8achbyh0ywil6hif6rqf284z-bootstrap-stage0-binutils-wrapper-boot.drv 0029mr9vdfdkd3wzh6785w578j179m0l-pkg-config-wrapper-0.29.2.drv 0039201phg77q8n7djd530f46qq259yh-Librpcsvc-26.tar.gz.drv 0039fczca4mmdvs16v278q09njzibz9n-python3.11-gevent-22.10.2.drv 003bv9i3pipypw9zwcfg1s7wqa5g0ggw-fmtutil.sed 003ri07b1gfl0v2svxlapk5yjdrr0zhn-System.Reflection.Emit.4.0.1.nupkg.drv 003wkhdk1ph54r6gx6lxax53bpf0bf4s-aws-c-sdkutils-0.1.12.drv 004z9fdf9jqg29pgwh17a0qkqv7xwb3j-utf8-string-1.0.2.tar.gz.drv 0053frndrzncxwzyxffr9lnwx4kngvl5-texlive-fullwidth-0.1.drv
Same as above.
| dependency software versions were changed, this will result in a new hash for the final | ||
| package. | ||
|
|
||
| Using symlinks to install a package and link all the right dependencies to it also enables |
There was a problem hiding this comment.
Not only symlinks, but also commonly used environment variables are used for installing packages.
For example, nix develop will add packages' store paths into PATH and other environment variables directly, to make the specified packages are available inside the shell session.
# Enter a dev shell
$ nix develop
# Show env vars
$ env | grep PATH
# ...
PATH=/nix/store/2lisyqjjjz9lh8v44dhn3yxy4w3yjk61-pre-commit-3.6.0/bin:/nix/store/9l3wb8qqq68pa6pnabci8zz64f0w9jvd-bash-interactive-5.2p26/bin:/nix/store/vvfi9744bmr82kqz4wb4ii3rx37mbn5f-gcc-wrapper-12.3.0/bin:/nix/store/im19yhcbwyabhra9jlv2k64w8719dvah-gcc-12.3.0/bin:/nix/store/qpd69yfix5psmsbv251ky18mc1lxnx70-coreutils-9.4/bin:/nix/store/li8d43x054b9f9h427cychbzkz5kkrgy-cctools-binutils-darwin-wrapper-16.0.6-973.0.1/bin:/nix/store/79fqcmdsj5prcn7xcmfaj6i0qcrz9s13-cctools-binutils-darwin-16.0.6-973.0.1/bin:/nix/store/5grmcfil0mxlxs36rmii3nfa6f04bcrl-alejandra-3.0.0/bin:/nix/store/j4bxhmfcsx3bd1akjy5y0ws0v0dd4i6a-deadnix-1.2.0/bin:/nix/store/gh67wnx16v7iyvwdl9ykd167hq9q2qdv-statix-0.5.8/bin:/nix/store/yqc3qw6a6x76g2vp4ycys0mgmanz48mq-typos-1.16.25/bin:/nix/store/iwma79c31ndda98zxw7djmhdw6l6jswr-prettier-3.2.4/bin:/nix/store/k687y4hiraslz65qz8pli1wb5wmcabvn-clang-wrapper-16.0.6/bin:/nix/store/3ijvz3w7ka2wfl3yk77r9c79c0fzb9j9-clang-16.0.6/bin:/nix/store/cwkgbzhwdkjp4sff8m3hbqisd7z333cn-cctools-binutils-darwin-wrapper-16.0.6-973.0.1/bin:/nix/store/qpd69yfix5psmsbv251ky18mc1lxnx70-coreutils-9.4/bin:/nix/store/lb9k8p8c8bf8b5v4gayzyx656a8n9cbp-findutils-4.9.0/bin:/nix/store/ri18k5b34qlapg1908afz4z51z1b3fbl-diffutils-3.10/bin:/nix/store/g44k88l2xg41bgsdhsw609smypmiz6j7-gnused-4.9/bin:/nix/store/av92p4b35n68p837dqq6jq6gjdk6s19q-gnugrep-3.11/bin:/nix/store/spskfl77py9hyvh3f9sjsqcrbsaiprh0-gawk-5.2.2/bin:/nix/store/ivklia9w69144iys517bvbc0d8paj2ai-gnutar-1.35/bin:/nix/store/m7bk558n8sli5z52splzgvrih5gasbvc-gzip-1.13/bin:/nix/store/d6ki959pnsa3w8iydkfjcjrggyplk66i-bzip2-1.0.8-bin/bin:/nix/store/mfygsgqn2jnm14918y96rb7c9avvnpk0-gnumake-4.4.1/bin:/nix/store/0j90xxj4in0yk2dp7mpvsm8jqd9mrbs3-bash-5.2p26/bin:/nix/store/1s37mqrr4szz2r3fk4hm9rzzgk36jyk6-patch-2.7.6/bin:/nix/store/v4l2p3sm90qyfsfs7kyw1xb629f93z53-xz-5.4.6-bin/bin:/nix/store/69pprys9925xm0byrcb3b3nk73a7lszw-file-5.45/bin:/Users/ryan/.local/bin:/Users/ryan/go/bin:/Users/ryan/.config/emacs/bin:/opt/homebrew/Caskroom/miniforge/base/condabin:/Users/ryan/.local/bin:/Users/ryan/go/bin:/Users/ryan/.config/emacs/bin:/Users/ryan/.nix-profile/bin:/etc/profiles/per-user/ryan/bin:/run/current-system/sw/bin:/nix/var/nix/profiles/default/bin:/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin:/opt/homebrew/bin:/usr/local/bin:/opt/homebrew/bin:/usr/local/binThere was a problem hiding this comment.
TBH, I'd prefer you directly propose your changes to my work to make our life easier :)
| atomic updating. To make this clearer, let's think of an example where software X is | ||
| installed in an older version and should be updated. Software X is installed in its very | ||
| own directory in the global nix store and symlinked to the right directory, let's say | ||
| `/usr/local/bin/`. When the update is triggered, the new version of X is installed into |
There was a problem hiding this comment.
Packages will be linked to /run/current-system/sw/(NixOS) or /etc/profiles/per-user/your-username/(home-manager).
I am not sure whether it is appropriate to add this to here, just for reference.
There was a problem hiding this comment.
TBH, I'd prefer you directly propose your changes to my work to make our life easier :)
|
I have submitted a new PR #144 to add a detailed explanation of Nix Store and Binary Cache. |
Cool, thanks for your kindness. |
Summary
As the title suggests. Add step-by-step tutorial for self-hosting binary cache storage with S3 (Minio).