Add nix-channels (preliminary)

[dramforever]

Having written this:

I have been asked (in personal communication) to move forward with Nix mirroring. So I've decided to slightly polish the script I originally had using a suggestion from @oxalica, and posting it here. Following the final exams I might post details on how I want this to work (or you can infer some from the script).

Some TODOs I can think of:

• Estimated size and rate of growth
• Maybe GC on old store paths

[dramforever]

Script dependency: Python 3 with requests and pyquery, Nix version >= 2.0 (Tested on 2.3.1)

[dramforever]

One thing I remembered: I have not contacted upstream about this! Please take care.

[dramforever]

File size data from https://groups.google.com/forum/#!msg/tuna-general/C1ljVn2xT78/XHHuFpgOCQAJ

一些实验数据

下面是这几天对nixos-19.09频道进行实验，统计的一些数据：

• store-paths.xz行数约38K；
• 对应的Nix存储（通过nix copy下载）：
  • 约45K个narinfo文件（大多数小于1K）；
  • 约45K个nar文件（xz压缩格式），大小不一；
    • >10K的约34K个；
    • >1M的约5K个；
    • >10M的约1K个；
    • >100M的约150个；
    • 最大的约900M；
  • 一共80G。
• 2020年1月11日的store-paths.xz有37969行，其中36187行和2020年1月6日的相同：
  • 约99%的包是不变的；
  • 考虑到底层依赖的包，这个比例应该更高。

[zhsj]

What nix copy does? Can we avoid some external commands. Otherwise you need to install it on the mirror server, or in the base docker image.

[yuchangyuan]

nix copy is necessary for operate on a Nix store. I think docker image is a good approach.

NixOS provide a way to generate docker image via a nix expression, so a we can create a nix package for the sync script and create the docker image easily based on this package.

And install Nix on other Linux distro is not hard, so I think we can also create a docker image based on Alpine(need to try) if docker image generated by nix is too large.

[dramforever]

nix copy in this context is essentially the official way to mirror. (Its more general use is copying store paths from one store to another, and in this case we are copying all paths in a channel from upstream to a local binary cache.) If we implement this ourselves it will need to find dependencies recursively by parsing the metadata and skip already existing paths, so there's quite a bit of logic to duplicate.

I believe (but have not tried) that installing Nix in an existing docker image is as easy as copying Nix itself in (Nix and deps from /nix/store). Nix wouldn't be a functioning package manager, but nix copy should work.

[dramforever]

Details of this script at: https://gist.github.com/dramforever/d2ff99318c70f44149db6070a87da5a0.