Update https://nixos.org/nixpkgs/packages-unstable.json.gz more frequently

[matthewbauer]

We have started to use "Repology" data in Nixpkgs through nix-update. Repology takes the .json file provided by nixos.org and finds outdated packages. It updates hourly.

This works well, but the data is frequently out of date:

$ curl -I https://nixos.org/nixpkgs/packages-unstable.json.gz
HTTP/1.1 200 OK
Date: Sun, 25 Mar 2018 03:22:34 GMT
Server: Apache/2.4.29 (Unix) OpenSSL/1.0.2n
Strict-Transport-Security: max-age=15552000
Last-Modified: Wed, 07 Mar 2018 16:00:44 GMT
ETag: "13c573-566d4a9650dba"
Accept-Ranges: bytes
Content-Length: 1295731
Content-Type: application/json
Content-Encoding: x-gzip

Can we set this to update more frequently? It's currently more than 2 weeks old, giving us bad data from Repology.

[vcunat]

Doesn't this happen because of channel being blocked?

[matthewbauer]

Not exactly. The update happens once per week:

https://github.com/NixOS/nixos-org-configurations/blob/cf5d68f9e5d735f8ed47c710d31d8f821b5258cf/nixos-org/webserver.nix#L267

If the channel hasn't update since then, it will take even longer. Maybe this can be put into a release job?

[grahamc]

@matthewbauer easiest bet would be to just update the startAt to happen more frequently, and send a PR.

[edolstra]

@matthewbauer That actually means it runs every 20 minutes.

[vcunat]

There's some problem there. The last update of both JSONs points to commits done on July 10. We've had lots of channel bumps since then.

[vcunat]

@edolstra: could we possibly expose logs in some way for similar processes around channel updating? If only a couple people are authorized to see the logs, any issues are bottle-necked on them, even though typically anyone with nixpkgs push access can fix.

If you fear potential for misuse, maybe it wouldn't be too hard to restrict the access to people with push access or just an explicit subset who want to keep fixing these problems.

[samueldr]

This time, it looks like the error is:

[nix-shell:~/.../nixos/nixos-homepage]$ make nixpkgs/packages-unstable.json.gz
nixpkgs=$(nix-instantiate --find-file nixpkgs -I nixpkgs=https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz); \
(echo -n '{ "commit": "' && cat $nixpkgs/.git-revision && echo -n '","packages":' \
  && nix-env -f '<nixpkgs>' -I nixpkgs=https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz -qa --json --arg config 'import ./discover-more-packages-config.nix' \
  && echo -n '}') \
  | sed "s|$nixpkgs/||g" > nixpkgs/packages-unstable.json.tmp
error: 'mkHomepage' at /nix/store/kzah1l6dcmc0ci49bk2cpa0b2pjhi4m8-nixexprs.tar.xz/pkgs/development/r-modules/default.nix:54:18 called with unexpected argument 'biocVersion', at /nix/store/kzah1l6dcmc0ci49bk2cpa0b2pjhi4m8-nixexprs.tar.xz/pkgs/development/r-modules/default.nix:35:21
python -mjson.tool < nixpkgs/packages-unstable.json.tmp > /dev/null
No JSON object could be decoded
make: *** [Makefile:226: nixpkgs/packages-unstable.json] Error 1

But yeah, monitoring in some way that is accessible to at least a trusted set of users could help.

---

Fixed in e6d3c654427bafb68269c4b9bf7ed43e8090d7d2.

Verified using this change with a checkout of master.

[samueldr]

Well, there was another error 👀. It just updated.

[zimbatm]

How hard would it be to have the manual, packages-unstable.json.gz, NixOS options set be generated and published by a hydra jobset directly? It would be more natural if these things were updated simultaneous to a channel release.

[grahamc]

Closing this, as the issue is separate (monitoring, or a hydra publish job, etc.)