Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mirror: Add packages/options data to channel release. #35

Merged
merged 13 commits into from Mar 26, 2020

Conversation

@samueldr
Copy link
Member

samueldr commented Mar 24, 2020

Though, I heavily suggest you read the commits, there's more to this PR.

  • Adds shell.nix using flake-compat.
  • Allows running in dry-run mode for easier development.
  • Outputs the current running command.
  • Minor cleanups related to system use.

I can reduce the scope of this PR to the basics needed for the json data if desired.


The new version was tested by forcing the download of incomplete evals. Note latest instead of latest-finished!

DRY_RUN=1 ./mirror-nixos-branch.pl nixpkgs-unstable https://hydra.nixos.org/job/nixpkgs/trunk/tarball/latest

The same could not be done for NixOS yet.

This will fail with current channels, as there is no options/packages json file. This was tested using this temp workaround.

diff --git a/mirror-nixos-branch.pl b/mirror-nixos-branch.pl
index 4bdbd65..1c64006 100755
--- a/mirror-nixos-branch.pl
+++ b/mirror-nixos-branch.pl
@@ -207,8 +207,12 @@ if ($bucketReleases && $bucketReleases->head_key("$releasePrefix")) {
     if ($channelName =~ /nixos/) {
         downloadFile("nixos.channel", "nixexprs.tar.xz");
         downloadFile("nixos.iso_minimal.x86_64-linux");
+        eval {
         downloadFile("nixpkgs.tarball", "packages.json.br", "json-br");
+        } or print STDERR "warning: packages.json.br skipped!\n";
+        eval {
         downloadFile("nixos.options", "options.json.br", "json-br");
+        } or print STDERR "warning: options.json.br skipped!\n";
 
         if ($channelName !~ /-small/) {
             downloadFile("nixos.iso_minimal.i686-linux");

I don't like discriminating on buildproducts by using subtype, as I assume subtypes could realistically be duplicated in a build.

Though, we do not have a proper way to choose which build product.

For this build:

This is the buildproducts info:

          'buildproducts' => {
                               '1' => {
                                        'defaultpath' => '',
                                        'sha256hash' => 'd823328e76005677a227c8e5926ee0822cabbd720c8cfa6fecded9ce576a555d',
                                        'type' => 'file',
                                        'filesize' => 2740220,
                                        'name' => 'packages.json.br',
                                        'path' => '/nix/store/97i5pah4vy2x672z1kalh5g125s81gl0-nixpkgs-tarball-20.09pre218220.35b2ad79ff6/packages.json.br',
                                        'subtype' => 'json-br',
                                        'sha1hash' => '7516167caf27a1451ea38489260db1c67f4a07c6'
                                      },
                               '2' => {
                                        'type' => 'file',
                                        'defaultpath' => '',
                                        'sha256hash' => 'a3489657e72d1840a5efe37a65e59053ef83d5b2c31eb6403162f89d0e4c5e94',
                                        'subtype' => 'source-dist',
                                        'path' => '/nix/store/97i5pah4vy2x672z1kalh5g125s81gl0-nixpkgs-tarball-20.09pre218220.35b2ad79ff6/tarballs/nixpkgs-20.09pre218220.35b2ad79ff6.tar.xz',
                                        'name' => 'nixpkgs-20.09pre218220.35b2ad79ff6.tar.xz',
                                        'sha1hash' => '3b8677e95264d2041dafa121a485a007aec92156',
                                        'filesize' => 16762476
                                      }
                             },

The best alternative implementation, and more correct one, would be to allow passing a regex pattern to match on path, so even name could be duplicated. We know that a path can't.

@samueldr samueldr force-pushed the samueldr:feature/channel-json-data branch from 3284c6d to b5cccee Mar 24, 2020
@samueldr samueldr marked this pull request as ready for review Mar 24, 2020
@samueldr samueldr force-pushed the samueldr:feature/channel-json-data branch from b5cccee to 0568ef3 Mar 25, 2020
Passing a list of parameters to `system` automatically gives us peace of
mind with regards to shell escaping. This is because it will not rely on
a shell invocation to run the given parameters, but directly exec to it.

The left-over shell invocations all use redirection.
@garbas garbas requested review from garbas and edolstra Mar 25, 2020
mirror-nixos-branch.pl Show resolved Hide resolved
@samueldr samueldr force-pushed the samueldr:feature/channel-json-data branch from 8bf9d0c to 3cd3da0 Mar 25, 2020
@edolstra edolstra merged commit 3cd3da0 into NixOS:master Mar 26, 2020
@samueldr samueldr deleted the samueldr:feature/channel-json-data branch Mar 26, 2020
@nixos-discourse

This comment has been minimized.

Copy link

nixos-discourse commented on badde7c Mar 28, 2020

This commit has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/announcement-moving-nixos-org-to-netlify/6212/36

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.