Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve pkgs/by-name code, minor fix #287083

Merged
merged 4 commits into from
Feb 8, 2024
Merged

Conversation

infinisil
Copy link
Member

@infinisil infinisil commented Feb 7, 2024

Description of changes

Mainly code cleanup, but that also discovered a problem, which is fixed with the cleanup.

See the commit messages for more details.

I'll merge this myself soon after CI is successful.

This work is sponsored by Antithesis and Tweag

Things done

  • Ran nix-build -A tests.nixpkgs-check-by-name.tests successfully on x86_64-linux

Add a 👍 reaction to pull requests you find important.

For some previously untested cases. In a future commit, those tests will
also be adjusted slightly
@infinisil infinisil added this to the RFC 140 milestone Feb 7, 2024
@infinisil infinisil changed the title Improve pkgs/by-name checking code Improve pkgs/by-name code, move sbcl out of pkgs/by-name Feb 7, 2024
@infinisil infinisil changed the title Improve pkgs/by-name code, move sbcl out of pkgs/by-name Improve pkgs/by-name code, minor fix Feb 7, 2024
@infinisil
Copy link
Member Author

We can't avoid pkgs/by-name CI failing here because it's a bit too clever: It detects that I'm moving sbcl out of pkgs/by-name again, but this is a rare case where it's totally justified :)

@hraban
Copy link
Member

hraban commented Feb 8, 2024

I'm not exactly clear on the details but I believe it :) thanks

Copy link
Contributor

@philiptaron philiptaron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I particularly like the changes in eval.rs

pkgs/test/nixpkgs-check-by-name/src/eval.nix Outdated Show resolved Hide resolved
pkgs/test/nixpkgs-check-by-name/src/eval.nix Outdated Show resolved Hide resolved
pkgs/test/nixpkgs-check-by-name/src/eval.rs Outdated Show resolved Hide resolved
@infinisil
Copy link
Member Author

@hraban The intuitive explanation is that for any pkgs/by-name/fo/foo, the corresponding attribute pkgs.foo should ideally be defined as callPackage pkgs/by-name/fo/foo/package.nix { }. This means that there's a one-to-one mapping from attributes to files without any other necessary context:

  • pkgs.foo is defined in pkgs/by-name/fo/foo/package.nix
  • pkgs/by/name/fo/foo/package.nix defines pkgs.foo

In reality we still need to allow customising the { } argument in all-packages.nix. In order to keep the necessary context minimal, CI checks that all definitions for packages in pkgs/by-name look like callPackage pkgs/by-name/fo/foo <attrs>, where only <attrs> can change.

- Detect manual use of _internalCallByNamePackageFile for packages in
  `pkgs/by-name` (can't be done for others though)
- Separate error message for when attribute locations can't be
  determined for `pkgs/by-name` attributes
- Much better structure of the code in eval.rs, representing more
  closely what is being checked
- Much more extensive comments
This adds a test to check that a commit like 0a3dab4 would fail CI

After doing some improvements to the `pkgs/by-name` check I discovered
that sbcl shouldn't have been allowed in `pkgs/by-name` after all as is.

Specifically, the requirement is that if `pkgs/by-name/sb/sbcl` exists,
the definition of the `sbcl` attribute must look like

    sbcl = callPackage ../by-name/sb/sbcl/package.nix { ... };

However it currently is an alias like

    sbcl = sbcl_2_4_1;

This wasn't detected before because `sbcl_2_4_1` was semantically
defined using `callPackage`:

    sbcl_2_4_1 = wrapLisp {
      pkg = callPackage ../development/compilers/sbcl { version = "2.4.1"; };
      faslExt = "fasl";
      flags = [ "--dynamic-space-size" "3000" ];
    };

However this doesn't syntactically match what is required.

In NixOS#285089 I introduced syntactic
checks for exactly this, but they were only used for packages not
already in `pkgs/by-name`.

Only now that I'm doing the refactoring to also use this check for
`pkgs/by-name` packages this problem is noticed.

While introducing this new check is technically an increase in
strictness, and therefore would justify adding a new ratchet, I consider
this case to be rare enough that we don't need to do that.

This commit introduces a test to prevent such regressions in the
future

Moving sbcl back out of `pkgs/by-name` will be done when the pinned CI is updated
This reverts commit 0a3dab4

See the parent commit as to why this is necessary
@infinisil
Copy link
Member Author

@philiptaron Thanks for the quick review! ❤️

Typos now fixed :D

@infinisil infinisil merged commit c442ab0 into NixOS:master Feb 8, 2024
22 of 25 checks passed
@infinisil infinisil deleted the by-name-cleanup branch February 8, 2024 14:51
infinisil added a commit to tweag/nixpkgs that referenced this pull request Feb 12, 2024
Emantor pushed a commit to Emantor/nixpkgs that referenced this pull request Feb 13, 2024
infinisil added a commit to NixOS/nixpkgs-check-by-name that referenced this pull request Mar 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants