pre-commit: fixing import bug for built-in hooks

[MatrixManAtYrService]

Context

Pre-commit has built-in "meta" hooks. If they are isolated from pre-commit itself, they don't work.

#235123 prevented pre-commit's dependencies from propagating into devshells. This was necessary because those dependencies were clobbering others in the devshell definition.

This increased isolation broke the "meta" hooks. A workaround emerged (see comments below) which involved configuring the pre-commit wrapper script to always run with its PYTHONPATH set such that the pre-commit internals are accessible.

The workaround clutters up the devshell definition, and it needlessly exposes system type hooks to pre-commit internals. Really, only the meta type hooks need to see pre-commit internals.

Description of changes

This PR patches pre-commit so that it treats meta and system hook types separately, ensuring that pre-commit itself (via PYTHONPATH) is available only to the meta hooks. This allows users to just add pkgs.pre-commit to their devshell without thinking about PYTHONPATH at all.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.11 Release Notes (or backporting 23.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[MatrixManAtYrService]

Where I broke this: #235123
A previous attempt to fix it: #241698
Fixes: #238345

[MatrixManAtYrService]

Pre-commit has a "language" called system (docs) which lets you bypass pre-commit's encapsulation and just run commands in your "system".

By setting the pythonpath in the wrapper, we've fixed what we set out to fix. But we're clobbering pre-commit's idea of what's available in "system". In my case, I've got python311 in the calling shell, but my system hooks are using python310 because that's the version that pre-commit is packaged with.

Perhaps the wrapper needs to either:

• merge the calling environment with the packaged values instead of clobbering the former with the latter
• make note of what's in the env before clobbering it so that system hooks can un-clobber

To reproduce:

# flake.nix
{
  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:MatrixManAtYrService/nixpkgs?ref=fix-pre-commit";
  };

  outputs = { self, nixpkgs, flake-utils,}:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
        };
      in rec
      {

        devShells.default = pkgs.mkShell {
          packages = [
            pkgs.python311
            pkgs.pre-commit
          ];
        };
      });
}

# .pre-commit-hooks.yaml
---
repos:
  - repo: local
    hooks:
      - id: print-python-version
        name: print-python-version
        language: system
        entry: python --version
        pass_filenames: false

default_language_version:
  python: python3.11

❯ python --version
Python 3.11.4

❯ pre-commit run --verbose
print-python-version.....................................................Passed
- hook id: print-python-version
- duration: 0.02s

Python 3.10.12

[MatrixManAtYrService]

Ok, I fixed it by patching pre-commit to only expose the calling environment's PATH and PYTHONPATH to the hook runner if they're written in the system language. This broke the internal hooks (which are also system), so I created a new language (meta) for these.

• meta -> sees PATH and PYTHONPATH as prepared by nix
• system -> sees the calling environment's PATH and PYTHONPATH
• others -> pre-commit manages its environment so that its is isolated from both (no change here)

meta hooks are hard-coded into pre-commit. So far as I know, nobody is extending them. So this distinction need not be thought about by users (:crossed_fingers:).

[MatrixManAtYrService]

pandas has a nontrivial .pre-commit-config.yaml. I've been testing this by adding the following flake to that repo and running its pre-commit hooks.

{
  inputs = {
    flake-utils.url = "github:numtide/flake-utils";

    # switch the comment below to activate the other codebase
    nixpkgs.url = "github:MatrixManAtYrService/nixpkgs?ref=fix-pre-commit";
    # nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";

  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
        };
      in
      {

        devShells.default = pkgs.mkShell {
          packages = [
            pkgs.pre-commit
            pkgs.nodejs
          ];
        };
      });
}

Ruff is still a problem, but it's not a problem for this PR. As for everything else, this PR fixes a wide swath of hooks, all of which previously failed with errors like:

Unwanted patterns.......................................................................................Failed
- hook id: unwanted-patterns
- exit code: 1

/nix/store/pkj7cgmz66assy7l18zc7j992npb41nx-python3-3.10.12/bin/python3.10: Error while finding module specification for 'pre_commit.languages.pygrep' (ModuleNotFoundError: No module named 'pre_commit')

Also, it introduces no new failures.

It seems pretty clear that it's much better than it was before. I think it can be merged.

[MatrixManAtYrService]

@FRidh you made a very helpful comment on my previous attempt to address this issue, thanks for that. Unfortunately I introduced problems with that PR. It's a little more invasive, but I think this actually does trick while also preventing those problems. Can I get your thoughts and/or review on it?

[MatrixManAtYrService]

Thanks to @anund for pointing me at the wrapper script as a place to address this kind of problem. I've had to go a bit further than you recommended but thanks to that help I think I'm on the right path here. If you have any thoughts about this approach, they'd be valued.

[anund]

@MatrixManAtYrService I think you may be able to just define the python version your dev shell uses rather than patching pre-commit. This comes from thinking about what the 'system' is for your example. if pre-commit is using the wrong python version and you're setting that version in nix develop the question that came to mind was "how do you tell pre-commit how to use a different python?". First I look at override, and overrideAttrs which cover swapping things around at a slight cost on not getting upstream caching for free anymore.

To reproduce I had to adjust somethings in the example flake.nix you provided. I need a version of nixpkgs with the bug and to use overridePythonAttrs on pre-commit to add in just the makeWrapperArgs change.

#flake.nix
{
  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:nixos/nixpkgs?rev=a4832e490d62bdb1e60cab751a9ae38b4431934e";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
      in rec
      {
        devShells.default = pkgs.mkShell {
          packages = [
            pkgs.python311
            (pkgs.pre-commit.overrideAttrs (previousAttrs: {
              makeWrapperArgs = ''--set PYTHONPATH $PYTHONPATH'';
            }))
          ];
        };
      });
}

$nix develop and pre-commit install && pre-commit run --all --verbose

This gets us to the mismatch behaviour if we create .pre-commit-config.yaml the way you helpfully define above. Note: there is a overridePythonAttrs to interact with the buildPythonPackage call you may need to use in some other cases.

Updating the python version is ~easy since pre-commit takes python3Packages as an input. (python3Packages has the reference to python buildPythonPackage depends on)

#flake.nix
{
  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:nixos/nixpkgs?rev=a4832e490d62bdb1e60cab751a9ae38b4431934e";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
      in rec
      {
        devShells.default = pkgs.mkShell {
          packages = [
            pkgs.python311
            # example getting the parens right, you should probably define pre-commit above in the let
            ((pkgs.pre-commit.override{ python3Packages = pkgs.python311Packages; }).overrideAttrs (previousAttrs: {
              makeWrapperArgs = ''--set PYTHONPATH $PYTHONPATH'';
            }))
          ];
        };
      });
}

Note you'll probably use python311Packages.<package-name> to keep any nix supplied python deps in line with your python3 version. (propogatedBuildInputs would probably bring in a valid python version for other python libs the way you had to work around for pre-commit)

[anund]

Hmm, looking a bit closer at the suggested fix the way the python path is built is a bit wrong. It captures the pythonPath generated during build that includes more than just a reference to python. A path like the following might be closer to what is needed. There is also --prefix or --suffix to consider see make-wrapper.sh. pre-commit lib would be $out/${pkgs.python311Packages.python.sitePackages}.

--prefix PYTHONPATH ${pkgs.python311Packages.python}/${pkgs.python311Packages.python.sitePackages}`

[MatrixManAtYrService]

I finally got a chance to meditate on @anund's comments. For the particular repo I'm most worried about (work stuff), this seems pretty close:

        devShells.default = pkgs.mkShell {
          packages = [
            pkgs.poetry
            pkgs.python311
            # pkgs.pre-commit
            (pkgs.pre-commit.overrideAttrs (previousAttrs: {
              makeWrapperArgs = ''
                --set PYTHONPATH $PYTHONPATH
                --suffix PYTHONPATH : ${pkgs.python311Packages.ruamel-yaml}/lib/python3.11/site-packages
                --suffix PYTHONPATH : .
              '';
            }))

The above overrides won't work for everyone. Your hooks may need different paths, but the --set line ought to get most people pretty far.

I'm now considering just closing this PR and writing a blog post about how to make pre-commit work inside of nix.

[f1rstlady]

@anund's approach almost solves the problem. The last hurdle for me was that the propagated build inputs needed to be injected to the environment:

pkgs.pre-commit.overrideAttrs(prevAttrs:
  with pkgs.python3Packages;
  let depsPath = makePythonPath prevAttrs.propagatedBuildInputs;
  in {
    makeWrapperArgs = (prevAttrs.makeWrapperArgs or []) ++ [
      "--prefix PYTHONPATH : ${depsPath}"
      "--prefix PYTHONPATH : $out/${python.sitePackages}"
    ];
  }
);

I'm not sure whether a difference between python3Packages.python and the python version used by the pre-commit derivation might occur and pose a problem, though.

[l0b0]

Thanks for the fix! I've written a test for a different issue which might've been caused by the same underlying problem. Feel free to cherry-pick it.

[MatrixManAtYrService]

Thanks for the test! I'm unaccustomed to nixpkgs development so it'll be good for me to look that over.

It's been a little while since I've thought about this PR, but it's very nice to see it pop back up. I was feeling a little bit alone in my struggles against pre-commit.

[imincik]

Result of nixpkgs-review pr 251418 run on x86_64-linux 1

4 packages failed to build:

• python311Packages.asteroid-filterbanks
• python311Packages.asteroid-filterbanks.dist
• python311Packages.pyannote-audio
• python311Packages.pyannote-audio.dist

2 packages built:

• pre-commit
• pre-commit.dist

[MatrixManAtYrService]

I didn't know about that tool, that's pretty cool. I'll investigate how those packages are using pre-commit so I can avoid breaking them.

[fidgetingbits]

Just ran into this while working on a flake. I used the override from #251418 (comment) for now and it works, but think it would be nice when this gets in rather than having to manually update every flake that uses pre-commit.

I've never used nixpkgs-review before, but just gave it a run on x86_64-linux as well and didn't get any of the build errors mentioned above (maybe I did something wrong):

nix-shell -p nixpkgs-review.out --run 'nixpkgs-review pr 251418'
this path will be fetched (0.06 MiB download, 0.32 MiB unpacked):
  /nix/store/rx6laqr63fhb15sysy8z4magbijb8g0i-nixpkgs-review-2.10.4
copying path '/nix/store/rx6laqr63fhb15sysy8z4magbijb8g0i-nixpkgs-review-2.10.4' from 'https://cache.nixos.org'...
$ git -c fetch.prune=false fetch --no-tags --force https://github.com/NixOS/nixpkgs master:refs/nixpkgs-review/0 pull/251418/head:refs/nixpkgs-review/1
remote: Enumerating objects: 183036, done.
remote: Counting objects: 100% (29281/29281), done.
remote: Compressing objects: 100% (1251/1251), done.
remote: Total 183036 (delta 28476), reused 28580 (delta 27952), pack-reused 153755
Receiving objects: 100% (183036/183036), 109.76 MiB | 6.52 MiB/s, done.
Resolving deltas: 100% (108115/108115), completed with 7618 local objects.
From ssh://github.com/NixOS/nixpkgs
 * [new branch]                master                -> refs/nixpkgs-review/0
 * [new ref]                   refs/pull/251418/head -> refs/nixpkgs-review/1
$ git worktree add /home/aa/.cache/nixpkgs-review/pr-251418/nixpkgs 971ea0a28ad94b23d221c8377120340a303fc736
Preparing worktree (detached HEAD 971ea0a28ad9)
Updating files: 100% (41726/41726), done.
HEAD is now at 971ea0a28ad9 Merge pull request #320833 from matteo-pacini/easytag-resurrection
$ git merge --no-commit --no-ff ed76e4ddcfde2f8b2891455e7e61b71cfc6da502
Auto-merging pkgs/tools/misc/pre-commit/default.nix
Automatic merge went well; stopped before committing as requested
$ nix build --file /nix/store/rx6laqr63fhb15sysy8z4magbijb8g0i-nixpkgs-review-2.10.4/lib/python3.11/site-packages/nixpkgs_review/nix/review-shell.nix --nix-path nixpkgs=/home/aa/.cache/nixpkgs-review/pr-251418/nixpkgs nixpkg
s-overlays=/run/user/1000/tmpbsohbw02 --extra-experimental-features nix-command no-url-literals --no-link --keep-going --no-allow-import-from-derivation --option build-use-sandbox relaxed --argstr system x86_64-linux --argst
r nixpkgs-path /home/aa/.cache/nixpkgs-review/pr-251418/nixpkgs --argstr nixpkgs-config-path /run/user/1000/tmpufvqw8fw.nix --argstr attrs-path /home/aa/.cache/nixpkgs-review/pr-251418/attrs.nix

Link to currently reviewing PR:
https://github.com/NixOS/nixpkgs/pull/251418

6 packages built:
pre-commit pre-commit.dist python311Packages.asteroid-filterbanks python311Packages.asteroid-filterbanks.dist python311Packages.pyannote-audio python311Packages.pyannote-audio.dist

$ /nix/store/j7rp0y3ii1w3dlbflbxlv4g7hbaaz3bs-nix-2.18.2/bin/nix-shell --argstr system x86_64-linux --argstr nixpkgs-path /home/aa/.cache/nixpkgs-review/pr-251418/nixpkgs --argstr nixpkgs-config-path /run/user/1000/tmpufvqw8
fw.nix --argstr attrs-path /home/aa/.cache/nixpkgs-review/pr-251418/attrs.nix --nix-path nixpkgs=/home/aa/.cache/nixpkgs-review/pr-251418/nixpkgs nixpkgs-overlays=/run/user/1000/tmpbsohbw02 /nix/store/rx6laqr63fhb15sysy8z4ma
gbijb8g0i-nixpkgs-review-2.10.4/lib/python3.11/site-packages/nixpkgs_review/nix/review-shell.nix

[MatrixManAtYrService]

@imincik, I wonder if something has changed, because like @fidgetingbits, I also find that nixpkgs-review pr 251418 succeeds.

I spent a bit of time with asteroid-filterbanks and had a hard time getting the pre-commit hooks to succeed with or without this change (see ☝️ for details about the failure).

I also tested this change with pyannote-audio. It fails with the same error in all three cases:

• pre-commit installed via pip
• pre-commit installed via nix (without this change)
• pre-commit installed via nix (with this change)

pyannote/audio/core/io.py:37:1: F401 'numpy as np' imported but unused

I'm not sure if these are cases where pre-commit is failing to be as deterministic as it aims to, or whether they're cases where maintainers have pushed commits without first running their pre-commit hooks, but as far as I can see, there's no indication that this change makes things any worse in those repos.

[l0b0]

I'm getting errors from nixpkgs-review pr 251418 after rebasing onto master.