pre-commit can't find modules

[austinbutler]

Describe the bug

pre-commit just doesn't seem to work for me any longer.

pre-commit run --all-files                                                                                                                                                                                           
Check hooks apply to the repository......................................Failed                                                                                                                                      
- hook id: check-hooks-apply                                                                              
- exit code: 1                                                                                                                                                                                                       
                                                                                                                                                                                                                     
Traceback (most recent call last):                                                                                                                                                                                   
  File "/nix/store/6qk2ybm2yx2dxmx9h4dikr1shjhhbpfr-python3-3.10.11/lib/python3.10/runpy.py", line 196, in _run_module_as_main                                                                                       
    return _run_code(code, main_globals, None,                                                                                                                                                                       
  File "/nix/store/6qk2ybm2yx2dxmx9h4dikr1shjhhbpfr-python3-3.10.11/lib/python3.10/runpy.py", line 86, in _run_code                                                                                                  
    exec(code, run_globals)                                                                                                                                                                                          
  File "/nix/store/z5ahhpxfx6jpcsn8r4lx96a93a7vpj33-pre-commit-3.3.3/lib/python3.10/site-packages/pre_commit/meta_hooks/check_hooks_apply.py", line 7, in <module>                                                   
    from pre_commit import git                                                                                                                                                                                       
  File "/nix/store/z5ahhpxfx6jpcsn8r4lx96a93a7vpj33-pre-commit-3.3.3/lib/python3.10/site-packages/pre_commit/git.py", line 9, in <module>                                                                            
    from pre_commit.util import CalledProcessError                                                                                                                                                                   
  File "/nix/store/z5ahhpxfx6jpcsn8r4lx96a93a7vpj33-pre-commit-3.3.3/lib/python3.10/site-packages/pre_commit/util.py", line 16, in <module>                                                                          
    from pre_commit import parse_shebang                                                                                                                                                                             
  File "/nix/store/z5ahhpxfx6jpcsn8r4lx96a93a7vpj33-pre-commit-3.3.3/lib/python3.10/site-packages/pre_commit/parse_shebang.py", line 7, in <module>
    from identify.identify import parse_shebang_from_file                                                                                                                                                            
ModuleNotFoundError: No module named 'identify'  

pre-commit run --all-files                                                                                                                                                                                           
Check hooks apply to the repository......................................Failed                                                                                                                                      
- hook id: check-hooks-apply                                                                              
- exit code: 1                                                                                                                                                                                                       
                                                                                                                                                                                                                     
Traceback (most recent call last):                                                                                                                                                                                   
  File "/nix/store/6qk2ybm2yx2dxmx9h4dikr1shjhhbpfr-python3-3.10.11/lib/python3.10/runpy.py", line 196, in _run_module_as_main                                                                                       
    return _run_code(code, main_globals, None,                                                                                                                                                                       
  File "/nix/store/6qk2ybm2yx2dxmx9h4dikr1shjhhbpfr-python3-3.10.11/lib/python3.10/runpy.py", line 86, in _run_code                                                                                                  
    exec(code, run_globals)                                                                                                                                                                                          
  File "/nix/store/z5ahhpxfx6jpcsn8r4lx96a93a7vpj33-pre-commit-3.3.3/lib/python3.10/site-packages/pre_commit/meta_hooks/check_hooks_apply.py", line 7, in <module>                                                   
    from pre_commit import git                                                                                                                                                                                       
  File "/nix/store/z5ahhpxfx6jpcsn8r4lx96a93a7vpj33-pre-commit-3.3.3/lib/python3.10/site-packages/pre_commit/git.py", line 9, in <module>                                                                            
    from pre_commit.util import CalledProcessError                                                                                                                                                                   
  File "/nix/store/z5ahhpxfx6jpcsn8r4lx96a93a7vpj33-pre-commit-3.3.3/lib/python3.10/site-packages/pre_commit/util.py", line 16, in <module>                                                                          
    from pre_commit import parse_shebang                                                                                                                                                                             
  File "/nix/store/z5ahhpxfx6jpcsn8r4lx96a93a7vpj33-pre-commit-3.3.3/lib/python3.10/site-packages/pre_commit/parse_shebang.py", line 7, in <module>
    from identify.identify import parse_shebang_from_file                                                                                                                                                            
ModuleNotFoundError: No module named 'identify'

Steps To Reproduce

I have a flake.nix like this:

{
  description = "Dev env";
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
    flake-utils.url = "github:numtide/flake-utils";
    flake-compat.url = "github:edolstra/flake-compat";
    flake-compat.flake = false;
  };
  outputs = {
    nixpkgs,
    flake-utils,
    ...
  }:
    flake-utils.lib.eachDefaultSystem (system: let
      pkgs = import nixpkgs {
        inherit system;
        overlays = [];
      };
    in {
      devShells.default = pkgs.mkShell {
        name = "dev-env";
        buildInputs = with pkgs; [
          just
          mariadb
          mycli
          pre-commit
        ];
        shellHook = ''
          pre-commit install
        '';
      };
    });
}

Running pre-commit run --all-files in the Nix env fails.

Expected behavior

pre-commit run --all-files works. This was working until the last week or two I think.

Notify maintainers

• @borisbabic
• @FRidh
• @MatrixManAtYrService

Metadata

The nixpkgs rev for the flake is at 7cc30fd5372ddafb3373c318507d9932bd74aafe.
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 6.3.7, NixOS, 23.11 (Tapir), 23.11pre494015.0eeebd64de8`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.15.1`
 - channels(root): `"home-manager, nixos"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[MatrixManAtYrService]

I was able to replicate this by using the flake.nix above and adding the following .pre-commit-config.yaml.

repos:
    - repo: meta
      hooks:
        - id: check-hooks-apply

It was probably caused by this: #235123. I'm surprised because I thought that pre-commit did its magic in virtualenvs built for the job. Perhaps this is not the case for the "meta" repo.

I'd love it if there were a smart fix which both kept pre-commit from cluttering up the PYTHONPATH (that's what the above PR fixes) and also didn't break pre-commit in this way. If such a fix doesn't show up soon, I won't complain about just reverting that PR. It's probably now brokener than it was 😞 .

[MatrixManAtYrService]

It's only the meta repo ones that fail. For instance:

# .pre-commit-config.yaml
repos:
  - repo: meta
    hooks:
      - id: check-hooks-apply
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.4.0
    hooks:
      - id: trailing-whitespace

...works half-way...

pre-commit run --all-files
Check hooks apply to the repository......................................Failed
- hook id: check-hooks-apply
- exit code: 1

/nix/store/6jx4nc20v3cfphlbypqid30q41i5vh5x-python3-3.10.11/bin/python3.10: Error while finding module specification for 'pre_commit.meta_hooks.check_hooks_apply' (ModuleNotFoundError: No module named 'pre_commit')

trim trailing whitespace.................................................Passed

Other package managers have solved this. For instance, you can brew install pre-commit and remove it from the flake and then you get both:

• functional meta hooks
• a PYTHONPATH that is not populated by pre-commit's dependencies.

I don't know how they achieve this.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/where-did-you-get-stuck-in-the-nix-ecosystem-tell-me-your-story/31415/36

[autra]

Just a wild guess, did you try to remove all the installed hooks and run pre-commit install -t <hook> again? (I need to do that every time there is an update)