Guidance on pre-commit + monorepos

[snakescott]

Any thoughts on using pre-commit in a monorepo (http://danluu.com/monorepo/) setting, with each project having its own .pre-commit-config.yaml file? For example, will pre-commit be happy/sad with

• project root not being repo root
• multiple invocations of pre-commit install
• etc

I expect to play around and experiment but any early guidance (e.g., "I have seen people try and fail at this, beware" or "Yelp actually does exactly this with steps X, Y, Z") is welcome!

[asottile]

Aww man I typed up a response to this and never pressed reply. I guess I'll try and remember what I said the first time!

Though it's not explicitly designed for a monorepo I can imagine one way you could use the existing configuration to support this goal with a single configuration file.

# configure shared hooks that you want to run on all repositories here
-   repo: https://github.com/pre-commit/pre-commit-hooks.git
    sha: v0.6.1
    hooks:
    -   id: trailing-whitespace
    -   id: end-of-file-fixer

# For hooks that only apply to specific repositories, list them below
-   repo: https://github.com/pre-commit/pre-commit-hooks.git
    sha: v0.6.1
    hooks:
    -   id: flake8
        name: 'flake8 (for repo1)'
        # Only run on repo1 files
        files: '^repo1/.*\.py'
    -   id: flake8
        name: 'flake8 (for repo2)'
        # Only run on repo2 files
        files: '^repo2/.*\.py'
        # maybe repo2 wants specific argument configuration
        args: [--max-line-length=131]
    # maybe repo3 is python3 only
    -   id: flake8
        name: 'flake8 (for repo3)'
        files: '^repo3/.*\.py'
        language_version: python3

    # etc.

Note that the same repo can be configured multiple times and within the repo the same hook id can also be repeated

[asottile]

@snakescott any additional thoughts on this, does this seem reasonable?

[asottile]

Going to close this as answered, haven't heard back in a few months. If this isn't sufficient, please comment and I'll reopen :)

[alkuzad]

@asottile wouldn't that be beneficial to create "folders" array which is compiled into file regexp equal to ^(folder1|folder2|folder3){os.sep}* ?

[asottile]

I see no reason to have two ways to do things

[dpinol]

The problem with this approach is that you need to specify the path for each project multiple times.

Without explicit support, I need to specify the project path 3 times for pylint:

    -   id: pylint
        name: pylint
        entry: env PYTHONPATH=hubtype_babel/deploy_app python -m pylint.__main__ --rcfile=hubtype_babel/deploy_app/pylintrc
        language: system
        types: [python]
        files: hubtype_babel/deploy_app/.*/.*py

Maybe adding a new "working_folder" hook field which just performs a "cd" before running the hook could be a quick win?

thanks

[asottile]

I'm not inclined to extend support for this:

• it's complicated to implement
  • have to mutate the arguments that git hands us for filenames
  • have to change directory (breaks a lot of current invariants)
  • the configuration would have to understand that directory has changed
    • this is perhaps the biggest one as it would then be confusing what files means, does it refer to patterns from the root of the repository?
• monorepos are by-and-far the special case and not the normal case
• I won't personally use the feature so it'll likely be broken / unpolished
• there's already a working solution which does exactly what you want
• pylint is pretty weird as-is requiring two of the three oddities (flake8 for instance would discover the config file from the paths passed to it, and in all likelihook it would also detect the path based on the config file)

fwiw you can optimize your configuration slightly by using:

    files: ^hubtype_babel/

(no need for .*/.*py, the .* already matches everything and types: [python] already limits you to python files)

[thejcannon]

One drawback is I think two different unrelated subfolders would clobber the other's config when you pre-commit install in their respective directories (or at least, that's what I would gather from looking at the hardcoded paths in pre-commit git hook that was installed).

That means I can't really concurrently work on two projects in the same monorepo.

[asottile]

if you name it anything other than .pre-commit-config.yaml in the subdirectories pre-commit install will only pick up the root one (you'll of course need to use --config when invoking pre-commit-in-pre-commit

[thejcannon]

The core issue is I can only have one pre-commit file, and it lives at the repo root, and can't live in subdirectory roots. Which means there isn't really anything pre-commit (the tool) can reasonably do to address it.

[asottile]

you can likely adapt this solution

[SebastianoF]

Thanks for developing pre-commit!
Also great that it is possible to have multiple keys, as id in the same yaml, this simplifies a lot for the situation of a monorepo.

Back on the best practice, do you think it makes sense to have a pre-commit-config in each subfolder containing a library and then a config in the versioned controlled root folder, pointing at them?

root folder monorepo
- subfolder1 # first library
    - .pre-commit-config.yaml
-subfolder2 # second library
    - .pre-commit-config.yaml
.pre-commit-config.yaml

Where the root folder monorepo/pre-commit-config.yaml contains something like:

# .pre-commit-config.yaml
configs:
- subfolder1//pre-commit-config.yaml
- subfolder2//pre-commit-config.yaml

?

In this way each library can be run independently and can have its own configuration (or read the configuration from its pyproject.toml if using the PEP-518 convention), and there is a "global" .pre-commit-config.yaml to install the git hooks.

[asottile]

please don't grave dig 4 year old issues

and no, that will not be implemented