Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suppress 'duplicate module named xx' errors? #4008

Closed
OJFord opened this issue Sep 25, 2017 · 26 comments
Closed

Suppress 'duplicate module named xx' errors? #4008

OJFord opened this issue Sep 25, 2017 · 26 comments

Comments

@OJFord
Copy link

OJFord commented Sep 25, 2017

Am I right in thinking it's not presently possible to ignore errors for having multiple (as the error should really say, rather than 'duplicate') modules with the same name?

I'm running mypy on files in a repo that contains multiple packages with some commonality among them; for example they each have a src/wsgi.py, which renders:

error: Duplicate module named 'src.wsgi'

  1. I think this should be reworded Multiple modules named xx
  2. Would you be open to an --allow-same-name-modules/allow_same_name_modules = True or similar?
@gvanrossum
Copy link
Member

Those probably aren't packages, they're folders (or there are __init__.py files that shouldn't be there). You should really run mypy separately over each package in that case, since otherwise it won't know what to do when one of these is imported from another place being checked.

For this reason, implementing such a flag would not be easy, and probably not very useful.

@leinardi
Copy link

leinardi commented Sep 1, 2018

Hi @gvanrossum, I am writing a PyCharm/IntelliJ IDEA plugin to run mypy scanning and I am encountering the same issue.

PyCharm/IntelliJ IDEA code inspection API sends me a list of open files in the IDE but, if the file list contains both a __init__.py and a python file from the same package, I get the "Duplicate module named" error:

$ mypy --show-column-numbers  /home/leinardi/PycharmProjects/gsi/gsi/__init__.py /home/leinardi/PycharmProjects/gsi/gsi.py
gsi.py: error: Duplicate module named 'gsi'

Do you have any suggestion to scan both this files without running mypy separately for each of them?

@ilevkivskyi
Copy link
Member

I am writing a PyCharm/IntelliJ IDEA plugin to run mypy scanning

Would you like to contribute to existing one instead? https://github.com/dropbox/mypy-PyCharm-plugin

@leinardi
Copy link

leinardi commented Sep 1, 2018

Hi @ilevkivskyi, normally that would be the best thing to do, but I have already done adapting the code of my Pylint plugin to work with mypy.

This plugin is based on Checkstyle-IDEA plugin and offers also real-time scanning, checkin validations and various on-demand scan actions.

This is my latest build, is still not production ready but should give you an idea of what feature are there: Mypy-0.7.0.zip

@gvanrossum
Copy link
Member

$ mypy --show-column-numbers  /home/leinardi/PycharmProjects/gsi/gsi/__init__.py /home/leinardi/PycharmProjects/gsi/gsi.py
gsi.py: error: Duplicate module named 'gsi'

This is because apparently you seem to have both a folder named gsi and a file name gsi.py. This is a problem for Python too (IIRC the folder/package wins) so you can't really blame mypy for complaining about this.

@leinardi
Copy link

leinardi commented Sep 1, 2018

Oh ok, thanks for the clarification 👍.

@leinardi
Copy link

leinardi commented Sep 1, 2018

Hi @gvanrossum, I am now getting the same error when I try to check these files from mypy project:

$ mypy  /home/leinardi/PycharmProjects/mypy/test-data/packages/typedpkg-stubs/setup.py /home/leinardi/PycharmProjects/mypy/extensions/setup.py /home/leinardi/PycharmProjects/mypy/setup.py
extensions/setup.py: error: Duplicate module named 'setup'

or just simply these 2 files:

$ mypy /home/leinardi/PycharmProjects/mypy/extensions/setup.py /home/leinardi/PycharmProjects/mypy/setup.py
setup.py: error: Duplicate module named 'setup'

Is it also in this case a project problem? And why it does not happen when I scan the entire project using mypy /home/leinardi/PycharmProjects/mypy/?

@gvanrossum
Copy link
Member

Presumably the folders containing the setup.py files don't have __init__.py files, so mypy treats them both as top-level modules. You can fix this using --scripts-are-modules: https://mypy.readthedocs.io/en/latest/command_line.html#miscellaneous

@leinardi
Copy link

leinardi commented Sep 2, 2018

Hey @gvanrossum, thank you for your answer. You are correct, the folders don't have __init__.py files.

But unfortunately the option --scripts-are-modules is not fixing the issue:

$ mypy --scripts-are-modules extensions/setup.py setup.py
setup.py: error: Duplicate module named 'setup'

@gvanrossum
Copy link
Member

gvanrossum commented Sep 2, 2018 via email

@leinardi
Copy link

leinardi commented Sep 2, 2018

Done, but still no answer:
image

But isn't this a mypy bug? why I cannot scan these 2 files together but it is fine if I scan the entire directory with mypy .?

@petergaultney
Copy link

petergaultney commented Oct 31, 2018

I have this problem frequently where I'm scanning separate packages with pre-commit hooks. It would be very useful to be able to disable this check completely, or else have an obvious workaround.

Scripts-are-modules does not help.

@gvanrossum
Copy link
Member

You'll have to write a script that scans the packages and produces a full list of files to pass to mypy except for the files you don't want.

@OJFord
Copy link
Author

OJFord commented Oct 31, 2018

That's exactly when this problem occurs - when that full list of files contains 'duplicate modules named x'.

(An example of the script that does that is pre-commit --all-files.)

@gvanrossum
Copy link
Member

Obviously that's a bug in your script, not in mypy. :-)

@OJFord
Copy link
Author

OJFord commented Feb 5, 2020

I hit this again, but found a solution (for specifically the pre-commit case) that seems to be workable:

- id: mypy
   name: mypy-one
    files: ^one/
    entry: mypy one/
    pass_filenames: false
- id: mypy
   name: mypy-two
   files: ^two/
   entry: mypy two/
   pass_filenames: false
...

and of course whatever args or other options you were using on your single entry, but that's the barebones.

Bit of a pain to have to maintain this - especially because if you add a three/ it's not linted unless you remember to add it to the whitelist - but at least it seems to work.

(NB: Note that the id is the same on each (just not the name) and it's that that's the optional argument to pre-commit run, so you can still run them all with pre-commit run mypy.)

@br3ndonland
Copy link

br3ndonland commented Oct 13, 2020

Edit by hauntsaninja: I strongly recommend against disabling error code misc, putting in a dropdown so people don't copy it

The "duplicate module" error is in the misc category. You can see the error code by adding show_error_codes = True to mypy.ini. If you're not concerned about omitting the other miscellaneous checks, try adding disable_error_code = misc to mypy.ini.

[mypy]
disable_error_code = misc
show_error_codes = True
files = **/*.py

For the pre-commit case mentioned above (#4008 (comment), also see https://github.com/pre-commit/mirrors-mypy/issues/5), try using pre-commit/mirrors-mypy with language: system and pass_filenames: false.

language: system tells pre-commit to "rely on whatever state the user's machine is in" (https://github.com/pre-commit/mirrors-mypy/issues/3#issuecomment-547999959), which may include mypy already installed in the environment and a mypy.ini configuration file. This may be an acceptable tradeoff if running both pre-commit and mypy from the same environment, such as a virtual environment or CI environment. When using language: system for pre-commit in CI such as GitHub Actions, it is necessary to configure the Python environment before running pre-commit (set up Python and install dependencies).

pass_filenames: false tells pre-commit not to override the list of files to check. This can be helpful simply to avoid maintaining duplicate lists of files. mypy.ini uses glob pattern matching to specify files to check, but .pre-commit-config.yaml uses Python regex for its files key.

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v0.790
    hooks:
      - id: mypy
        language: system
        pass_filenames: false

@hauntsaninja
Copy link
Collaborator

hauntsaninja commented Nov 17, 2020

You should not use @br3ndonland's disable_error_code = misc hack... more details at #9727

@tuukkamustonen
Copy link

tuukkamustonen commented May 5, 2021

Unless I'm mistaken, none of the workarounds allow to scan all the files in a monorepo, which has something like:

component1/
    setup.py
    ...
component2/
    setup.py
   ...
tasks.py
some-other-file.py
  1. I don't want to ignore all "misc" errors, only this one issue in particular.
  2. Setting pass_filenames: false will cause scan of all files, not just those that were modified (for pre-commit hooks).
  3. I don't want --exclude any files - want everything scanned.
  4. Cannot use --scripts-are-modules (it's intended for non .py files)

Scanning above-like monorepo from main level simply does not work now, as each component holds their "duplicate" setup.py (tests/conftest.py, etc) files. In order to scan this sort of a structure, I would need to scan in parts:

  • Scan each directory separately (e.g. component1, component2)
  • Then scan the main level, excluding all components/sub-directories (something like .*/.*, I guess)

(And I'm not sure if I can even do this with pre-commit framework, need to study.)

Besides, what's the harm in having similarly named files in different directories? They are standalone .py files and those directories are not declared packages (there's no __init__.py anywhere). Even if they were considered packages in above example, there wouldn't be no clash (component1.setup vs component2.setup).

Now that 0.800+ supports recursive file scanning, could this issue be re-considered?

@tuukkamustonen
Copy link

Opened #10428 to perhaps better describe (or ask about) the problem.

benma added a commit to benma/bitbox02-firmware that referenced this issue Nov 18, 2021
Mypy now is severely confused by there being a bitbox02.py inside a
bitbox02 folder, resulting in duplicate module errors. See
python/mypy#4008.

gvanrossum writes:

> This is because apparently you seem to have both a folder named gsi
and a file name gsi.py. This is a problem for Python too (IIRC the
folder/package wins) so you can't really blame mypy for complaining
about this.

I don't understand this, as Python seems to handle it fine in our
package, but mypy does not.
benma added a commit to benma/bitbox02-firmware that referenced this issue Nov 22, 2021
Mypy now is severely confused by there being a bitbox02.py inside a
bitbox02 folder, resulting in duplicate module errors. See
python/mypy#4008.

gvanrossum writes:

> This is because apparently you seem to have both a folder named gsi
and a file name gsi.py. This is a problem for Python too (IIRC the
folder/package wins) so you can't really blame mypy for complaining
about this.

I don't understand this, as Python seems to handle it fine in our
package, but mypy does not.
Jasha10 added a commit to Jasha10/hydra that referenced this issue Feb 14, 2022
- check subdirs separately (python/mypy#4008)
- use mypy flags --install-types --non-interactive
- fixup noxfile: use lint_plugins to run mypy on plugins
- noxfile: run mypy on tools/ without --strict flag
@jaklan
Copy link

jaklan commented Apr 8, 2022

@tuukkamustonen described the issue perfectly in terms of monorepo usage - I also don't find any of these workarounds to resolve the issue. pass_filenames: false is an overkill, creating separate mypy hook for each of the projects in monorepo - as well. For now we are just excluding the problematic files, which is not really a solution... Is there any chance to have that issue revisited? Monorepos would stay there and the problem would be just more and more significant.

@billcrook
Copy link

To add another case where this breaks down, when authoring data pipelines in airflow it is common practice to name each pipeline dag.py. I cannot use mypy because of this.

adam-grant-hendry pushed a commit to adam-grant-hendry/pyvistaqt that referenced this issue Jun 28, 2022
`pre-commit` is known to override behavior from config files in
certain instances. This commit fixes known issues for:

- `isort`
- `black`
- `mypy`

For relevant details, see:

isort:
  + https://jugmac00.github.io/blog/isort-and-pre-commit-a-friendship-with-obstacles/
  + PyCQA/isort#885

black:
  + psf/black#1584

mypy:
  + python/mypy#4008 (comment)
  + https://pre-commit.com/#hooks-pass_filenames
adam-grant-hendry pushed a commit to adam-grant-hendry/pyvistaqt that referenced this issue Jun 28, 2022
`pre-commit` is known to override behavior from config files in
certain instances. This commit fixes known issues for:

- `isort`
- `black`
- `mypy`
- `pydocstyle`

For relevant details, see:

isort:
  + https://jugmac00.github.io/blog/isort-and-pre-commit-a-friendship-with-obstacles/
  + PyCQA/isort#885

black:
  + psf/black#1584

mypy/pydocstyle:
  + python/mypy#4008 (comment)
  + https://pre-commit.com/#hooks-pass_filenames
akaszynski pushed a commit to pyvista/pyvistaqt that referenced this issue Jun 29, 2022
* refactor(trove-classifiers): update to python 3.7-3.9

* refactor(linters): add `.flake8` and `.codespellrc` configuration files

Also add `PullRequest` to `ignore_words.txt`

* feat(pyproject): add `pyproject.toml`

Currently, this is only use for configuring linters and formatters (no
build file specifications are set). Configuration options for `black`
and `pydocstyle` are added here in this commit.

* feat(pre-commit): add pre-commit

* fix(mypy): add `mypy` dependency to `environment.yml`

This supports conda test/build environments.

* feat(pylint): Update `.pylintrc`

Make this match settings specififed in `Makefile`.

* feat(test): align checks

All checks are placed in appropriate config files. Test runner scripts
point to these so that there exists a single source of truth for all
configurations and that the tests performed across systems is the same.
This may be adjusted, of course, if different settings must be used on
a per-system basis.

* feat(requirements): add `toml`

This package supports reading `pyproject.toml`, which is required for
some of our tests.

* fix(pre-commit): fix file passing errors

`pre-commit` is known to override behavior from config files in
certain instances. This commit fixes known issues for:

- `isort`
- `black`
- `mypy`
- `pydocstyle`

For relevant details, see:

isort:
  + https://jugmac00.github.io/blog/isort-and-pre-commit-a-friendship-with-obstacles/
  + PyCQA/isort#885

black:
  + psf/black#1584

mypy/pydocstyle:
  + python/mypy#4008 (comment)
  + https://pre-commit.com/#hooks-pass_filenames

* feat(ignores): ignore specific linting errors

Ignore linting errors in source code in this PR. The purpose of this PR
is to update linting tools. A future PR will correct the errors. This is
done to separate concerns.

Co-authored-by: Hendry, Adam <adam.hendry@medtronic.com>
@adam-grant-hendry
Copy link

adam-grant-hendry commented Aug 18, 2022

For the pre-commit case mentioned above (#4008 (comment), also see pre-commit/mirrors-mypy#5), try using pre-commit/mirrors-mypy with language: system and pass_filenames: false.

language: system tells pre-commit to "rely on whatever state the user's machine is in" (pre-commit/mirrors-mypy#3 (comment)), which may include mypy already installed in the environment and a mypy.ini configuration file. This may be an acceptable tradeoff if running both pre-commit and mypy from the same environment, such as a virtual environment or CI environment. When using language: system for pre-commit in CI such as GitHub Actions, it is necessary to configure the Python environment before running pre-commit (set up Python and install dependencies).

pass_filenames: false tells pre-commit not to override the list of files to check. This can be helpful simply to avoid maintaining duplicate lists of files. mypy.ini uses glob pattern matching to specify files to check, but .pre-commit-config.yaml uses Python regex for its files key.

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v0.790
    hooks:
      - id: mypy
        language: system
        pass_filenames: false

Can you use a github repo if specifying language: system, or do you have to use repo: local instead?

@adam-grant-hendry
Copy link

adam-grant-hendry commented Aug 18, 2022

pass_filenames: false tells pre-commit not to override the list of files to check. This can be helpful simply to avoid maintaining duplicate lists of files. mypy.ini uses glob pattern matching to specify files to check, but .pre-commit-config.yaml uses Python regex for its files key.

I agree with @tuukkamustonen and @jaklan

This has already been said, but to reiterate: By default, pre-commit only runs against changed files (i.e. files that have been modified and added to your git staging area with git add). Unstaged files will not be checked. Specifying pass_filenames: false tells pre-commit to not pass staged files as input to mypy. As a result, mypy will run against all files in your repo (except those excluded in a glob in your .ini/toml config settings) every time pre-commit is run.

Hence everytime pre-commit is run, if pass_filenames: false is set, it will run against all files (not just changed ones) in your project (as if you simply entered mypy . --config-file=pyproject.toml), which the pre-commit author has stated will make your mypy hook run slower.

Sometimes specifying pass_filenames: false can be useful if you want to always run a hook against the same set of files (not just changed files in your staging area), for example if you have to use pytest as a hook. However, I don't think using it if you're only wanting to avoid "maintaining a duplicate list of files" is a good idea. I think using the files: and exclude: options are better suited for this.

The real issue has to do with mypy (not pre-commit) because it can't work well with namespace packages and monorepos due to the Duplicate module error.

mzat-msft added a commit to Azure/plato that referenced this issue Jun 2, 2023
Running mypy on the whole repo generates errors when two folders contain two
python files with the same name. This generates a mypy error and a simple
solution to it is not possible.
See related issues:
- python/mypy#10428
- python/mypy#4008

For this reason, we restrict type checking only on the main package code.
mzat-msft added a commit to Azure/plato that referenced this issue Jun 7, 2023
Running mypy on the whole repo generates errors when two folders contain two
python files with the same name. This generates a mypy error and a simple
solution to it is not possible.
See related issues:
- python/mypy#10428
- python/mypy#4008

For this reason, we restrict type checking only on the main package code.
mzat-msft added a commit to Azure/plato that referenced this issue Jun 12, 2023
Running mypy on the whole repo generates errors when two folders contain two
python files with the same name. This generates a mypy error and a simple
solution to it is not possible.
See related issues:
- python/mypy#10428
- python/mypy#4008

For this reason, we restrict type checking only on the main package code.
mzat-msft added a commit to Azure/plato that referenced this issue Jun 12, 2023
Running mypy on the whole repo generates errors when two folders contain two
python files with the same name. This generates a mypy error and a simple
solution to it is not possible.
See related issues:
- python/mypy#10428
- python/mypy#4008

For this reason, we restrict type checking only on the main package code.
mzat-msft added a commit to Azure/plato that referenced this issue Jun 12, 2023
Running mypy on the whole repo generates errors when two folders contain two
python files with the same name. This generates a mypy error and a simple
solution to it is not possible.
See related issues:
- python/mypy#10428
- python/mypy#4008

For this reason, we restrict type checking only on the main package code.
mzat-msft added a commit to Azure/plato that referenced this issue Jun 14, 2023
Running mypy on the whole repo generates errors when two folders contain two
python files with the same name. This generates a mypy error and a simple
solution to it is not possible.
See related issues:
- python/mypy#10428
- python/mypy#4008

For this reason, we restrict type checking only on the main package code.
mzat-msft added a commit to Azure/plato that referenced this issue Jun 15, 2023
Running mypy on the whole repo generates errors when two folders contain two
python files with the same name. This generates a mypy error and a simple
solution to it is not possible.
See related issues:
- python/mypy#10428
- python/mypy#4008

For this reason, we restrict type checking only on the main package code.
mzat-msft added a commit to Azure/plato that referenced this issue Jun 15, 2023
Running mypy on the whole repo generates errors when two folders contain two
python files with the same name. This generates a mypy error and a simple
solution to it is not possible.
See related issues:
- python/mypy#10428
- python/mypy#4008

For this reason, we restrict type checking only on the main package code.
@macieyng
Copy link

macieyng commented Jul 5, 2023

To those that are still looking for some solution for monorepo:

  - repo: local
    hooks:
      - id: mypy
        name: mypy
        entry: bash -c 'SUBREPO=$(echo "$@" | sed "s/\/.*//"); ABS_PATH="$PWD/$@"; cd $SUBREPO; mkdir -p .mypy_cache; mypy "$ABS_PATH" --install-types --non-interactive'
        language: python
        language_version: python3
        types: [python]
  1. First of all, specify pyproject.toml/mypy.ini for each of your sub-repo. You're likely to have different configs for each project.
  2. For mypy specifically - make sure you're using mypy installed in your local environment. By default pre-commit runs mypy in isolated environment that doesn't have all the info about other packages (like pydantic, sqlalchemy, etc.) and therefore will make mistakes. This shouldn't be an issue if you work in monorepo environment.
  3. This solution assumes that each sub-repo in monorepo has a sub-root directory under root (/project1, /project2, etc). SUBREPO is exactly this - the name of your project.
  4. ABS_PATH - absolute path to the file that will be checked, to avoid duplicate module named.
  5. We cd to the sub-repo directory and run mypy here on absolute path. Here is where your mypy config should be, if you share configuration among other sub-repos use --config-file ../.mypy.ini or similar. Use other args and flags as you wish.

Now mypy should have correct idea of what is going on. I did the same trick with bandit. It works correctly even if you want to commit edits on few projects in your monorepo. This solution my not be "clean", but works. It's the best I could come up with and pre-commit maintainer have said his word about supporting monorepos, so here we are.

I hope that some of you will find it useful.

kshramt pushed a commit to kshramt/evidence_based_scheduling that referenced this issue Jul 15, 2023
kshramt pushed a commit to kshramt/evidence_based_scheduling that referenced this issue Jul 15, 2023
@Soneji
Copy link

Soneji commented Sep 5, 2023

My solution was to add this option in the mypy configuration:

[tool.mypy]
packages = ["package1", "package2"]

tek08 added a commit to tek08/mypy-pycharm that referenced this issue Jan 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests