Mix packages and modules in command line parameters for Pylint #818

oleg-babintsev · 2021-10-08T16:19:15Z

Pylint is meant to be called from the command line. The usage is

pylint [options] modules_or_packages

You shoud passed modules OR packages. If you pass mix of modules and packages you should check that its not intercect between each other, because Pylint searches modules in packages and we got double messages in report (can live with it) or code duplication message in bad case (its a problem).

Test project (https://drive.google.com/file/d/13n1RZFdgwitfgUDOv76ZMYOPIrBYVsAT/view?usp=sharing):

a/__init__.py
a/module.py
a/b/__init__.py
a/b/module.py
b/__init__.py
b/module.py

Run pyb analyze -X:

[INFO]  Executing pylint on project sources
[DEBUG] Executing command: 'c:\\work\\projects\\misc\\pybuilder-pylint-issue\\.pybuilder\\plugins\\cpython-3.9.7.final.0\\scripts\\pylint.EXE' '--max-line-length=100' '--no-docstring-rgx=.*' 'a.module' 'a' 'a.b.module' 'a.b' 'b.module' 'b'

Report:

************* Module a.module
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\a\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\a\module.py:1:0: W0611: Unused import sys (unused-import)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\a\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\a\module.py:1:0: W0611: Unused import sys (unused-import)
************* Module a.b.module
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\a\b\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\a\b\module.py:1:0: W0611: Unused import sys (unused-import)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\a\b\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\a\b\module.py:1:0: W0611: Unused import sys (unused-import)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\a\b\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\a\b\module.py:1:0: W0611: Unused import sys (unused-import)
************* Module b.module
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\b\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\b\module.py:5:0: C0103: Function name "b" doesn't conform to snake_case naming style (invalid-name)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\b\module.py:1:0: W0611: Unused import sys (unused-import)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\b\module.py:2:0: W0611: Unused module imported from a as a (unused-import)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\b\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\b\module.py:5:0: C0103: Function name "b" doesn't conform to snake_case naming style (invalid-name)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\b\module.py:1:0: W0611: Unused import sys (unused-import)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\b\module.py:2:0: W0611: Unused module imported from a as a (unused-import)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\b\module.py:1:0: R0801: Similar lines in 2 files
==b.module:[0:6]
==b.module:[0:6]
import sys
from a import module as a
from a.b import module as ab

def b():
    print('b') (duplicate-code)

------------------------------------------------------------------

Your code has been rated at 0.50/10 (previous run: 0.50/10, +0.00)

If I execute Pylint and pass only modules or only packages - all works great.

pylint 'a' 'b':

************* Module a.module
a\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
a\module.py:1:0: W0611: Unused import sys (unused-import)
************* Module a.b.module
a\b\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
a\b\module.py:1:0: W0611: Unused import sys (unused-import)
************* Module b.module
b\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
b\module.py:5:0: C0103: Function name "b" doesn't conform to snake_case naming style (invalid-name)
b\module.py:5:0: C0116: Missing function or method docstring (missing-function-docstring)
b\module.py:1:0: W0611: Unused import sys (unused-import)
b\module.py:2:0: W0611: Unused module imported from a as a (unused-import)

------------------------------------------------------------------
Your code has been rated at 0.00/10 (previous run: 0.00/10, +0.00)

pylint 'a.module' 'a.b.module' 'b.module':

************* Module a.module
a\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
a\module.py:1:0: W0611: Unused import sys (unused-import)
************* Module a.b.module
a\b\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
a\b\module.py:1:0: W0611: Unused import sys (unused-import)
************* Module b.module
b\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
b\module.py:5:0: C0103: Function name "b" doesn't conform to snake_case naming style (invalid-name)
b\module.py:5:0: C0116: Missing function or method docstring (missing-function-docstring)
b\module.py:1:0: W0611: Unused import sys (unused-import)
b\module.py:2:0: W0611: Unused module imported from a as a (unused-import)

------------------------------------------------------------------
Your code has been rated at 0.00/10 (previous run: 0.00/10, +0.00)

The text was updated successfully, but these errors were encountered:

arcivanov · 2021-10-08T20:54:06Z

Hi. Thanks for the report.

Although I'm quite confused about what this is. Are you reporting a bug in the PyBuilder PyLint plugin (https://github.com/pybuilder/pybuilder/blob/master/src/main/python/pybuilder/plugins/python/pylint_plugin.py)?

arcivanov · 2021-10-08T20:58:16Z

Nevermind, I reread and I think I understood what you meant. Let me see if I can fix it quickly.

arcivanov · 2021-10-08T21:10:38Z

Actually no. This is not a bug and the behavior is correct and expected.

a/__init__.py
a/module.py
a/b/__init__.py
a/b/module.py
b/__init__.py
b/module.py

Please observe here that modules are a, a.module, a.b, a.b.module, b and b.module. If you don't want a, a.b or b to be modules, they need to be namespaces, i.e. __init__.py needs to be removed.

arcivanov · 2021-10-08T21:11:44Z

Let me see what the actual pylint story is on package modules vs modules.

arcivanov · 2021-10-08T21:44:25Z

Nevermind. It is a bug. I'll fix it.

PyLint is renovated, unit and integration tests are added fixes pybuilder#818

oleg-babintsev · 2021-10-09T07:33:01Z

Sorry if my explanation of problem was unclear and confusing.
Thanks for the fix.

arcivanov · 2021-10-09T08:41:19Z

Once it deploys please let me know if the solution works for you. There were additional output changes.

arcivanov · 2021-10-09T21:00:01Z

@oleg-babintsev please test the changes and let me know if they work for you.

oleg-babintsev · 2021-10-10T05:02:29Z

@arcivanov Yes, sure. I can test it on Monday and will definitely report the result

arcivanov · 2021-10-10T05:03:59Z

@oleg-babintsev Thanks! You'll need to install with pip install -U --pre

oleg-babintsev · 2021-10-11T07:33:25Z

@arcivanov I tested it and now it works properly for previous cases.
But I found another disadvatage.
Now pybuilder passes to pylint the list of python-packages only, but pylint doesn't work with packages which contain folder without __init__.py file in their path, example:

/
└── a/
    └── b/
        ├── __init__.py
        └── module.py

And we execute pylint with argument 'a.b' the following error occurs:

$ pylint 'a.b'
Traceback (most recent call last):
  File "C:\Work\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Work\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Work\Python\Python39\Scripts\pylint.exe\__main__.py", line 7, in <module>
  File "C:\Work\Python\Python39\lib\site-packages\pylint\__init__.py", line 24, in run_pylint
    PylintRun(sys.argv[1:])
  File "C:\Work\Python\Python39\lib\site-packages\pylint\lint\run.py", line 374, in __init__
    linter.check(args)
  File "C:\Work\Python\Python39\lib\site-packages\pylint\lint\pylinter.py", line 970, in check
    self._check_files(
  File "C:\Work\Python\Python39\lib\site-packages\pylint\lint\pylinter.py", line 1006, in _check_files
    for file in file_descrs:
  File "C:\Work\Python\Python39\lib\site-packages\pylint\lint\pylinter.py", line 1073, in _iterate_file_descrs
    for descr in self._expand_files(files_or_modules):
  File "C:\Work\Python\Python39\lib\site-packages\pylint\lint\pylinter.py", line 1080, in _expand_files
    result, errors = expand_modules(
  File "C:\Work\Python\Python39\lib\site-packages\pylint\lint\expand_modules.py", line 129, in expand_modules
    modpath = _modpath_from_file(
  File "C:\Work\Python\Python39\lib\site-packages\pylint\lint\expand_modules.py", line 14, in _modpath_from_file
    return modutils.modpath_from_file_with_callback(
  File "C:\Work\Python\Python39\lib\site-packages\astroid\modutils.py", line 312, in modpath_from_file_with_callback
    raise ImportError(
ImportError: Unable to find module for C:\Work\Projects\misc\pybuilder-pylint-issue\src\main\python\a\b\module.py in C:\Work\Projects\misc\pybuilder-pylint-issue\src\main\python

I'm not talking about is such package correct or not, I'm only describe current behaviour of pylint.
I see that there are 3 variants of usage pylint:

pass to it the list of packages (has disadvantage above)
pass to it the list of all modules includes the __init__.py files
pass to it the asterisk symbol - that works properly too

I think that variants 2 or 3 more flexible for support any structure of project.

arcivanov · 2021-10-11T09:07:32Z

This seems like a typo, no: __init_.py missing an "_" in your example?

oleg-babintsev · 2021-10-11T09:09:08Z

@arcivanov Yes, its typo in message, not in real test.
In additionally I tested pylint on real project and found that pylint correctly worked (and returns the similar result) only in cases 2 and 3.
If I execute pylint with a the list of packages (all packages has __init__.py) then I get an unexpected result with many code duplications warning - it looks like it's a pylint issue (looks like it analyze same modules more than once).

arcivanov · 2021-10-11T12:35:13Z

So would you prefer we rather switch to files (option 2)?
Asterisks seem to be dangerous for a variety of reasons.

oleg-babintsev · 2021-10-11T14:36:31Z

I think yes, because that variant works properly right now.
Or more flexible variant - do configurable option for pylint plugin and developer can choose acceptable variant for him.

Also switch to ExecuteCommand fixes pybuilder#818

arcivanov · 2021-10-13T21:30:04Z

@oleg-babintsev please try again with --pre. I have added ITs to cover implicit namespaces as well.

arcivanov · 2021-10-14T06:42:37Z

@oleg-babintsev could you please check the latest version ASAP? I'd like to release .3.

oleg-babintsev · 2021-10-14T06:57:33Z

@arcivanov Yes, give me 5 minutes if you can

oleg-babintsev · 2021-10-14T07:14:08Z

@arcivanov Yes, its works.
Minor disadvantage of usage modules with absolute path to them it is a full path to module in report:

************* Module a.module
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\a\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\a\module.py:1:0: W0611: Unused import sys (unused-import)
************* Module a.b.module
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\a\b\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\a\b\module.py:1:0: W0611: Unused import sys (unused-import)
************* Module b.module
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\b\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\b\module.py:5:0: C0103: Function name "b" doesn't conform to snake_case naming style (invalid-name)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\b\module.py:1:0: W0611: Unused import sys (unused-import)
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\b\module.py:2:0: W0611: Unused module imported from a as a (unused-import)
************* Module d.module
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\c\d\module.py:1:0: C0114: Missing module docstring (missing-module-docstring)
************* Module d.__init__
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\c\d\__init__.py:2:0: C0304: Final newline missing (missing-final-newline)
************* Module d
c:\work\projects\misc\pybuilder-pylint-issue\src\main\python\c\d\__init__.py:1:0: C0114: Missing module docstring (missing-module-docstring)

-----------------------------------

Your code has been rated at 0.00/10

But the main goal (properly code validation) achived.
Thanks for your work.

arcivanov · 2021-10-14T07:19:37Z

Glad it works! You're welcome, thanks for helping to sort this out!

arcivanov added bug and removed bug labels Oct 8, 2021

arcivanov added the bug label Oct 8, 2021

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Oct 9, 2021

PyLint needs either packages or individual modules, but not both

a28fa26

PyLint is renovated, unit and integration tests are added fixes pybuilder#818

arcivanov mentioned this issue Oct 9, 2021

PyLint needs either packages or individual modules, but not both #820

Merged

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Oct 9, 2021

PyLint needs either packages or individual modules, but not both

8abb88f

PyLint is renovated, unit and integration tests are added fixes pybuilder#818

arcivanov closed this as completed in #820 Oct 9, 2021

arcivanov mentioned this issue Oct 11, 2021

ImportError when using PEP 420 namespaces pylint-dev/pylint#1667

Closed

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Oct 12, 2021

Workaround pylint-dev/pylint#1667, pylint-dev/pylint#2763

1fc6b48

Also switch to ExecuteCommand fixes pybuilder#818

arcivanov mentioned this issue Oct 12, 2021

Workaround PyCQA/pylint#1667, PyCQA/pylint#2763 #830

Merged

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Oct 13, 2021

Workaround pylint-dev/pylint#1667, pylint-dev/pylint#2763

c4ea7b1

Also switch to ExecuteCommand fixes pybuilder#818

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mix packages and modules in command line parameters for Pylint #818

Mix packages and modules in command line parameters for Pylint #818

oleg-babintsev commented Oct 8, 2021 •

edited

Loading

arcivanov commented Oct 8, 2021

arcivanov commented Oct 8, 2021

arcivanov commented Oct 8, 2021

arcivanov commented Oct 8, 2021

arcivanov commented Oct 8, 2021

oleg-babintsev commented Oct 9, 2021

arcivanov commented Oct 9, 2021

arcivanov commented Oct 9, 2021

oleg-babintsev commented Oct 10, 2021

arcivanov commented Oct 10, 2021

oleg-babintsev commented Oct 11, 2021 •

edited

Loading

arcivanov commented Oct 11, 2021

oleg-babintsev commented Oct 11, 2021 •

edited

Loading

arcivanov commented Oct 11, 2021

oleg-babintsev commented Oct 11, 2021 •

edited

Loading

arcivanov commented Oct 13, 2021

arcivanov commented Oct 14, 2021

oleg-babintsev commented Oct 14, 2021

oleg-babintsev commented Oct 14, 2021

arcivanov commented Oct 14, 2021

Mix packages and modules in command line parameters for Pylint #818

Mix packages and modules in command line parameters for Pylint #818

Comments

oleg-babintsev commented Oct 8, 2021 • edited Loading

arcivanov commented Oct 8, 2021

arcivanov commented Oct 8, 2021

arcivanov commented Oct 8, 2021

arcivanov commented Oct 8, 2021

arcivanov commented Oct 8, 2021

oleg-babintsev commented Oct 9, 2021

arcivanov commented Oct 9, 2021

arcivanov commented Oct 9, 2021

oleg-babintsev commented Oct 10, 2021

arcivanov commented Oct 10, 2021

oleg-babintsev commented Oct 11, 2021 • edited Loading

arcivanov commented Oct 11, 2021

oleg-babintsev commented Oct 11, 2021 • edited Loading

arcivanov commented Oct 11, 2021

oleg-babintsev commented Oct 11, 2021 • edited Loading

arcivanov commented Oct 13, 2021

arcivanov commented Oct 14, 2021

oleg-babintsev commented Oct 14, 2021

oleg-babintsev commented Oct 14, 2021

arcivanov commented Oct 14, 2021

oleg-babintsev commented Oct 8, 2021 •

edited

Loading

oleg-babintsev commented Oct 11, 2021 •

edited

Loading

oleg-babintsev commented Oct 11, 2021 •

edited

Loading

oleg-babintsev commented Oct 11, 2021 •

edited

Loading