Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pylint is crashing with astroid.exceptions.StatementMissing: Statement not found on <Module.builtins l.0 #8079

Open
jooola opened this issue Jan 18, 2023 · 8 comments
Open

Pylint is crashing with astroid.exceptions.StatementMissing: Statement not found on <Module.builtins l.0 #8079

jooola opened this issue Jan 18, 2023 · 8 comments
Labels
Needs decision 🔒 Needs a decision before implemention or rejection
Milestone
4.0.0

Comments

@jooola
Copy link

jooola commented Jan 18, 2023 •
edited

Bug description

Pylint is crashing with:

Exception on node <FunctionDef.license l.155 at 0x7fb37394e940> in file '....'
// ...
astroid.exceptions.StatementMissing: Statement not found on <Module.builtins l.0 at 0x7f97337e4b20>

The line of code that is crashing pylint is:

https://dev.funkwhale.audio/jooola/funkwhale/-/blob/pylint_bug_licence/api/funkwhale_api/music/factories.py#L155

When renaming licence to license_ pylint doesn't crash.

You can clone the branch and run pylint:

git clone -b pylint_bug_licence https://dev.funkwhale.audio/jooola/funkwhale
cd funkwhale/api
poetry install
poetry run pylint \
    --jobs=4 \
    --output-format=colorized \
    --recursive=true \
    funkwhale_api/music/factories.py

Configuration

[tool.pylint.master]
load-plugins = ["pylint_django"]
django-settings-module = "config.settings.testing"

[tool.pylint.messages_control]
disable = [
  "invalid-name",
  "missing-class-docstring",
  "missing-function-docstring",
  "missing-module-docstring",
]

[tool.pylint.design]
max-parents = 13

[tool.pylint.format]
max-line-length = 120

Command used

poetry run pylint \
    --jobs=4 \
    --output-format=colorized \
    --recursive=true \
    funkwhale_api/music/factories.py

Pylint output

2023-01-18 21:09:12,028 funkwhale_api.config INFO     Running with the following plugins enabled: funkwhale_api.contrib.scrobbler, funkwhale_api.contrib.listenbrainz, funkwhale_api.contrib.maloja
Exception on node <FunctionDef.license l.155 at 0x7fb37394e940> in file '/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/funkwhale_api/music/factories.py'
Traceback (most recent call last):
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/utils/ast_walker.py", line 90, in walk
    callback(astroid)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint_plugin_utils/__init__.py", line 60, in __call__
    self.augmentation_func(Chain(self.old_method, node), node)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint_plugin_utils/__init__.py", line 119, in __call__
    chain()
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint_plugin_utils/__init__.py", line 69, in __call__
    self.old_method(self.node)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/checkers/classes/class_checker.py", line 1241, in visit_functiondef
    for obj in ancestor.lookup(node.name)[1]:
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/node_classes.py", line 367, in lookup
    return self.scope().scope_lookup(self, name)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 2340, in scope_lookup
    return frame._scope_lookup(node, name, offset)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/scoped_nodes/mixin.py", line 86, in _scope_lookup
    return pscope.scope_lookup(node, name)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 380, in scope_lookup
    return self._scope_lookup(node, name, offset)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/scoped_nodes/mixin.py", line 75, in _scope_lookup
    stmts = _filter_stmts(node, self.locals[name], self, offset)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/filter_statements.py", line 105, in _filter_stmts
    statements = _get_filtered_node_statements(base_node, stmts)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/filter_statements.py", line 17, in _get_filtered_node_statements
    statements = [(node, node.statement(future=True)) for node in stmt_nodes]
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/filter_statements.py", line 17, in <listcomp>
    statements = [(node, node.statement(future=True)) for node in stmt_nodes]
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/node_ng.py", line 324, in statement
    return self.parent.statement(future=future)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 470, in statement
    raise StatementMissing(target=self)
astroid.exceptions.StatementMissing: Statement not found on <Module.builtins l.0 at 0x7fb37c5bfb20>
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/lint/pylinter.py", line 831, in _check_file
    check_astroid_module(ast_node)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/lint/pylinter.py", line 1060, in check_astroid_module
    retval = self._check_astroid_module(
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/lint/pylinter.py", line 1110, in _check_astroid_module
    walker.walk(node)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/utils/ast_walker.py", line 93, in walk
    self.walk(child)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/utils/ast_walker.py", line 93, in walk
    self.walk(child)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/utils/ast_walker.py", line 90, in walk
    callback(astroid)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint_plugin_utils/__init__.py", line 60, in __call__
    self.augmentation_func(Chain(self.old_method, node), node)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint_plugin_utils/__init__.py", line 119, in __call__
    chain()
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint_plugin_utils/__init__.py", line 69, in __call__
    self.old_method(self.node)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/checkers/classes/class_checker.py", line 1241, in visit_functiondef
    for obj in ancestor.lookup(node.name)[1]:
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/node_classes.py", line 367, in lookup
    return self.scope().scope_lookup(self, name)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 2340, in scope_lookup
    return frame._scope_lookup(node, name, offset)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/scoped_nodes/mixin.py", line 86, in _scope_lookup
    return pscope.scope_lookup(node, name)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 380, in scope_lookup
    return self._scope_lookup(node, name, offset)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/scoped_nodes/mixin.py", line 75, in _scope_lookup
    stmts = _filter_stmts(node, self.locals[name], self, offset)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/filter_statements.py", line 105, in _filter_stmts
    statements = _get_filtered_node_statements(base_node, stmts)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/filter_statements.py", line 17, in _get_filtered_node_statements
    statements = [(node, node.statement(future=True)) for node in stmt_nodes]
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/filter_statements.py", line 17, in <listcomp>
    statements = [(node, node.statement(future=True)) for node in stmt_nodes]
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/node_ng.py", line 324, in statement
    return self.parent.statement(future=future)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 470, in statement
    raise StatementMissing(target=self)
astroid.exceptions.StatementMissing: Statement not found on <Module.builtins l.0 at 0x7fb37c5bfb20>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/lint/parallel.py", line 72, in _worker_check_single_file
    _worker_linter.check_single_file_item(file_item)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/lint/pylinter.py", line 743, in check_single_file_item
    self._check_file(self.get_ast, check_astroid_module, file)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/lint/pylinter.py", line 833, in _check_file
    raise astroid.AstroidError from e
astroid.exceptions.AstroidError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/__init__.py", line 35, in run_pylint
    PylintRun(argv or sys.argv[1:])
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/lint/run.py", line 207, in __init__
    linter.check(args)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/lint/pylinter.py", line 674, in check
    check_parallel(
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/lint/parallel.py", line 152, in check_parallel
    for (
  File "/usr/lib/python3.9/multiprocessing/pool.py", line 870, in next
    raise value
astroid.exceptions.AstroidError

Expected behavior

Pylint should print a linting error, instead of crashing.

Pylint version

$ poetry run pylint --version
pylint 2.15.6
astroid 2.13.2
Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110]

OS / Environment

Debian Bullseye

Additional dependencies

$ poetry show pylint
name : pylint
version : 2.15.6
description : python code static checker

dependencies

  • astroid >=2.12.12,<=2.14.0-dev0
  • colorama >=0.4.5
  • dill >=0.2
  • isort >=4.2.5,<6
  • mccabe >=0.6,<0.8
  • platformdirs >=2.2.0
  • tomli >=1.1.0
  • tomlkit >=0.10.1
  • typing-extensions >=3.10.0

required by

  • pylint-django >=2.0,<3
  • pylint-plugin-utils >=1.7

$ poetry show pylint-django
name : pylint-django
version : 2.5.3
description : A Pylint plugin to help Pylint understand the Django web framework

dependencies

  • pylint >=2.0,<3
  • pylint-plugin-utils >=0.7
@jooola jooola added the Needs triage 📥 Just created, needs acknowledgment, triage, and proper labelling label Jan 18, 2023
@jooola jooola changed the title astroid.exceptions.StatementMissing: Statement not found on <Module.builtins l.0 at 0x7f97337e4b20> astroid.exceptions.StatementMissing: Statement not found on <Module.builtins l.0 Jan 18, 2023
@jooola jooola changed the title astroid.exceptions.StatementMissing: Statement not found on <Module.builtins l.0 Pylint is crashing with astroid.exceptions.StatementMissing: Statement not found on <Module.builtins l.0 Jan 18, 2023
@DanielNoord
Copy link
Collaborator

This is probably related to us thinking that every builtin is actually from builtins. This can cause all kind of weird behaviour. I would advise to use license_ anyway as it's inadvisable to overwrite builtin functions anyway.

@jooola
Copy link
Author

jooola commented Jan 18, 2023 •
edited

This is probably related to us thinking that every builtin is actually from builtins. This can cause all kind of weird behaviour. I would advise to use license_ anyway as it's inadvisable to overwrite builtin functions anyway.

I agree that we shouldn't overwrite builtin functions, I was just hoping to prevent a crash.

I am unsure if I can rename a field so easily without breaking some API. So the possibility to ignore the normally raised linting error until we provide a good fix would be very welcome.

EDIT: I renamed the problematic function, so I have my work around. Feel free to close if you think this is an expected behavior.

@DanielNoord
Copy link
Collaborator

Good to hear you fixed it! This is definitely something we should fix but will probably be low priority as it has minimal impact and can hopefully be averted by small refactors by users.

@Pierre-Sassoulas Pierre-Sassoulas added Needs decision 🔒 Needs a decision before implemention or rejection and removed Needs triage 📥 Just created, needs acknowledgment, triage, and proper labelling labels Jan 18, 2023
@Pierre-Sassoulas
Copy link
Member

I'm not sure we should fix, checking that a bultin is indeed a builtin is going to cost an enormous amount of computing power over all pylint users for this one use case. I think we should consider modified builtins an unsupported broken environment.

@DanielNoord
Copy link
Collaborator

It feels though as if we are incorrectly populating the cache with builtins. I don't know, something to investigate I guess!

@jacobtylerwalls
Copy link
Member

Is the problem only reproducible in when using --jobs? We may not be catching AstroidError as robustly as in the single-process code, in which case I'd flag this as topic-multiprocessing.

@jooola
Copy link
Author

jooola commented Jan 18, 2023 •
edited

Running without jobs allows pylint to finish and report some more expressive error message, it feels less like a crash:

$ poetry run pylint \ 
    --output-format=colorized \
    --recursive=true \
    funkwhale_api/music/factories.py
2023-01-18 23:29:03,224 funkwhale_api.config INFO     Running with the following plugins enabled: funkwhale_api.contrib.scrobbler, funkwhale_api.contrib.listenbrainz, funkwhale_api.contrib.maloja
************* Module funkwhale_api.music.factories
funkwhale_api/music/factories.py:70:4: R0903: Too few public methods (0/2) (too-few-public-methods)
funkwhale_api/music/factories.py:95:4: R0903: Too few public methods (0/2) (too-few-public-methods)
funkwhale_api/music/factories.py:122:4: R0903: Too few public methods (0/2) (too-few-public-methods)
funkwhale_api/music/factories.py:135:4: E0202: An attribute defined in funkwhale_api.music.factories line 143 hides this method (method-hidden)
Exception on node <FunctionDef.license l.155 at 0x7fe1f0e800a0> in file '/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/funkwhale_api/music/factories.py'
Traceback (most recent call last):
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/utils/ast_walker.py", line 90, in walk
    callback(astroid)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint_plugin_utils/__init__.py", line 60, in __call__
    self.augmentation_func(Chain(self.old_method, node), node)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint_plugin_utils/__init__.py", line 119, in __call__
    chain()
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint_plugin_utils/__init__.py", line 69, in __call__
    self.old_method(self.node)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/pylint/checkers/classes/class_checker.py", line 1241, in visit_functiondef
    for obj in ancestor.lookup(node.name)[1]:
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/node_classes.py", line 367, in lookup
    return self.scope().scope_lookup(self, name)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 2340, in scope_lookup
    return frame._scope_lookup(node, name, offset)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/scoped_nodes/mixin.py", line 86, in _scope_lookup
    return pscope.scope_lookup(node, name)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 380, in scope_lookup
    return self._scope_lookup(node, name, offset)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/scoped_nodes/mixin.py", line 75, in _scope_lookup
    stmts = _filter_stmts(node, self.locals[name], self, offset)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/filter_statements.py", line 105, in _filter_stmts
    statements = _get_filtered_node_statements(base_node, stmts)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/filter_statements.py", line 17, in _get_filtered_node_statements
    statements = [(node, node.statement(future=True)) for node in stmt_nodes]
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/filter_statements.py", line 17, in <listcomp>
    statements = [(node, node.statement(future=True)) for node in stmt_nodes]
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/node_ng.py", line 324, in statement
    return self.parent.statement(future=future)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/astroid/nodes/scoped_nodes/scoped_nodes.py", line 470, in statement
    raise StatementMissing(target=self)
astroid.exceptions.StatementMissing: Statement not found on <Module.builtins l.0 at 0x7fe1f9ac6af0>
funkwhale_api/music/factories.py:1:0: F0002: funkwhale_api/music/factories.py: Fatal error while checking 'funkwhale_api/music/factories.py'. Please open an issue in our bug tracker so we address this. There is a pre-filled template that you can use in '/home/jo/.cache/pylint/pylint-crash-2023-01-18-22-29-04.txt'. (astroid-error)

------------------------------------------------------------------
Your code has been rated at 0.00/10 (previous run: 0.00/10, +0.00)

@jacobtylerwalls
Copy link
Member

Aha, so we have two problems! The underlying issue and then a topic-multiprocessing problem that --jobs prevents catching AstroidError. Can you open a second issue?

@Pierre-Sassoulas Pierre-Sassoulas added this to the 4.0.0 milestone Sep 25, 2023
@John-Robbins John-Robbins mentioned this issue Apr 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Needs decision 🔒 Needs a decision before implemention or rejection
Projects
None yet
Milestone
4.0.0
Development

No branches or pull requests
4 participants
@Pierre-Sassoulas @DanielNoord @jooola @jacobtylerwalls