Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrading from pylint 2.14.5 to 2.15.0 causes django.core.exceptions.ImproperlyConfigured to be raised #370

Closed
simensol opened this issue Aug 31, 2022 · 15 comments
Labels
Upstream Bug 🪲 The problem happens in pylint, not pylint-django

Comments

@simensol
Copy link

simensol commented Aug 31, 2022

After upgrading pylint from 2.14.5 to 2.15.0, running pylint **/*.py now raises django.core.exceptions.ImproperlyConfigured. If I downgrade to pylint 2.14.5, the issue disappears.

With pylint 2.15.0:

$ pylint --version
pylint 2.15.0
astroid 2.12.5
Python 3.10.6 (main, Aug  3 2022, 10:13:24) [GCC 10.2.1 20210110]

$ pylint **/*.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pylint_django/checkers/foreign_key_strings.py", line 92, in open
    django.setup()
  File "/usr/local/lib/python3.10/site-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/usr/local/lib/python3.10/site-packages/django/conf/__init__.py", line 92, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.10/site-packages/django/conf/__init__.py", line 72, in _setup
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: Requested setting LOGGING_CONFIG, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pylint_django/checkers/foreign_key_strings.py", line 120, in open
    settings.configure(Settings(self.config.django_settings_module))
  File "/usr/local/lib/python3.10/site-packages/django/conf/__init__.py", line 190, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
  File "/usr/local/lib/python3.10/site-packages/pylint/__init__.py", line 35, in run_pylint
    PylintRun(argv or sys.argv[1:])
  File "/usr/local/lib/python3.10/site-packages/pylint/lint/run.py", line 207, in __init__
    linter.check(args)
  File "/usr/local/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 672, in check
    with self._astroid_module_checker() as check_astroid_module:
  File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 962, in _astroid_module_checker
    checker.open()
  File "/usr/local/lib/python3.10/site-packages/pylint_django/checkers/foreign_key_strings.py", line 125, in open
    self.add_message(
  File "/usr/local/lib/python3.10/site-packages/pylint/checkers/base_checker.py", line 164, in add_message
    self.linter.add_message(
  File "/usr/local/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 1284, in add_message
    self._add_one_message(
  File "/usr/local/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 1217, in _add_one_message
    self.stats.increase_single_module_message_count(
  File "/usr/local/lib/python3.10/site-packages/pylint/utils/linterstats.py", line 309, in increase_single_module_message_count
    self.by_module[modname][type_name] += increase
KeyError: 'Command line or configuration file'

With pylint 2.14.5:

$ pylint --version
pylint 2.14.5
astroid 2.11.7
Python 3.10.6 (main, Aug  3 2022, 10:13:24) [GCC 10.2.1 20210110]

$ pylint **/*.py

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

The Django settings module is provided through pyproject.toml (see below). If I provide the Django settings module through environmental variables, a somewhat different exception is raised with pylint 2.15.0:

$ pylint --version
pylint 2.15.0
astroid 2.12.5
Python 3.10.6 (main, Aug  3 2022, 10:13:24) [GCC 10.2.1 20210110]

$ export DJANGO_SETTINGS_MODULE="config.settings"

$ pylint **/*.py
Traceback (most recent call last):
  File "/usr/local/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
  File "/usr/local/lib/python3.10/site-packages/pylint/__init__.py", line 35, in run_pylint
    PylintRun(argv or sys.argv[1:])
  File "/usr/local/lib/python3.10/site-packages/pylint/lint/run.py", line 207, in __init__
    linter.check(args)
  File "/usr/local/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 672, in check
    with self._astroid_module_checker() as check_astroid_module:
  File "/usr/local/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.10/site-packages/pylint/lint/pylinter.py", line 962, in _astroid_module_checker
    checker.open()
  File "/usr/local/lib/python3.10/site-packages/pylint_django/checkers/foreign_key_strings.py", line 92, in open
    django.setup()
  File "/usr/local/lib/python3.10/site-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/usr/local/lib/python3.10/site-packages/django/conf/__init__.py", line 92, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.10/site-packages/django/conf/__init__.py", line 79, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.10/site-packages/django/conf/__init__.py", line 190, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'config'

This exception disappears with pylint 2.14.5:

$ pylint --version
pylint 2.14.5
astroid 2.11.7
Python 3.10.6 (main, Aug  3 2022, 10:13:24) [GCC 10.2.1 20210110]

$ export DJANGO_SETTINGS_MODULE="config.settings"

$ pylint **/*.py

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

Output from pip freeze:

asgiref==3.5.2
astroid==2.11.7
attrs==22.1.0
Babel==2.10.3
black==22.6.0
certifi==2022.6.15
charset-normalizer==2.1.1
click==8.1.3
commonmark==0.9.1
coverage==6.4.4
defusedxml==0.7.1
dill==0.3.5.1
distlib==0.3.6
dj-database-url==1.0.0
dj-email-url==1.0.5
Django==4.1
django-cache-url==3.4.2
django-extensions==3.2.0
django-fake-model==0.1.4
django-money==2.1.1
django-stubs-ext==0.5.0
django-types==0.16.0
djangorestframework==3.13.1
djangorestframework-types==0.8.0
environs==9.5.0
execnet==1.9.0
filelock==3.8.0
get-docker-secret==1.0.2
httpie==3.2.1
idna==3.3
importlib-resources==5.9.0
inclusive-django-range-fields==0.2.3
iniconfig==1.1.1
isort==5.10.1
lazy-object-proxy==1.7.1
lorem-text==2.1
marshmallow==3.17.1
mccabe==0.7.0
multidict==6.0.2
mypy-extensions==0.4.3
packaging==21.3
pathspec==0.10.0
pipenv==2022.8.24
platformdirs==2.5.2
pluggy==1.0.0
psycopg2==2.9.3
py==1.11.0
py-moneyed==1.2
Pygments==2.13.0
pylint-django==2.5.3
pylint-plugin-utils==0.7
pyparsing==3.0.9
PySocks==1.7.1
pytest==7.1.2
pytest-cov==3.0.0
pytest-django==4.5.2
pytest-forked==1.4.0
pytest-xdist==2.5.0
python-dotenv==0.20.0
pytz==2022.2.1
PyYAML==6.0
requests==2.28.1
requests-toolbelt==0.9.1
rich==12.5.1
sqlparse==0.4.2
StrEnum==0.4.8
tomli==2.0.1
tomlkit==0.11.4
types-psycopg2==2.9.21
typing_extensions==4.3.0
uritemplate==4.1.1
urllib3==1.26.12
virtualenv==20.16.4
virtualenv-clone==0.5.7
wrapt==1.14.1
yachalk==0.1.5

+ pylint==2.14.5 or 2.15.0
+ two editable local python packages

pylint config in pyproject.toml:

[tool.pylint.master]
ignore-patterns = ["manage.py", "migrations/"]
load-plugins = [
    "pylint.extensions.docparams",
    "pylint_django",
    "pylint_django.checkers.migrations",
    "local_pylint_plugin",
]

[tool.pylint.miscellaneous]
notes = ["BUG"]

[tool.pylint.parameter_documentation]
accept-no-raise-doc = "no"

[tool.pylint.similarities]
min-similarity-lines = 10

[tool.pylint.basic]
no-docstring-rgx = "([a-zA-Z]+Inline)|([a-zA-Z]+Admin)|([a-zA-Z]+Config)|([a-zA-Z]+Config)"
class-const-rgx = "([A-Z]{1}[a-zA-Z_]+)"

[tool.pylint."DJANGO FOREIGN KEYS REFERENCED BY STRINGS"]
django-settings-module = "config.settings"

[tool.pylint."MESSAGES CONTROL"]
disable = [
    "too-few-public-methods",
    "wrong-import-order",
    "unsubscriptable-object",
]
@DayDotMe
Copy link

Same here, I upgraded to 2.15.0 to get rid of unsupported-binary-operation warning in DRF ViewSet permission_classes but it's not compatible.

As a quick fix, I'm running pylint through code:

from pylint.lint import Run
Run(["my_module_name"])

@andylamp
Copy link

I concur, we encounter the same behavior but we load the django settings file through setup.cfg, if that makes any difference...

enku added a commit to enku/gentoo-build-publisher that referenced this issue Sep 5, 2022
There is a bug in pylint-django that causes it to fail with
pylint-2.15.0.  Block it for now.

pylint-dev/pylint-django#370
@pmyjavec
Copy link

pmyjavec commented Sep 6, 2022

For some reason, my environment is missing present working directory in the sys.path so the import doesn't work.

Adding the following, checkers/foreign_key_strings.py#L119, solved my issues.

import sys
sys.path.append('.')

For some reason when I was debugging around that point I noticed the python path didn't include the present when the foreign_key_strings check occurs.

I noticed (but haven't proved) that https://github.com/PyCQA/pylint/blob/main/pylint/__init__.py#L80 has some code which removes things from the sys.path, might it be related, but might not? Someone who knows both code bases has a clue I'm sure.

@ramwin
Copy link

ramwin commented Sep 7, 2022

I met the same problem.
I didn't set the DJANGO_SETTINGS_MODULE, I just ran

[#45#wangx@localhost:testsystembackend (master)] $ pylint --load-plugins pylint_django --django-settings-module=testsystembackend.settings.local   tmss/models.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pylint_django/checkers/foreign_key_strings.py", line 92, in open
    django.setup()
  File "/usr/local/lib/python3.8/dist-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/usr/local/lib/python3.8/dist-packages/django/conf/__init__.py", line 92, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.8/dist-packages/django/conf/__init__.py", line 72, in _setup
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: Requested setting LOGGING_CONFIG, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pylint_django/checkers/foreign_key_strings.py", line 120, in open
    settings.configure(Settings(self.config.django_settings_module))
  File "/usr/local/lib/python3.8/dist-packages/django/conf/__init__.py", line 190, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'testsystembackend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
  File "/usr/local/lib/python3.8/dist-packages/pylint/__init__.py", line 35, in run_pylint
    PylintRun(argv or sys.argv[1:])
  File "/usr/local/lib/python3.8/dist-packages/pylint/lint/run.py", line 207, in __init__
    linter.check(args)
  File "/usr/local/lib/python3.8/dist-packages/pylint/lint/pylinter.py", line 672, in check
    with self._astroid_module_checker() as check_astroid_module:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.8/dist-packages/pylint/lint/pylinter.py", line 962, in _astroid_module_checker
    checker.open()
  File "/usr/local/lib/python3.8/dist-packages/pylint_django/checkers/foreign_key_strings.py", line 125, in open
    self.add_message(
  File "/usr/local/lib/python3.8/dist-packages/pylint/checkers/base_checker.py", line 164, in add_message
    self.linter.add_message(
  File "/usr/local/lib/python3.8/dist-packages/pylint/lint/pylinter.py", line 1284, in add_message
    self._add_one_message(
  File "/usr/local/lib/python3.8/dist-packages/pylint/lint/pylinter.py", line 1217, in _add_one_message
    self.stats.increase_single_module_message_count(
  File "/usr/local/lib/python3.8/dist-packages/pylint/utils/linterstats.py", line 309, in increase_single_module_message_count
    self.by_module[modname][type_name] += increase
KeyError: 'Command line or configuration file'

It seems that some thing is wrong with LOGGING config. Here is my logging CONFIG:

DEFAULT_HANDLERS = ['debug_file', 'info_file',
                    'warning_file', 'error_file']
if DEBUG:
    DEFAULT_HANDLERS.append("console")
    DEFAULT_HANDLERS.remove("info_file")
DEFAULT_LOGGER = {
    "handlers": DEFAULT_HANDLERS,
    "level": "INFO",
}

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': ('[%(levelname)5s] %(asctime)s %(pathname)s '
                       '%(funcName)s (line: %(lineno)d)'
                       '    %(message)s'),
        },
        'simple': {
            'format': '[%(levelname)s] %(message)s ',
        },
    },
    'handlers': {
        'error_file': {
            'level': "ERROR",
            'class': 'logging.FileHandler',
            'filename': os.path.join(BASE_DIR, 'log/error.log'),
            'formatter': 'verbose',
        },
        'warning_file': {
            'level': "WARNING",
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': os.path.join(BASE_DIR, 'log/warning.log'),
            'maxBytes': 1024 * 1024 * 10,
            'backupCount': 20,
            'formatter': 'verbose',
        },
        'info_file': {
            'level': "INFO",
            'class': 'logging.handlers.RotatingFileHandler',
            'maxBytes': 1024 * 1024 * 10,
            'backupCount': 20,
            'filename': os.path.join(BASE_DIR, 'log/info.log'),
            'formatter': 'verbose',
        },
        'debug_file': {
            'level': "DEBUG",
            'class': 'logging.handlers.RotatingFileHandler',
            'maxBytes': 1024 * 1024 * 10,
            'backupCount': 20,
            'filename': os.path.join(BASE_DIR, 'log/debug.log'),
            'formatter': 'verbose',
        },
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'verbose',
        },
    },
    'loggers': {
        'core': DEFAULT_LOGGER,
        'account': DEFAULT_LOGGER,
        'default': DEFAULT_LOGGER,
        'django': DEFAULT_LOGGER,
        'testapp': DEFAULT_LOGGER,
        'otherapp': DEFAULT_LOGGER,
    },
}

@emartinm
Copy link

I experience the same problem: I can use pylint in version 2.14.5 but as soon I update it to a newer version (2.15.X) y obtain the same ImproperlyConfigured exception. In my case I configure the django-settings-module using a .pylintrc config file.

@jgb
Copy link

jgb commented Sep 27, 2022

I'm encountering the same problem. Downgrading from 2.15.x to 2.14.5 fixes the issue.

@trent-abc
Copy link

Hi, any updates on this it's breaking our build (Yes we can downgrade but we'd like to... not).

@Pierre-Sassoulas
Copy link
Member

It look like pylint_django isn't able to add a message regarding the configuration file because the entry for 'Command line or configuration file' is missing in the stats. I don't know about the resolution that need to be done in pylint_django but to fix your issue locally I would suggest to remove pylint django temporarily from the load-plugins option, and launch pylint without it. Then fix the message related to the configuration file or the command line used and re-add pylint_django. This does not fix the base issue but it look like it could work.

@emartinm
Copy link

Thanks for the suggestion, @Pierre-Sassoulas . Unfortunately it doesn't work in my project. I've commented the load-plugins and django-settings-module from the .pylintrc:

# load-plugins=pylint_django
# django-settings-module=...

With this new .pylintrc file and after upgrading pylint, it does not throw any KeyError: 'Command line or configuration file' exception (so I cannot fix anything) but incorrectly detects many pylint errors like has no 'objects' member (no-member) for every model use.

@Pierre-Sassoulas
Copy link
Member

Are you sure there isn't any configuration related error (bad-option-value or useless-option-value in particular) raised between the no-member ?

@emartinm
Copy link

No, I've tried again with pylint 2.15.4 and all the errors I obtain are because Pylint cannot understand the particularities of Django code when removing the plugin pylint_django:

  • too-few-public-methods (in model classes)
  • no-member (in model classes)

In particular, I don't get any bad-option-value or useless-option-value error message.

@emartinm
Copy link

emartinm commented Nov 7, 2022

I've found a workaround in the related issue #325 (comment): set the PYTHONPATH to the current directory before invoking pylint (with pylint_django enabled) .

export PYTHONPATH=$PWD
pylint --ignore=apps.py,migrations ...

@craiga
Copy link

craiga commented Dec 20, 2022

Looks like this is fixed in pylint-dev/pylint#7940, which is part of pylint 2.15.9.

@emartinm
Copy link

Thank you very much for the information, @craiga . I can confirm that pylint-django is working again (removing the workaround mentioned in #370 (comment)) with the following versions:

pylint==2.15.9
pylint-django==2.5.3

As the ImproperlyConfigured exception is no longer raised, I think that this issue could be closed.

@Pierre-Sassoulas Pierre-Sassoulas added the Upstream Bug 🪲 The problem happens in pylint, not pylint-django label Dec 20, 2022
@Pierre-Sassoulas
Copy link
Member

Thanks for the confirmation @emartinm !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Upstream Bug 🪲 The problem happens in pylint, not pylint-django
Projects
None yet
Development

No branches or pull requests

10 participants