Error: Apps aren't loaded yet when user model imported in authentication class

[mdefeche]

Hello,

I'm having the following issue after having installed your app. Django doesn't start anymore.

django_1       | /opt/project/django/safir/settings.py changed, reloading.
django_1       | Watching for file changes with StatReloader
django_1       | Exception in thread django-main-thread:
django_1       | Traceback (most recent call last):
django_1       |   File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
django_1       |     self.run()
django_1       |   File "/usr/local/lib/python3.8/threading.py", line 870, in run
django_1       |     self._target(*self._args, **self._kwargs)
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/utils/autoreload.py", line 64, in wrapper
django_1       |     fn(*args, **kwargs)
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 110, in inner_run
django_1       |     autoreload.raise_last_exception()
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/utils/autoreload.py", line 87, in raise_last_exception
django_1       |     raise _exception[1]
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
django_1       |     autoreload.check_errors(django.setup)()
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/utils/autoreload.py", line 64, in wrapper
django_1       |     fn(*args, **kwargs)
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
django_1       |     apps.populate(settings.INSTALLED_APPS)
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 91, in populate
django_1       |     app_config = AppConfig.create(entry)
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/apps/config.py", line 224, in create
django_1       |     import_module(entry)
django_1       |   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
django_1       |     return _bootstrap._gcd_import(name[level:], package, level)
django_1       |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
django_1       |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
django_1       |   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
django_1       |   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
django_1       |   File "<frozen importlib._bootstrap_external>", line 843, in exec_module
django_1       |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
django_1       |   File "/usr/local/lib/python3.8/site-packages/drf_standardized_errors/__init__.py", line 3, in <module>
django_1       |     from .formatter import ExceptionFormatter
django_1       |   File "/usr/local/lib/python3.8/site-packages/drf_standardized_errors/formatter.py", line 8, in <module>
django_1       |     from .types import Error, ErrorResponse, ErrorType, ExceptionHandlerContext
django_1       |   File "/usr/local/lib/python3.8/site-packages/drf_standardized_errors/types.py", line 6, in <module>
django_1       |     from rest_framework.views import APIView
django_1       |   File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 17, in <module>
django_1       |     from rest_framework.schemas import DefaultSchema
django_1       |   File "/usr/local/lib/python3.8/site-packages/rest_framework/schemas/__init__.py", line 33, in <module>
django_1       |     authentication_classes=api_settings.DEFAULT_AUTHENTICATION_CLASSES,
django_1       |   File "/usr/local/lib/python3.8/site-packages/rest_framework/settings.py", line 225, in __getattr__
django_1       |     val = perform_import(val, attr)
django_1       |   File "/usr/local/lib/python3.8/site-packages/rest_framework/settings.py", line 168, in perform_import
django_1       |     return [import_from_string(item, setting_name) for item in val]
django_1       |   File "/usr/local/lib/python3.8/site-packages/rest_framework/settings.py", line 168, in <listcomp>
django_1       |     return [import_from_string(item, setting_name) for item in val]
django_1       |   File "/usr/local/lib/python3.8/site-packages/rest_framework/settings.py", line 177, in import_from_string
django_1       |     return import_string(val)
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/utils/module_loading.py", line 17, in import_string
django_1       |     module = import_module(module_path)
django_1       |   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
django_1       |     return _bootstrap._gcd_import(name[level:], package, level)
django_1       |   File "/opt/project/django/safir/authentication.py", line 7, in <module>
django_1       |     from user.models import UserAPIKey
django_1       |   File "/opt/project/django/user/models.py", line 1, in <module>
django_1       |     from django.contrib.auth.models import AbstractUser, BaseUserManager
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/contrib/auth/models.py", line 3, in <module>
django_1       |     from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/contrib/auth/base_user.py", line 48, in <module>
django_1       |     class AbstractBaseUser(models.Model):
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 108, in __new__
django_1       |     app_config = apps.get_containing_app_config(module)
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 253, in get_containing_app_config
django_1       |     self.check_apps_ready()
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/apps/registry.py", line 136, in check_apps_ready
django_1       |     raise AppRegistryNotReady("Apps aren't loaded yet.")
django_1       | django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

After some investigations, it appears that my custom authentication class (extending BaseAuthentication) which imports user model is the root cause. If I remove that authentication class, the application starts ...

I believe this can be reproduced by creating a dummy authentication class which imports the user model, then register it in settings

django_1       |   File "/opt/project/django/user/models.py", line 1, in <module>
django_1       |     from django.contrib.auth.models import AbstractUser, BaseUserManager
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/contrib/auth/models.py", line 3, in <module>
django_1       |     from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/contrib/auth/base_user.py", line 48, in <module>
django_1       |     class AbstractBaseUser(models.Model):
django_1       |   File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 108, in __new__
django_1       |     app_config = apps.get_containing_app_config(module)

I don't have the issues with other apps I mentioned.

Your library looks promising :-)

[mdefeche]

I managed to fix it by retrieving the model in init() via a string like in get_user_model()
self.user_api_model = django_apps.get_model("user.UserAPIKey", require_ready=False)

Still don't understand why it works fine with other apps

[ghazi-git]

Thanks for taking the time to report the issue along with the steps to reproduce it.

A quick workaround for now is to remove drf_standardized_errors from INSTALLED_APPS. Adding drf_standardized_errors to INSTALLED_APPS is more of a formality since the package does not provides anything that can be auto-discovered by django like models, templates, ...

The issue here is that the exception handler and other classes are imported in drf_standardized_errors.__init__. So, when django.setup is called, installed apps (including this package) are loaded. That results in importing the exception handler which imports the APIView from drf, which in turn imports default auth classes and, in your case, the custom auth class requires the user model but django isn't properly setup yet. This is similar to a circular import.

My initial idea is that anything imported in __init__ would eventually be considered as public API. However, with django and how installed apps are loaded to be inspected for auto-discoverable models, templates, ... it's actually a bad idea to do that because it runs into edge cases like this one. When checking the third party apps you mentioned and even others in the django ecosystem, all of them don't import anything in the top-level __init__ file. I even managed to find this commit in django-debug-toolbar that fixed a similar issue.

I will be releasing a new version soon (hopefully this weekend) to remove any import in drf_standardized_errors.__init__ fixing this issue

[mdefeche]

Thank you for your explanation, it makes sense. I'll also keep the patch I made as other authentication classes avoid importing models directly.

Will the new release also contain the pull requests you made for drf-spectacular :-) ?

[ghazi-git]

I just pushed v0.11.0 to fix this issue. You should be able to quickly upgrade to the new version after updating the import paths as mentioned here.

While the main work for auto-generating error responses with drf-spectacular is done, there are quite some tests and docs that I have yet to add and that would take me quite some time to finish, so it's not part of this release :(