refactor: Prepare autodiscover and template loader for v1 #533

JuroOravec · 2024-06-22T08:08:14Z

MR for moving autodiscover to it's own file + documenting it, as mentioned in #527 (comment).

for more information, see https://pre-commit.ci

JuroOravec · 2024-06-22T08:15:12Z

src/django_components/utils.py

-    matched_files: List[Path]
-
-
-def search(search_glob: Optional[str] = None, engine: Optional[Engine] = None) -> SearchResult:


I've split this function to get_dirs and search_dirs and moved them to autodiscover.py

src/django_components/template_loader.py

JuroOravec · 2024-06-22T08:39:26Z

src/django_components/template_loader.py

            if hasattr(settings, "BASE_DIR"):
-                path = (Path(settings.BASE_DIR) / component_dir).resolve()
+                path: Path = (Path(settings.BASE_DIR) / component_dir).resolve()


This branch allows us to treat a relative path in STATICFILES_DIRS as relative to the project root. The question is whether we want to support this, since in docs we ask users to set the dirs as:

STATICFILES_DIRS = [ ..., os.path.join(BASE_DIR, "components"), ]

At first I was in favour as it could be nicer experience. However, looking at Django's docs on STATICFILES_DIRS, they ask for full paths. So if we on the other hand ask users to put relative paths into STATICFILES_DIRS, it could lead to unexpected behaviour. Instead, we should expect STATICFILES_DIRS to have full paths only.

Which means we'd remove the feature of supplying relative paths to STATICFILES_DIRS. I think this is fine, as resolving the paths relative to BASE_DIR is simple to be done on user-side:

Path(settings.BASE_DIR) / component_dir).resolve()

UPDATE: I've refactored the Loader to expect only full paths from STATICFILES_DIRS.

JuroOravec · 2024-06-22T09:07:40Z

src/django_components/template_loader.py

                if path.is_dir():
                    curr_directories.add(path)

            # Add the directory that holds the settings file
+            # NOTE: This applies only to relative paths
            if settings.SETTINGS_MODULE:


Same issue is here as I mention in https://github.com/EmilStenstrom/django-components/pull/533/files#r1649603172. Basically we treat the dirs in STATICFILES_DIRS as relative altho Django docs ask for absolute paths.

At first I didn't understand why we resolve relative to SETTINGS_MODULE, but turns out that was the initial implementation in https://github.com/EmilStenstrom/django-components/blob/58c6e476f1da7a227512f360946fc8f4f88796eb/django_components/template_loader.py. And then, couple of months ago from now, it was branched out in https://github.com/EmilStenstrom/django-components/blob/567c8ccc3d5747bc01a0a4b83c6d54f3880c4c7a/django_components/template_loader.py to fix #290.

Correct me if wrong, but I think the code branch can be summarized as "Look for component_dir as relative subpath in settings dir (e.g. my_app/settings/), the app where settings are set (e.g. my_app), and the root".

If that's right, we're practically trying to get to the same place as we do in the code branch above with settings.BASE_DIR. So we could just use BASE_DIR instead of relying on SETTINGS_MODULE.

And as I mention in the comment above on BASE_DIR, I think that it's simple enough to use to let users specify the full path via BASE_DIR as:

Path(settings.BASE_DIR) / component_dir).resolve()

So personally I would remove this code block for settings.SETTINGS_MODULE, and tell users to replace that with something like:

Path(settings.BASE_DIR) / "app_with_settings" / "settings").resolve()

Is there maybe something I'm missing about settings.SETTINGS_MODULE?

UPDATE: I've refactored the Loader to use BASE_DIR instead of SETTINGS_MODULE

src/django_components/template_loader.py

JuroOravec · 2024-06-22T09:18:32Z

src/django_components/template_loader.py

+    #         but the default suggests relative, and code suggests both. Document all the supported cases.
+    #      -> Why do we add settings directory?
+    #      -> Could we merge this with `autodiscover_modules()` in `autodiscover.py`, since there we also
+    #         look for `[app_name]/components`, or do these two accomplish different things?


Q&A:

Are STATICFILES_DIRS absolute paths or relative to project root? README suggest absolute, but the default suggests relative, and code suggests both. Document all the supported cases.

I've already touched on this in other comments . But Django asks for full paths.

Why do we add settings directory?

Addressed in https://github.com/EmilStenstrom/django-components/pull/533/files#r1649615698

Could we merge this with autodiscover_modules() in autodiscover.py, since there we also look for [app_name]/components, or do these two accomplish different things?

No. autodiscover_modules imports individual modules (AKA files), whereas in STATICFILES_DIRS, we set DIRS as inputs.

TODO: Use this for updating README

JuroOravec · 2024-06-22T10:30:43Z

src/django_components/apps.py

+        #
+        # TODO - Allow only if autodiscovery is true?
+        #        Because right now it's kinda weird that we import the COMPONENTS.libraries
+        #        Even if autodiscovery is off.
        self.module.autodiscover()


I think it makes sense to make the autodiscover function as part of the public API, so that people can possibly run it when they need to. However, that invites a discussion on how autodiscover should behave:

So autodiscover currently actually imports from 3 places:

Component modules - [app_name]/components.py for each app in the project

Component dirs - As defined by Loader.get_dirs, which takes the STATICFILES_DIRS, and for each one tries to look in a couple of places

Libraries - Modules listed in COMPONENTS.libraries.

How things worked up to now is that:

and 2., so [app_name]/components.py and STATICFILES_DIRS, were loaded only if AUTODISCOVER setting was ON.

3., the COMPONENTS.libraries, were ALWAYS imported.

This feels a bit random, and raises a few questions:

Should we split the logic or rename the function? The "autodiscovery" is really only the part where we search STATICFILES_DIRS for python files. The importing of COMPONENTS.libraries and [app_name]/components.py is more like "import_libraries" or "import_modules"

Should we drop support for automatically searching for [app_name]/components.py, and let users set that manually via COMPONENTS.libraries? Or we can externalize it as a helper function that generates a list of paths like [app_name]/components.py, so users can then pass the results to COMPONENTS.libraries.

Note that the [app_name]/components.py feature is NOT documented currently.

from django_components.utils import gen_app_paths COMPONENTS = { "libraries": [ # Generates `[app_name]/components.py` for each app *gen_app_paths("components"), # Other "my_app.some.path", ... ], }

How should we handle when AUTODISCOVER is ON / OFF?
First of all, I think we should remove [app_name]/components.py (1. from the beginning of this comment) and let users set the same thing via COMPONENTS.libraries

Then, there would be only imports via COMPONENTS.libraries (3.) and STATICFILES_DIRS (2.).

If user wanted to turn off imports via COMPONENTS.libraries, they could just set it to an empty list

And if user wanted to turn off imports from STATICFILES_DIRS, they would set AUTODISCOVER to False.

So that means that AUTODISCOVER would only decide whether to search for python files in STATICFILES_DIRS.

So all-in-all, my approach would be:

The autodiscover.py file would export two functions: autodiscover and import_libraries

autodiscover function would have signature

def autodiscover( map_module_names: Optional[Callable[[str], str]] = None, ) -> List[str]:

Where:

map_module_names is renamed from map_components_modules, and it is kept as we use that in some of the tests.

autodiscover function returns a list of module names that were imported

autodiscover function only searches STATICFILES_DIRS for python files, and does NOT import from COMPONENTS.libraries, nor from [app_name]/components.py.

autodiscover function would run inside the ready hook only if AUTODISCOVER setting is ON. However, user could import the function and run it themselves whenever.

import_libraries function would have signature

def import_libraries( map_module_names: Optional[Callable[[str], str]] = None, ) -> List[str]:

Where:

map_module_names is added to have the same functionality as autodiscover

import_libraries function returns a list of module names that were imported

import_libraries only imports from COMPONENTS.libraries, nothing else.

import_libraries would run inside the ready hook ALWAYS. However, user could import the function and run it themselves whenever.

So src/django_components/apps.py would look like this:

from django.apps import AppConfig class ComponentsConfig(AppConfig): name = "django_components" def ready(self) -> None: # This is the code that gets run when user adds django_components to Django's INSTALLED_APPS from django_components.app_settings import app_settings from django_components.autodiscover import autodiscover, import_libraries import_libraries() if app_settings.AUTODISCOVER: autodiscover()

UPDATE: Updated as described above

src/django_components/autodiscover.py

JuroOravec · 2024-06-22T10:44:30Z

@EmilStenstrom This is a WIP, but I'm eager to hear your thoughts on this, especially on #533 (comment)

for more information, see https://pre-commit.ci

…MODULE

… and make test setup configurable

for more information, see https://pre-commit.ci

JuroOravec · 2024-07-05T14:34:44Z

src/django_components/__init__.py


 if django.VERSION < (3, 2):
    default_app_config = "django_components.apps.ComponentsConfig"
-
-
-def autodiscover(map_import_paths: Optional[Callable[[str], str]] = None) -> List[str]:


Moved autodiscovery to its own file

src/django_components/template_loader.py

JuroOravec · 2024-07-05T14:36:27Z

src/django_components/template_loader.py

            component_dirs = settings.STATICFILES_DIRS
        else:
-            component_dirs = ["components"]
+            component_dirs = [settings.BASE_DIR / "components"]


This has to be now full path instead of relative path, because also settings.STATICFILES_DIRS must be absolute paths.

JuroOravec · 2024-07-05T14:41:25Z

src/django_components/template_loader.py

-            # and, for each app, check if the STATICFILES_DIRS dir is within that app dir.
-            # If so, we add the dir as a valid source.
-            # The for loop is based on Django's `get_app_template_dirs`.
-            for app_config in apps.get_app_configs():


Removed the IF logic on this line (https://github.com/EmilStenstrom/django-components/pull/533/files#diff-756de4c35ec676b6ad61e551979238b310a7806c7c612add84aaec6f376c1b16L65):

if hasattr(settings, "BASE_DIR"):

So that we treat STATICFILES_DIRS as full paths. This means that if people previously specified relative paths in STATICFILES_DIRS, they will now have to set them to full paths themselves.

Also removed the IF logic on this line https://github.com/EmilStenstrom/django-components/pull/533/files#diff-756de4c35ec676b6ad61e551979238b310a7806c7c612add84aaec6f376c1b16L71

if settings.SETTINGS_MODULE:

Instead of relying on SETTINGS_MODULE, we now use only STATICFILES_DIRS and BASE_DIR. So if people used SETTINGS_MODULE before, they will have to set BASE_DIR instead.

JuroOravec · 2024-07-05T14:42:51Z

tests/django_test_setup.py

 import django
 from django.conf import settings

-if not settings.configured:
+
+def setup_test_config(components: Optional[Dict] = None):


Instead of configuring Django setting when we import django_test_setup, it's now wrapped in a function, so we can configure the settings.

JuroOravec · 2024-07-05T14:43:48Z

tests/test_template_loader.py

@@ -0,0 +1,76 @@
+from pathlib import Path


I've split the test_autodiscover.py file and move the tests for template loader into test_template_loader.py.

JuroOravec · 2024-07-05T14:44:54Z

tests/testutils.py

@@ -89,7 +89,7 @@ def autodiscover_with_cleanup(*args, **kwargs):
 ContextBehParam = Union[ContextBehStr, Tuple[ContextBehStr, Any]]


-def parametrize_context_behavior(cases: List[ContextBehParam]):
+def parametrize_context_behavior(cases: List[ContextBehParam], settings: Optional[Dict] = None):


Updated this test helper so that we can pass also other settings to override_settings.

JuroOravec · 2024-07-05T14:46:32Z

As of now, I still need to:

Update the README
I want to test these changes on 1. sampleproject, and 2. my own Django project. Because while the tests are passing, I want to confirm that everything's working in live projects too.

JuroOravec · 2024-07-08T19:48:24Z

Happy to report that everything works well both in sampleproject, and in my own Django project! So this MR is now ready for review @EmilStenstrom

EmilStenstrom

Excellent additions and clearups as usual. I've added a couple of comments and thoughts, but all in all looks good.

README.md

tests/test_autodiscover.py

JuroOravec · 2024-07-29T18:27:22Z

Thanks again for all the feedback!

JuroOravec and others added 2 commits June 22, 2024 10:07

refactor: wip autodiscover refactor

5a84061

[pre-commit.ci] auto fixes from pre-commit.com hooks

73f34ee

for more information, see https://pre-commit.ci

JuroOravec commented Jun 22, 2024

View reviewed changes

src/django_components/template_loader.py Outdated Show resolved Hide resolved

JuroOravec commented Jun 22, 2024

View reviewed changes

src/django_components/template_loader.py Show resolved Hide resolved

JuroOravec commented Jun 22, 2024

View reviewed changes

src/django_components/template_loader.py Show resolved Hide resolved

JuroOravec commented Jun 22, 2024

View reviewed changes

src/django_components/template_loader.py Show resolved Hide resolved

JuroOravec commented Jun 22, 2024

View reviewed changes

src/django_components/autodiscover.py Outdated Show resolved Hide resolved

JuroOravec added 2 commits July 5, 2024 11:35

refactor: v2

dd4a81e

refactor: linter fix

e156263

JuroOravec force-pushed the jo-polish-autodiscover branch from eb13c70 to e156263 Compare July 5, 2024 09:38

pre-commit-ci bot and others added 11 commits July 5, 2024 09:38

[pre-commit.ci] auto fixes from pre-commit.com hooks

55fc67a

for more information, see https://pre-commit.ci

refactor: remove extraneous docstring

690e6be

refactor: update docstrings

a23274d

refactor: update template loader to use BASE_DIR instead of SETTINGS_…

13a4101

…MODULE

refactor: remove unused test structures

3096031

refactor: allow to pass other settings to parametrize_context_behavior

ef38b40

refactor: tests for autodiscover and Loader, simplify settings tests,…

a28bb2b

… and make test setup configurable

refactor: fix typos

f45d5fa

[pre-commit.ci] auto fixes from pre-commit.com hooks

d937d8c

for more information, see https://pre-commit.ci

Merge branch 'master' into jo-polish-autodiscover

a7c9f07

refactor: update docstrings

4bbd06b

JuroOravec commented Jul 5, 2024

View reviewed changes

src/django_components/template_loader.py Outdated Show resolved Hide resolved

JuroOravec commented Jul 5, 2024

View reviewed changes

JuroOravec added 2 commits July 6, 2024 10:16

docs: update README autodiscovery section

7276cb6

refactor: update release notes

d2255a7

JuroOravec changed the title ~~WIP: Autodiscover refactor~~ refactor: Prepare autodiscover and template loader for v1 Jul 6, 2024

Merge branch 'master' into jo-polish-autodiscover

f48390d

JuroOravec marked this pull request as ready for review July 8, 2024 19:46

JuroOravec requested a review from EmilStenstrom July 8, 2024 19:48

EmilStenstrom approved these changes Jul 11, 2024

View reviewed changes

README.md Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

tests/test_autodiscover.py Outdated Show resolved Hide resolved

JuroOravec added 5 commits July 29, 2024 20:00

refactor: update docs

de92125

refactor: use test-level setup/teardown in tests

3b29553

Merge branch 'master' into jo-polish-autodiscover

d4b9058

Merge branch 'master' into jo-polish-autodiscover

446358e

chore: bump version

e15292c

JuroOravec merged commit 8cb8855 into EmilStenstrom:master Jul 29, 2024
6 checks passed

JuroOravec deleted the jo-polish-autodiscover branch July 29, 2024 18:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: Prepare autodiscover and template loader for v1 #533

refactor: Prepare autodiscover and template loader for v1 #533

JuroOravec commented Jun 22, 2024 •

edited

Loading

JuroOravec Jun 22, 2024

JuroOravec Jun 22, 2024 •

edited

Loading

JuroOravec Jul 5, 2024

JuroOravec Jun 22, 2024

JuroOravec Jul 5, 2024

JuroOravec Jun 22, 2024 •

edited

Loading

JuroOravec Jul 5, 2024

JuroOravec Jun 22, 2024

JuroOravec Jul 5, 2024

JuroOravec commented Jun 22, 2024

JuroOravec Jul 5, 2024

JuroOravec Jul 5, 2024

JuroOravec Jul 5, 2024

JuroOravec Jul 5, 2024

JuroOravec Jul 5, 2024

JuroOravec Jul 5, 2024

JuroOravec commented Jul 5, 2024 •

edited

Loading

JuroOravec commented Jul 8, 2024

EmilStenstrom left a comment

JuroOravec commented Jul 29, 2024

		matched_files: List[Path]


		def search(search_glob: Optional[str] = None, engine: Optional[Engine] = None) -> SearchResult:

refactor: Prepare autodiscover and template loader for v1 #533

refactor: Prepare autodiscover and template loader for v1 #533

Conversation

JuroOravec commented Jun 22, 2024 • edited Loading

Choose a reason for hiding this comment

JuroOravec Jun 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JuroOravec Jun 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JuroOravec commented Jun 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JuroOravec commented Jul 5, 2024 • edited Loading

JuroOravec commented Jul 8, 2024

EmilStenstrom left a comment

Choose a reason for hiding this comment

JuroOravec commented Jul 29, 2024

JuroOravec commented Jun 22, 2024 •

edited

Loading

JuroOravec Jun 22, 2024 •

edited

Loading

JuroOravec Jun 22, 2024 •

edited

Loading

JuroOravec commented Jul 5, 2024 •

edited

Loading