Use django-components as a vehicle to introduce new features into Django's templating

[JuroOravec]

Code of Conduct

• I agree to follow Django's Code of Conduct

Package Information

django-components (https://github.com/django-components/django-components)

Problem

TL;DR

• Use django-components as a vehicle to introduce new features into Django's templating.
• Templating with the "old" API like django.shortcuts.render() or Template.render() would remain unchanged
• To opt into "new" API, people would define Components instead of plain Templates. Component have templates. But a Template defined for a component would be handled differently from an "old style" API Template loaded via render().
• There's still a lot of work until django-components hit v1, easily ~6 months as I'm the only active developer (interested? Join me!). But the discussion can start already.

Intro

Hi, I'm one of maintainers of django-components. django-components brings modern frontend development practices (e.g. Vue, React, Laravel Livewire) to Django templates.

Some of you may know, I want to eventually make django-components the next generation of the Django's templating.

The other day I was working on an Rust-based AST parser for Django templates. The AST output could be used for linting, syntax highlight, formatting, etc. But it also replaces Django's Lexer in django-components in order to resolve the component inputs inside a template, e.g. turning {% slot "append" my_var=[1, 2, 3] / %} into args and kwargs ["append"] and {"my_var": [1, 2, 3]}.

The AST parser has ~8k lines of code whch includes ~120 tests. And every time I need to update it, it takes about a day. So I'm figuring out all the requirements for the AST parser ahead of the time, so that I will need to update the AST parser only once.

This led me to design the first look of how the interaction between Django and django-components could look like. So I wanted to share what I've got.

There's still a lot of work until django-components hit v1, easily ~6 months as I'm the only active developer (interested? Join me!). But the discussion can start already.

Why django-components

If you're not familiar, see this section to understand why django-components.

Example

Here's an example from a real project where django-components was used.

• All code related to the component is in the Python file:
  1. View function (django-ninja) that renders the component
  2. The component itself
  3. Endpoint for handling form submission
• Only the template HTML is in a separate file.

Click to expand

from django.http import HttpRequest
from django.shortcuts import get_object_or_404
from django_components import Component, get_component_url, register
from ninja import Body, Query, Schema
from pydantic import BaseModel

from api.models.chk import PhaseTemplate, Process
from api.helpers.process import process_create
from app.helpers.auth import auth_requirements
from app.helpers.settings import dliver_settings
from app.helpers.view import json_view
from app.ninja.router import view_router
from app_home.auth import processes_auth_spec
from app_home.components.martor.martor import MartorDependencies
from app_home.helpers.layout import LayoutData, get_layout_data
from app_home.routes import Routes, get_route_url
from app_process.components.process_edit_form.process_edit_form import CreateOrUpdateProcessInput


########################################################
# VIEWS
########################################################


class ProcessCreateQuery(Schema):
    redirect: str | None = None


@view_router.get(
    "/modules/add",
    url_name=Routes.PROCESS_CREATE_VIEW,
    tags=["Process"],
    include_in_schema=dliver_settings.INCLUDE_VIEWS_IN_OPENAPI,
)
@auth_requirements(processes_auth_spec)
def process_create_view(request: HttpRequest, query: Query[ProcessCreateQuery]):
    layout_data = get_layout_data()

    return ProcessCreatePage.render_to_response(
        request=request,
        kwargs=ProcessCreatePage.Kwargs(
            layout_data=layout_data,
            redirect=query.redirect,
        ),
    )


########################################################
# COMPONENTS
########################################################

@register("ProcessCreatePage")
class ProcessCreatePage(Component):
    template_file = "process_create.html"

    class Media:
        extend = [MartorDependencies]

    class Kwargs(BaseModel):
        layout_data: LayoutData
        redirect: str | None

    def get_template_data(self, args, kwargs: Kwargs, slots, context):
        success_redirect_url = get_route_url(
            # Placeholder for the Process ID, as it will be set in JS after the process is created
            Routes.PROCESS_VIEW,
            {"process_id": "00000"},
        )

        submit_url = get_component_url(ProcessCreatePage)

        return {
            "redirect": kwargs.redirect,
            "layout_data": kwargs.layout_data,
            "success_redirect_url": success_redirect_url,
            "submit_url": submit_url,
        }

    ########################################################
    # REST API
    ########################################################

    class View:
        # Create
        @auth_requirements(processes_auth_spec, mode="api", is_method=True)
        @json_view(name="create_process", standardized=False)
        def post(self, request: HttpRequest, input: Body[CreateOrUpdateProcessInput]):
            phase_template = get_object_or_404(PhaseTemplate, pk=input.phase_template)

            data = {
                **input.dict(),
                "phase_template": phase_template,
            }

            # Check if there isn't already a step with the same name
            new_name = data["name"]
            if Process.objects.filter(name=new_name).exists():
                raise ValueError(f"A process with the name '{new_name}' already exists")

            process = process_create(data)
            return process

Features

• Extended template syntax (docs)

  django-components is a superset of Django's template features. In django-components, you can:

  • Use Python expressions (Same safety "sandboxing" as Jinja does). These are identified by parentheses:

    {% nav max_height=(50 if not user.logged_in else 90) / %}

  • Use list and dict literals inside the template

    {% table data=[{"name": "John"}] headers=["name"] / %}

  • Easily format strings in template with nested tags {% ... %} and expressions {{ ... }}:

    {# title = "Welcome, John Green" #}
    {% card title="Welcome, {{ user.first_name }} {{ user.last_name }}" / %}

  • Allow special characters (# @ . - _) in keys:

    {% component "calendar" my-date="2015-06-19" @click.native=do_something #some_id=True / %}

  • Spreading a list of args or a dict of kwargs with ...:

    {# calendar_kwargs = {"my-date": "2015-06-19" } #}
    {% component "calendar" ...calendar_kwargs / %}
    
    {# Is the same as #}
    {% component "calendar" my-date="2015-06-19" / %}

  • Flat dicts - Use dict:key=value syntax instead of dict={"key": value}

    {# Same as attrs={"class": "pa-4"} #}
    {% card attrs:class="pa-4" / %}

    • NOTE: This was a crutch before the literal lists and dictionaries were supported.
  • Multiline support for tags {% ... %}, comments {# ... #}, and expressions {{ ... }}

    {% table
      data=[{"name": "John"}]
      headers=["name"]
    / %}

  • Standardized self-closing tags have / at the end

    {# So this #}
    {% table / %}
    
    {# Is the same as #}
    {% table %}{% endtable %}

  • Allow {# ... #} comments almost anywhere:

    {# outside #}
    {% table {# inside template tag #}
      data=[  {# inside a list literal #}
        {
          "name": "John"  {# inside a dict literal #}
        },
      ]
      some=my_var
        |default:"empty"  {# between vars, filters, and filter args #}
        |upper
    / %}

• Support for JS and CSS variables

  • Each component can define JS and CSS code that will be shipped with this component.
  • The JS and CSS code can either in a separate file or inlined (same applies to the template)
  • JS and CSS variables
    • Defined via Component.get_js_data() and Component.get_css_data()
    • CSS vars are used with var(--my-var)
    • JS vars are accessed by defining a callback for $onLoad()
    • Each component instance can have different values for the CSS and JS variables.
    • JS and CSS variables are separate from the component's JS and CSS code

  class Table(Component):
     def get_js_data(self, args, kwargs, slots, context):
         return {
             "options": kwargs["options"],
         }
  
     js = """
         // Code outside of `$onLoad()` runs ONLY ONCE no matter how many instances
         // there are.
         const onClick = () => { ... };
  
         // The callback passed to `$onLoad()` runs for EVERY component instance
         $onLoad(({ options }) => {
             const containerEl = document.querySelector(".my-container");
             options.forEach((option) => {
               const newDiv = document.createElement("div");
               newDiv.textContent = option;
               containerEl.appendChild(newDiv);
             });
         });
     """
  
     def get_css_data(self, args, kwargs, slots, context):
         return {
             # Warn user that no options are available
             "border_color": "red" if not kwargs["options"] else "blue",
         }
  
      css = """
        .my-container {
          border: 2px solid;
          border-color: var(--border_color);
        }
      """

• Support for HTML fragments (docs)

  • Tested with HTMX, AlpineJS, and vanilla JS.
  • Works with the JS/CSS variables - if a fragment defines components with JS/CSS variables, these variables will work when the fragment is inserted into the browser page.
  • Deduplication - If you insert multiple fragments with components with the same JS/CSS code, the code will NOT be re-downloaded again.

• Extensible (docs)

  • django-components has a rich plugin system meant to unify other existing Django libraries like django-cotton, etc.
  • Extensions can:
    • Modify component's HTML/JS/CSS
    • Modify or validate component inputs (e.g. validate with Pydantic)
    • Component-level caching is implemented as an extension
    • Or can simply use Components as metadata, like Storybook extension uses components to generate Story files.
  • See here all proposed or implemented extensions

• As flexible as Vue or React

  Components also offer features that allow component authors to do similarly advanced things as e.g. in Vuetify. These features are:

  • Component slots as known from Vue.
  • Context providers as known from Vue and React.
  • Hooks before/after rendering

  See this Tabs examples, where the component's API is made to be simple for the user - user just defines the Tablist and Tabs inside it.

  So user defines both the header and content in a single Tab component. But behind the scenes, the headers are placed in a different placed in the actual HTML than the tab contents.

  {% component "Tablist" id="my-tablist" name="My Tabs" %}
      {% component "Tab" header="Tab 1" %}
          This is the content of Tab 1
      {% endcomponent %}
      {% component "Tab" header="Tab 2" disabled=True %}
          This is the content of Tab 2
      {% endcomponent %}
  {% endcomponent %}

• Misc

  There's more, altho the remaining features are either small, or there may be an alternative to these:

  • Inside the component you can access the entire component tree metadata (parent, grandparent, etc), including the inputs for each of them.
  • Helper template tag for rendering HTML attributes
  • Helper for testing @djc_test

  See https://django-components.github.io/django-components/latest/examples/

• Performance

  • Infinite depth - In django-components, you can render an infinitely-deep tree. In original Django, you can go max ~40-60 layers before you reach maximum recursion depth exceeded error.

    This is handy when rendering deeply nested data structures like treeviews.

  • Further performance gains.

    As of v0.143.0, django-components is about 3.5x slower than Django templates.

    However, there's A LOT of potential speedups:

    • (???) Using the Rust-based AST parser.
    • (~50%) Replace {% ... %} and {{ ... }} with static text on first render #1083
    • (???) Expression-level caching #1473
    • (~15-20%) Replacing one validation that runs simply to nicely format error messages #935

  So I think we'll end up with a similar performance as original Django templates, while doing much much more.

• Maintenance / Development

  • Explicit component / template inputs

    When I was using vanilla Django templates, one footgun was that Django didn't offer any way to manage template inputs.

    • Inside the template, I didn't know what types the variables are
    • Outside the template, I had to manually track which variables (and what types) are being passed to the template

    Ideally, template inputs should be defined in a single location only, and all instances where we render the template (whether that's in Python or in template) should raise a linter / type checking error to help us find where we need to update code. So same as it works in Vue or React.

    django-components aims to solve this by:

    • Defining a single source of truth on the Component class, e.g.

      class Table(Component):
          class Kwargs:
              my_var: str
              other: int = 0

    • When passing data from Python, we can reuse the same class Table.Kwargs for type hints:

      rendered = Table.render(
          kwargs=Table.Kwargs(
              my_var="abc",
              other=12,
          )

    • Inside the component's methods it's the same, using Table.Kwargs for type hints:

      class Table(Component):
          class Kwargs:
              my_var: str
              other: int = 0
      
          def get_template_data(self, args, kwargs: Kwargs, slots, context):
              print(kwargs.my_var)

    • And from inside the template, the IDE extension (language server) would use Table.Kwargs class to check if the input to {% table %} template tag is correct.

  • Leaky isolation and empty string as "no value"

    Another footgun issue is the combination of 2 things:

    1. When you use {% include %}, then by default the child template has access to the WHOLE parent's context.
    2. Django has this feature where if you refer to an invalid variable, it will default to an empty string.

    When working in a team, the two create a setting ripe for errors:

    1. I have defined child.html, and updated the variables passed render("parent.html") to include all I need.
    2. Couple of months later, my collegue refactors parent.html. He notices that some_var IS NOT used in parent.html. So he removes it from the object when calling render().
    3. However, it was still used in child.html. And because of Django's silent errors, we won't find out about this until we visit the page itself, which is now broken.

    {# parent.html #}
    <div>
      {{ some_var }}
      {% include "child.html" %}   {# other_var not mentioned! #}
    </div>
    
    {# child.html #}
    <span>
      {{ other_var }}
    </span>

    django-components fixes this:

    1. Explicit component interfaces - When defining components, only the args and kwargs you explicitly pass down will be accessible. Thus, you can be 100% sure that if you don't see a variable inside a template, then you can safely remove it. This makes big refactors so much easier.
    2. Raise proper error on missing variables.

    {# parent.html #}
    <div>
      {{ some_var }}
      {% component "child" other_var=other_var %}   {# other_var mentioned! #}
    </div>
    
    {# child.html #}
    <span>
      {{ other_var }} 
    </span>

Integrating django-components

Ok, so by now I hope you agree that having django-components as part of Django would be cool af.

Let's discuss the integration.

Most of what I know right now has been written down here - django-components/django-components#1141 (comment)

• Stages

  1. django-components is a large and complex project, that also defines:

     • 1 NPM package for JS code that runs in the browser. It's structured as an NPM package so it can be written in TypeScript, tested, etc.
     • A couple of Rust packages to improve the speeds.

     So integration may be non-trivial.

     So the minimum that can be done is that, if django-components were to be part of Django, would be to keep it as a separate package and repo, and just encourage people to use it as the "new API for Django templates".

     Installation process is IMO not great, it requires too many steps when using also JS/CSS and HTML fragments. Potentially something could be done about that.

     django-components also has to make changes to a couple of Django classes like Template. But these have been implemented so they don't break original implementation. These changes could live upstream in Django.

     As per the Rust packages, as that can be a concern - Rust code is copiled using maturin, and the way it's built results in 95 different builds for different OS's and architectures. There can always be some exotic env that's not captured, but I think for the 99% of the cases this should work.

     • So for this reason it's also a good idea to initally only recommend django-components as a standalone package, to give us time to see if there are any compatibility issues with the Rust dependencies.

  2. If django-components was to be shipped with Django, initially it could just sit next to the existing Django templating logic.

     • It would be automatically installed, but otherwise it would work practically the same as currently.
       • People could insert a {% component %} into a regular template, and vice versa, as they can do now when they install django-components.
       • Component's templates would be handled differently, allowing for the extended syntax. While regular templates loaded with get_template() would use the old Django API.
     • Hard to say more on this, other than what I've already mentioned in [v2] Ideas django-components/django-components#1141 (comment)

  3. In the long run, django-components could become it's own templating backend. This would be the last phase, and the old Django syntax and API like get_template() would no longer be present.

• Separate versioning of Django and components

  As mentioned here, components might have a version attribute to set how they should render:

  1. V1 - Compatible with current Django. The extended template syntax works only inside the {% component %} tag.
  2. V2 - The extended syntax works for all tags {% .. %} and expressions {{ ... }} inside the component's template.

  class Table(Component):
      version = 2

  V2 will require to us (in django-components) to re-write Django's built-in template tags on top of our BaseNode.

  V2 will also break third-party template tags, as those too will need to be rewritten on top of BaseNode. But that will be the responsibility of the respective authors.

  The V2 would also introduce some breaking changes to the built-in template tags used inside component templates. E.g.

  {% lorem 2 w random %}

  Would have to change to:

  {% lorem 2 "w" "random" %}

  Note that this would apply ONLY IF user sets version = 2 in their component, and would apply ONLY to that component's template.

  So this could be a nice and gradual way how to migrate people's templates onto the new features.

• Where should code live?

  As mentioned earlier, since django-components project contains also JS and Rust code, I think 2 options could make sense:

  • Put all django-components code into a single repo (right now the Rust code is in different repo), and move that repo under Django org.
  • Alternatively, if Django repo was made into a monorepo that would support also multiple languages, then it could all live in the main Django repo.

Relevant ideas / proposals

1. Rewrite Django Templates in Rust (#1)

• The two works are complementary, in the sense that django-components could re-use some Rust-based classes (e.g. Context). These 2 implementations are not 1-to-1, as django-components introduces a new architecture and novel ways to improve the performance.

  (NOTE: django-components is already using Rust in some parts as well, 1) to help with Django template AST, and 2) with modifying the generated HTML.)

• Why has django-components different architecture? Because original Django templates are quite loose with the definition of template tags and their inputs. OTOH, django-components has standardized this and thus we can achieve much more nuanced optimizations.

  E.g. consider {% lorem %} and {% with %} template tags:

  {% lorem 2 w random %} 
  
  {% with my_var=w %}

  lorem treats w as a string. While with treats it as a variable. So we can't identify variables just by looking at them, we need to know the implementation of the tag too.

  Instead, django-components standardizes the input. Each tag interprets its inputs as args and kwargs, same way as Python functions work:

  {% component "table" rows=rows headers=["name", "job title"] %}

  • -> "table" - 1st positional arg, string literal
  • -> rows=rows - 1st kwarg, variable
  • -> headers=["name", "job title"] - 2nd kwarg; literal list

  With django-components it's possible to know statically what variables are used in the template. This can be used to enable IDE support, where a linter could check the template tag against the tag definition (defined as a class with class attributes, so it's again defined statically).

  E.g. here's a definition of a {% fill %} tag that statically declares its signature. The signature of FillNode.render() is what will be expected in the template tag,

  from django_components import BaseNode
  
  class FillNode(BaseNode):
      tag = "fill"
      end_tag = "endfill"
      allowed_flags = ()
  
      def render(
          self,
          context: Context,
          name: str,
          *,
          data: Optional[str] = None,
          fallback: Optional[str] = None,
          body: Optional[SlotInput] = None,
      ) -> str:
          ...

  So the {% fill %} tag requires to be passed a name positional arg, and then optional data, fallback and body kwargs:

  {% fill "pagination" data="data" %}
    Current page: {{ data.current_page }}
  {% endfill %}

  (NOTE: django-components tags also support flags, such as {% include %}'s only.)

Rationale

django-components cuts deep into core Django functionality, and introduces internal changes that Django codebase itself could benefit from - there's a possiblity that the more-nuanced design could speed up the template rendering overall.

Moreover, django-components is on par to Vue and React with expressivity and capabilities. From personal experience, when companies are choosing Django for new projects, it's usually that they use Django only as the backend/server, and use React (or other) for frontend. django-components could make new web projects much simpler to manage. You wouldn't need to run 2 projects side-by-side. Just Django and purely python-based UI.

Additional Details

No response

Implementation Details

No response

[github-actions]

Thank you JuroOravec for sharing your idea! We have a lot of them so please be patient. You can see the current queue here.

Community instructions

For commenters, please use the emoji reactions on the issue to express support, and/or concern easily. Please use the comments to ask questions or contribute knowledge about the idea. It is unhelpful to post comments of "I'd love this" or "What's the state of this?"

Reaction Guide

• 👍 This is something I would use
• 👎 This is something that would cause problems for me or Django
• 😕 I’m indifferent to this
• 🎉 This is an easy win

[JuroOravec]

Thanks to everyone who gave feedback so far, whether here, Reddit, or personal convos!

My take heavily depends on the performance aspect, so that's what I want to focus on while working on django-components:

• IMO if our implementation ends up being substantially faster than current Django templating (e.g. 20% faster), then IMO it would be a no-brainer - we'd get faster templating and lots of new features for relatively small cost.
• If it ends up slower, than we should stick with original django templates, so users consciously opt-into django-components if they want to.
• If it ends up somewhere in-between, we should carefully weigh the pros and cons.

Also, based on the convos, some people who saw this responded to the idea of merging django-components into django codebase.

The way I meant it initially was that even if we agreed only on stage 1 (django-components being recommended, but its own repo and project), to me even that would be considered a "merging".

So it's a tiered proposal - e.g. in the case that we decide not to merge django-components' code to django, but it turns out that django-components is faster at rendering templatings, I think it's only fair for the Django users to recommend to use django-components.

[JuroOravec]

The best counterargument I've read is on security (see Reddit thread):

Django would need per-architecture releases if it contained bundled compiled Rust. This would also change the security profile of the project as blobs are not auditable. It would also increase maintenance burden on the project if it was mainline. It's likely not reasonable to expect everyone who "pip install django"s to also compile some Rust extensions.

I don't have experience with these sort of audits, but I can imagine that, yes, if there are orgs that rely on Django's current security profile, and if compiled C/Rust can't be analysed, I can understand it would be potential dealbreaker / a lot of headache for all those orgs.

[JuroOravec]

Another good point I got on Reddit:

To be honest, I love the idea of this package. But also I don't think that everyone needs it. If you want to ship it with Django, you have to be sure that almost everyone will use it. That's the idea.

Otherwise, Django will be bloated as hell. Right now it isn't. And I hope it will stay that way.

Please, think about other applications, which do not require templating at all. Such as SPA apps which are API only. Or GraphQL ones. Or based on protobuf and gRPC. Those use cases won't benefit from those templates.

But this raises more questions:

Shouldn't it then be that the current Django templating be moved out of core Django? Django is batteries-included, but where is the line? E.g. if half of people use Django as a backend-only, and the other half uses a different framework for rendering the UI because the built-in one isn't powerful enough, does it even make sense to keep the original templating?

Of course, it does make sense to keep the original templating engine for the backwards compatibility, people rely on it.

But because we're in the situation where Django already has a built-in templating engine, then I think it's not unreasonable to make it better.

[knyghty]

Would I like components in core Django? Done right, yes.

But I'm not sure this package is it. There are other component libraries out there and there's nothing that says to me that this one is clearly the best.

I would rather start asking the question of what we would want in a components system rather than integrating something that already exists and looks quite opinionated.

[EmilStenstrom]

When I started building the django-component (DJC) many years ago, it was because I really believed in the idea that large sites should be broken up into components, not pages. Django does not have a component concept right now, and for larger sites, this means everyone is building their own. A component is not just an {% include %}, but something that is built to be reused with parameters, that handles component-specific CSS and JS, and makes development easier. From that core idea, a community (now led by Juro) has developed, that has made DJC a viable alternative for both beginners and very complex SPA:s.

@knyghty A big difference between DJC and the others is that DJC is a component framework, not just a library. When it comes to syntax, you can make it behave like many of the other popular component libraries. As that, it's more like the ORM (multiple databases) or template engines (multiple languages).

I'm sure some people will look at the defaults and think of them as opinionated, but we have to pick something, otherwise we'd end up with something that takes ages to get started with. When picking, we've tried to pick defaults that "feels like Django".

I agree that a good place to start the discussion is what you would want from a component framework? Because I'm sure that DJC could morph into that quite easily.