Add "import" and "include" template features

[derekatkins]

Proposed change

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Example entry for configuration.yaml:

# Example configuration.yaml
template: >
  {% import 'macros.html' as macros -%}
  {%- macros.test_macro() %}

# Example /config/templates/macros.html
{% macro test_macro() -%}
  This is a test macro!
{%- endmacro %}

Additional information

This PR enables the include and import jinga functionality. It will look in /config/templates for the imported (or included) templates.

• This PR fixes or closes issue: fixes Jinja template import or include results in stack trace and error #42224
• This PR is related to issue:
• Link to documentation pull request: Documentation for include and import jinja2 features in templates home-assistant.io#15372

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• The code has been formatted using Black (black --fast homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

• No score or internal
• 🥈 Silver
• 🥇 Gold
• 🏆 Platinum

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[frenck]

Thanks for the PR @derekatkins.

I've left a comment that needs to be addressed, furthermore, this hits core functionality and tests should be added.

[frenck]

This should not have a statically configured path, but use the actual configuration path of the system instead. Besides, should this be fixed to a templates subfolder?

[derekatkins]

Thanks @frenck . Very good point. How does one determine the actual configuration path?

As for the fixed templates subfolder, I did that for security reasons to separate out from other config files and ensure one could not, for example, {% include 'secrets.yaml' %}

[frenck]

Its in the hass object: hass.config.config_dir.

I kinda get the security concern, however, if one has access to this level of things, that might be a false sense of security already. The downside of limiting/locking it to the templates folder, is limiting the user to structure the configuration to their own likings (e.g., packages use).

[derekatkins]

Ah, thanks. Didn't see config_dir in the docs!

Well, there is nothing stopping the user from creating a /config/templates/{X,Y,Z}/... structure for packages.

[derekatkins]

Unfortunately, @frenck, that doesn't work:

homeassistant/helpers/template.py:1265: in __init__
    super().__init__(loader=jinja2.FileSystemLoader(hass.config.config_dir+"templates"))
E   AttributeError: 'NoneType' object has no attribute 'config'

Apparently __init__ is being called with hass=None at some point. I don't know if there is a way to lazy-eval that?

[derekatkins]

I think I know how to solve this... But it means that the feature will only work if there is a hass env.

[derekatkins]

@frenck -- I have this now working locally based on config_path, but the code is a little ugly because I can only instantiate the loader if hass is not None, so I have to call super.__init__() differently. I can push now if you want to see it.

Now to add tests as per your other request; how can I add a "file" in the config path in the test infrastructure? I can do the "stupid" thing and literally open/write/close/unlink -- but is there a better way to push something into the filesystem under the test?

[derekatkins]

I now believe I have made the config and test changes you requested.

[balloob]

Before you invest more time in this feature, I think that we first discuss if this is a feature that we want, and if it is worth the structural changes to Home Assistant.

What would be a use case that is covered?

[derekatkins]

@frenck Hi. So I pushed the code that fixes the hard-coded config-dir. I still think the they should live under their own subdirectory tree of the config tree.

As for adding tests, how can I mock up the filesystem so that jinja2.FileSystemLoader will properly open_in_exists my test-file data? mock_storage was suggested to me, but as far as I can tell that API only mocks the hass entity state, not the underlying filesystem?

@balloob Subroutines in templates. I'm building a set of automations to handle all my thermostats (I have 12) and want to do the following:

• Reset thermostats at specific times. E.g. morning, day, evening, night. When the thermostat-specific time fires, change the thermostat to the HA-configured setting (input_number for each setting).
• Track the current state. If you adjust the thermostat, I want HA to track that, but moreso I want to track it for the current time-bucket for that thermostat. That means I want to have a template that looks something like find_time_bucket_for_thermostat(thermostat_id) so I know which input_number to adjust.
• I have different kinds of thermostats; some of them have high and low temp settings, some only have temperature
• not all thermostats have all the settings, so I want to parametrize based on the thermostat_id. This means I need twice the code, one set to handle the high/low and one to handle the heat-only thermostats. I'd like to minimize the duplicated code (read: create macros).
• each thermostat might have different time frames (e.g., my office has a different 'evening' than my kids' playroom
• minimize the amount of duplicated code in my automations/scripts/etc.
• add a vacation mode, that overrides everything (and blocks manual adjustment storage).

Right now there is no way to write a script that actually returns a value, so I cannot create that "time_bucket_for_thermostat" function. The best I can do is create a hidden input_text for each thermostat_id and then have automations that set that based on now(). But to me, being able to build a template macro is a better solution. Jinja supports it, and has for a while. And this actually works and solves my underlying issue cleanly.

Moreover, people have been asking for macros for YEARS. So a better question is: what reasons would there before NOT to support macros?

I am curious, short of the test infrastructure, what structural changes are required? I'm running with this change and it works quite well for me in my tests (although I am not, yet, using it in production because I don't want to have to patch after every update).

Thanks!

[frenck]

I must admit, that for all above-listed reasons, I'm like: Why not use a script?

As for the "Why not", that is simply answered. Every feature adds maintenance and new possible cases of things that can go wrong. We are not aiming to be able to do everything, we are aiming to have a flexible and uniform set of things that are commonly shared/used. Hence, when adding a new feature the question is asked like balloob did and not the other way around.

[derekatkins]

I must admit, that for all above-listed reasons, I'm like: Why not use a script?

Because a script cannot return a result. You are right that I could probably write the master automation as a python_script, but there are other places where I'd like to use macros in my automations too. For now I've just copy-and-pasted my templates, but honestly it would be nice to be able to define them once.

As for the "Why not", that is simply answered. Every feature adds maintenance and new possible cases of things that can go wrong. We are not aiming to be able to do everything, we are aiming to have a flexible and uniform set of things that are commonly shared/used. Hence, when adding a new feature the question is asked like balloob did and not the other way around.

That's a fair point. As I mentioned above, this feature has been requested for years, going back to at least 2015 when there was a comment that suggested it could be done because jinja supports it. So there are clearly users who want it. I'm happy to jump through the hoops you all want to properly test that we pass the correct loader to jinja, but I also assume that jinja tests themselves that their loaders work.

[derekatkins]

As you may be able to tell, I'm a python n00b. With some helpful prodding from @raman325 on Discord he pointed me in the right direction to patch the loader in order to test that the templates render as expected when importing or including expected data. So I have added that test (positive and negative), so show that we get the expected template results when we load what we expect, and we get a TemplateError if we try to load something that doesn't exist.

As to why this is useful -- it allows anyone to build macros and de-duplicate any automation or script code by refactoring it and putting it into a single place. I.e., it can simplify a bunch of complicated and repetative templating.

[raman]

why did you tag me in the conversation?

[derekatkins]

Sorry, was trying to say that @ raman on HA discord helped me and wanted to thank them. But of course that could be someone completely different. My deepest, sincerest apologies for tagging you.

[derekatkins]

I meant to tag @raman325

[balloob]

Still not clear what can't be done inside a script using variables and choose. Or using the python script integration. Our templates are not supposed to be a programming language.

[derekatkins]

Hi. There are several reasons:

1. As far as I can tell there is no such thing as a conditional service call. In other words, I can't do something like this in a script (I've tried):

{% set ... %}  # A bunch of variables get set here in a complicated series of templates
{% if x %}
  - service: script.turn_on
    entity_id: script.script1
    ....  (data based on one part of X)
{% else %}
  - service: script.turn_on
    entity_id: script.script2
    .... (different data based on a different part of X)
{% endif %}

The only way I have figured out how to do this is by separate automations that call separate scripts based on the underlying type of entity, however....

2. If you have a series of template manipulations that you need to use in different automations or scripts, then you have to copy-and-past them.. Variables are only part of the solution, but if it takes several lines of template to compute what a variable needs to be, and you need that variable in different automations, it would be nice to be able to configure a macro.

3. I'm not trying to turn templating into a turing-complete language. I'm just trying to make it easier to handle repetitive templating, either by using a macro or including common text so you don't have to copy-and paste.

[balloob]

See choose (as I also mentioned, but not linked, in my previous comment)

[derekatkins]

Thanks. That potentially solves ONE of my problems and definitely will let me clean up some of my scripts and automations. Can choose be used in a script as well as an automation? The examples only show automations. If it only works there and not scripts then it probably doesn't help as much as a macro would.

Still, it doesn't solve the problem of having to repeat templates and variables in different automations due to being necessarily different automations but requiring similar computed data. For example, I have one automation to track thermostats. When any thermostat setting changes (i.e., you change the setting at the thermostat) it triggers the automation, which then has to perform a bunch of template operations to compute the correct backing-store input_number entities to use to save the current settings.

Then I have a different set of automations that kick off due to the time changes to set the thermostat(s) from the input_number settings due to changing from night -> morning -> day -> evening -> night. This automation has to perform the same set of template operations to compute the correct backing-store input_number entities to use to restore temperature settings. It doesn't make sense to build both of these into one automation, nor does it make sense to build this into one script. But currently it's like 6-8 lines of templating and multiple variable settings to do all the computations. This is why I want macros, so I can write those template operations once and use those in all the different automations that need to perform similar background computations. It's not quite as convenient as I would like, because I can't set variables within the macros that get exposed to my main template, but it would still allow me to condense template code (and keep it DRY).

[frenck]

Considering your questions, I'm pretty sure you are approaching things wrongly, mainly due to the missing functionality of the platform. Missing that knowledge, combined with a vision of your own solution, results in this PR.

choose is one example, your question about automations and scripts is another. The action of an automation == a script. Yes, you can use everything from a script in there. This is also documented in several places.

Still, it doesn't solve the problem of having to repeat templates and variables in different automations due to being necessarily different automations but requiring similar computed data. For example, I have one automation to track thermostats. When any thermostat setting changes (i.e., you change the setting at the thermostat) it triggers the automation, which then has to perform a bunch of template operations to compute the correct backing-store input_number entities to use to save the current settings.

This can all be done in scripts.

It doesn't make sense to build both of these into one automation, nor does it make sense to build this into one script.

You don't have to and isn't related to anything in this PR or issue you are describing. Even this PR adds macros, it doesn't change anything about that.

I'm even thinking if we (alternatively) should allow/create a possibility to allow one to plug in custom filters in the future (maybe even defined in YAML itself?). Even that is ambiguous, as automation/script variables can provide that functionality (kinda).

I'm leaning voting against this change at this point.

[derekatkins]

Considering your questions, I'm pretty sure you are approaching things wrongly, mainly due to the missing functionality of the platform. Missing that knowledge, combined with a vision of your own solution, results in this PR.

That is certainly a possibility; I've been a software developer for 30 years so sometimes old habits are hard to break, namely if you have to write the same code twice you're doing something wrong.

Still, it doesn't solve the problem of having to repeat templates and variables in different automations due to being necessarily different automations but requiring similar computed data. For example, I have one automation to track thermostats. When any thermostat setting changes (i.e., you change the setting at the thermostat) it triggers the automation, which then has to perform a bunch of template operations to compute the correct backing-store input_number entities to use to save the current settings.

This can all be done in scripts.

I'm still at a loss for how to do that without a macro to help me. Let me take my backing-store save and load example. I have an automation that triggers when climate.foo changes. So, given now() and the climate.foo entity, I strip I set climate. and set therm = foo (using appropriate template code). The entity has a corresponding input_datetime.{{therm}}_{weekday,weekend}_{morning,day,evening,night} I need to find the correct settings for doy (weekday or weekend) and tod (morning,day,evening,night) so that I can then save the settings to input_number.{{therm}}_cool_{{doy}}_{{tod}} and input_number.{{therm}}_heat_{{doy}}_{{tod}} (or, of course, if there is no high/low, then remove the cool/heat part). Yes, I can do this with a script, however...

Next, I need to do the inverse. Originally I was thinking that I would just have a script that would take a look at now() and, for each climate.* determine the appropriate tod and doy, but I think I'll have to go an make automation that trigger based on each input_datetime.* and use that as the {{trigger_to}}, but I still need the same templates to parse out the doy and tod. Again, this is why I want macros, so I don't have to copy-and-paste the same code into multiple places.

You don't have to and isn't related to anything in this PR or issue you are describing. Even this PR adds macros, it doesn't change anything about that.

But it is -- You can't share partial templates across different scripts or automations. If I want to set variables X, Y, and Z in the same way in multiple templates, I have to copy the same template code into each one. Unless there is yet another platform feature that I am missing that let's me build script A that sets variables so that scripts B, C, and D can see and use those variables?

I'm even thinking if we (alternatively) should allow/create a possibility to allow one to plug in custom filters in the future (maybe even defined in YAML itself?). Even that is ambiguous, as automation/script variables can provide that functionality (kinda).

This is (almost) exactly what I'm trying to do with macros! It would allow me to write a macro that computes a value and then let me use that value computation in multiple scripts. It's not quite as useful to me as I would like because I can't set multiple things at once (e.g, I'd like to have one macro that let's me compute the tod, doy, therm, etc in one shot)

[balloob]

Let's not add this feature. It is going to make things a lot more complicated and things are already complicated enough.

Alternatives:

• Use python_script with all your reusable functions
• Write a custom component that adds filters to the template environment:

  from homeassistant.helpers import template
  
  def your_function(…):
      …
  
  tpl = template.Template("", hass)
  tpl._env.globals = your_function

[kaijk]

With template entities the ablity to {% include %} jinja macros or maps would be a big help and vastly reduce code duplication. Here is my example that showed me that HA does not support jinja's include functionality.

template:
  sensor:
    - name: 'Openweather Icon'
      state: |-
        {% include '/config/set_icon.j2' %}
        {{ set_icon( "state_attr('sensor.openweather_report', 'current').weather[0]['id']" ) }}

I don't see a script doing this, and I'm not a python programmer. HA already has a strong commitment to include with !include for yaml