Experimental decorator manager

[dmamelin]

Introduction

⚠️ This PR is not meant to be merged.
Its goal is to discuss the architecture and to try out a new experimental subsystem.

I really appreciate the pyscript project — without it, my home would never have become a notsosmart home. I have many devices, and doing all automation only with GUI and YAML is not realistic.

I am not a Python expert and I will be happy to receive any feedback. There is still a lot of unfinished work, but I decided to pause development and gather external opinions before moving forward.

Motivation

The current trigger system in pyscript is very complex:

• Adding a new trigger requires changes in many different places.
• The code in trigger.py is tightly coupled and mixes the logic of all decorators.
• Debugging or extending functionality is difficult and risky.
• Even simple triggers require an asyncio.Queue abstraction.
• Logic is duplicated between function @decorators and task.wait_until.

High-level overview of the new architecture

• Each @decorator is implemented as an independent class with a clear lifecycle: validate, start, stop
• A central DecoratorManager: validates decorators, starts and stops them, dispatches trigger events
• A single unified implementation for: function @decorators and task.wait_until
• All decorator arguments are validated using voluptuous
• Low coupling:
  • decorators do not depend on each other
  • all interaction goes through the DecoratorManager
• Adding new decorators is straightforward
• The new subsystem can run in parallel with the existing one and is controlled via configuration

Just compare the commit that adds @webhook_trigger to the old system with the standalone file in the new architecture.

Integration with the existing system

To show how easy it is to integrate the new system, this PR includes a minimal integration commit:

→ b84cd43

• the new DecoratorManager can be added without removing the existing trigger system
• switching between the old and new implementations is done via configuration
• user scripts do not need to be changed

This allows the new architecture to be tested and improved without breaking existing behavior.

Tests

24b76bc

In this branch, DecoratorManager is enabled by default only for tests, and all tests pass with minimal changes .

The new code contains several test-specific adjustments:

• @service behaves slightly differently
• error messages were adjusted

This was intentional. The priority was not to change existing tests, in order to avoid unnecessary churn. All such places are marked with FIXME.

The new classes in decorator.py and decorator_abc.py are currently not covered by tests because:

• this is effectively a reverse-engineering of the existing system
• the architecture is still evolving

Tests will be added later.

Compatibility

I am confident that there are still bugs, but I tried to strictly follow both the documentation and the actual behavior of the existing implementation.

My home instance has been running this branch for about a week:

• ~185 devices
• heavy usage of: @state_*, @time_*, @service, @task_unique, task.wait_until

I do not use:@event_trigger, @mqtt_trigger, @webhook_trigger

However, these decorators are relatively simple and should be easy to fix if issues are discovered.

No critical issues have been observed so far.

How to try it

By default, the new system registers decorators with the e(experimental, enhanced) prefix .

@estate_trigger(...)
def test():
    etask.wait_until(state_trigger=...)

Legacy decorators continue to work as before.
Do not mix legacy and new decorators within the same function.

The new subsystem can be enabled globally(without prefix) via config.yaml: just add dm: true.
For tests, DM is enabled by default. To disable it: NODM=1 pytest tests

Technical details

Class Hierarchy

DecoratorManager(ABC)
    FunctionDecoratorManager(DecoratorManager)
    WaitUntilDecoratorManager(DecoratorManager)

Decorator(ABC)
    TriggerDecorator(Decorator, ABC)
        StateTriggerDecorator(TriggerDecorator, ExpressionDecorator, AutoKwargsDecorator)
        MQTTTriggerDecorator(TriggerDecorator, ExpressionDecorator, AutoKwargsDecorator)
        TimeTriggerDecorator(TriggerDecorator)
        EventTriggerDecorator(TriggerDecorator, ExpressionDecorator)
        WebhookTriggerDecorator(TriggerDecorator, ExpressionDecorator, AutoKwargsDecorator)
    TriggerHandlerDecorator(Decorator, ABC)
        StateActiveDecorator(TriggerHandlerDecorator, ExpressionDecorator)
        TimeActiveDecorator(TriggerHandlerDecorator, AutoKwargsDecorator)
    CallHandlerDecorator(Decorator, ABC)
        TaskUniqueDecorator(CallHandlerDecorator, AutoKwargsDecorator)
    CallResultHandlerDecorator(Decorator, ABC)
    ServiceDecorator(Decorator)
    AutoKwargsDecorator(Decorator, ABC)
        StateTriggerDecorator(TriggerDecorator, ExpressionDecorator, AutoKwargsDecorator)
        TaskUniqueDecorator(CallHandlerDecorator, AutoKwargsDecorator)
        MQTTTriggerDecorator(TriggerDecorator, ExpressionDecorator, AutoKwargsDecorator)
        TimeActiveDecorator(TriggerHandlerDecorator, AutoKwargsDecorator)
        WebhookTriggerDecorator(TriggerDecorator, ExpressionDecorator, AutoKwargsDecorator)
    ExpressionDecorator(Decorator, ABC)
        StateActiveDecorator(TriggerHandlerDecorator, ExpressionDecorator)
        StateTriggerDecorator(TriggerDecorator, ExpressionDecorator, AutoKwargsDecorator)
        MQTTTriggerDecorator(TriggerDecorator, ExpressionDecorator, AutoKwargsDecorator)
        EventTriggerDecorator(TriggerDecorator, ExpressionDecorator)
        WebhookTriggerDecorator(TriggerDecorator, ExpressionDecorator, AutoKwargsDecorator)

Trigger flow example (@event_trigger)

→ HA event
→ TriggerDecorator.dispatch
→ FunctionDecoratorManager.dispatch
→ TriggerHandlerDecorator.handle_dispatch
→ Context_creation
→ CallHandlerDecorator.handle_func_call
→ Function_call
→ CallResultHandlerDecorator.handle_func_result

DecoratorManager

Owns all decorators and controls their lifecycle.

FunctionDecoratorManager

Used for function @decorators.

WaitUntilDecoratorManager

Used for task.wait_until. It iterates over all TriggerDecorator instances and, when a trigger is found in the wait_until arguments, automatically resolves all required arguments using Decorator.kwargs_schema.
Timeout handling is simple:
a timeout is implemented by adding time_trigger("once(now + {timeout}s)").

Differences between function calls and task.wait_until are handled inside the triggers themselves.

Decorator

Base class. Handles argument validation using voluptuous.

TriggerDecorator (@*_trigger)

Base class for triggers:

• adds kwargs
• defines dispatch
• marks the decorator as usable in task.wait_until

TriggerHandlerDecorator (@*_active)

Runs immediately during dispatch. Can stop further processing.

CallHandlerDecorator

It is very similar to TriggerHandlerDecorator, but it is invoked immediately before the function call. At this point, a new AstEval instance and the Home Assistant context are already available. It can cancel the function call and is used by @task_unique.

CallResultHandlerDecorator

Runs after the function finishes.
Currently unused, but could be helpful for:

• returning data from @webhook_trigger
• making @service follow the same rules as other decorators
• future decorators with a return value — @sentence_trigger, @telegram_trigger, etc.

ExpressionDecorator

Helper for working with ast-expressions.

AutoKwargsDecorator

Helper for automatic kwargs handling. See @webhook_trigger — local_only and methods are filled in automatically during validate().

Implemented decorators

@event_trigger, @mqtt_trigger, @webhook_trigger

The code was moved and the unnecessary asyncio.Queue layer was removed. The base event.py, mqtt.py, and webhook.py files are not used in the new implementation.

@task_unique

According to the documentation, this is equivalent to task.unique, but I moved over the additional validation logic from trigger.py.
If we strictly followed the documentation, the implementation would be very simple:

async def handle_call(self, data: DispatchData) -> bool:
    await task_unique_func(self.args[0], self.kill_me)
    return True

@service

This decorator is an outlier. It registers the service before the context starts, so DecoratorManager contains some special handling to make the tests pass.

It also does not participate in *_active or @task_unique, which is why its implementation is relatively large and it calls the method directly.

If we decide to make it behave like other decorators and follow the common rules, the implementation could be significantly simplified.

@state_active, @time_trigger, @time_active

The logic was carefully migrated from trigger.py, with utility functions imported from trigger.py.

@state_trigger

Probably the most complex part. I initially migrated the logic from trigger.py, but in the new decorator design the old code felt out of place and overly complicated. I ended up rewriting it completely, and I believe the result is much clearer. Hopefully without bugs 🙂
It still uses State.notify_var_get, State.notify_add, and asyncio.Queue.

@pyscript_compile, @pyscript_executor

Left unchanged.
They differ substantially from other decorators, but there are ideas on how to integrate similar decorators into DM if needed.

Transition plan

0. @craigbarratt will reject the PR. That’s it — plan complete 🙂
1. Discuss the architecture and apply the necessary changes
2. Identify and eliminate all behavioral differences in DM
3. Release a version with DM disabled by default and ask users to test it
4. Collect all bugs and undefined behavior
5. Release a version where DM is enabled by default, with an option for users to disable the new implementation and continue using the old one
6. Perform a major cleanup of trigger.py, eval.py, global_ctx.py, event.py, webhook.py, and mqtt.py
7. Final clean release without legacy code

What I’m looking for

Feedback on the overall architecture.

It’s very possible that I over-engineered parts of the solution or missed a simpler approach.

Any comments or suggestions are very welcome.

[ALERTua]

• "I'm not a Python expert"
• Proceeds implementing deep architecture overhaul that I can not even dream about even in a hundred years

[dmamelin]

• "I'm not a Python expert"
• Proceeds implementing deep architecture overhaul that I can not even dream about even in a hundred years

There’s no conflict here :) I have experience building and maintaining large projects, just in other languages.
Python was mostly used for small personal utilities; Pyscript and Home Assistant are what pushed me to start learning it more seriously.

[craigbarratt]

This is really impressive! Thanks for developing and contributing this. It's a much cleaner and more elegant approach, and will be easier to add features and maintain going forward. I definitely support moving ahead with this. I like the idea of having the old version for a while in case there are regressions, perhaps via a configuration setting. This could be the major new change, and become the default, for 2.x.

I need a bit more time to review the code, but the my initial impressions are that I like the architecture and approach.

One thing that has bothered me is that the pyscript decorators aren't implemented as callable functions. I don't see any benefit of making them callable functions, but it would be more pythonic if they were implemented as such. The current implementation makes this hard to do. Any thoughts on this?

[dmamelin]

One thing that has bothered me is that the pyscript decorators aren't implemented as callable functions. I don't see any benefit of making them callable functions, but it would be more pythonic if they were implemented as such. The current implementation makes this hard to do. Any thoughts on this?

Wow! Great idea. It would simplify the implementation and be more correct.
However, it would definitely break compatibility and the tests: pyscript decorators behave in a specific way, and this behavior is covered by the tests.
I’ll think about it.