[Feat]: Introduce Jinja2-based Instruction Templating with a Feature Flag

[kuanpern]

This issue proposes enhancing the inject_session_state function by introducing Jinja2-based templating engine.

The current regex-based approach is limited, lacking support for control flow like conditionals and loops. A new, dedicated function _render_with_jinja2 will be created to handle Jinja2 rendering.

A feature flag, use_jinja2, will be added to the public inject_session_state function, allowing users to explicitly choose which engine to use during a transition period. This change maintains backward compatibility while enabling more complex and dynamic instruction templates for future development.

Motivation

The existing regex-based templating may be insufficient for advanced use cases, for examples

• Dynamically include or exclude parts of an instruction based on session state.
• Apply formatting to injected variables using filters.

While some of these features could be implemented via callbacks, native Jinja2 support provides a cleaner, more maintainable solution. Jinja2 also provides a robust and well-documented solution for all these needs, making agent instructions more expressive and adaptable.

Proposed Changes

• Refactor inject_session_state: Extract the current logc into a private function _render_with_regex,
  add _render_with_jinja2, A new, private asynchronous function will be implemented to handle Jinja2 rendering. This function will set up a Jinja2 environment and pass session state and artifact accessors to the template.

• Introduce use_jinja2 Flag: The public inject_session_state function will be updated to accept a boolean argument, use_jinja2. If True, it will call _render_with_jinja2; otherwise, it will default to _render_with_regex.

• Update Documentation: The inject_session_state docstrings and the project's contributing guide will be updated to explain the new use_jinja2 flag and the benefits of the Jinja2 templating syntax.

Expected Impact

No breaking changes: Existing codebases that do not use the use_jinja2 flag will continue to function as before.

Testing Plan

Unit Tests: New unit tests will be added to cover the _render_with_jinja2 function, ensuring it correctly handles:

• Basic variable substitution ({{ var_name }}).
• Conditional logic ({% if ... %}).
• For loops ({% for ... %}).
• Asynchronous artifact loading ({{ await artifact(...) }}).