Use Agent Docstring and Module Docstring as Agent system prompt by datvo06 · Pull Request #585 · BasisResearch/effectful

datvo06 · 2026-02-23T03:41:45Z

This PR attempts to partially address #567 for agent:

Adds an automated system prompt from the agent by concatenating the module docstring with the agent subclass docstring.
Also added test cases to ensure an invariant that there is only one system prompt in each call to the agent. Haven't added parameterized tests yet as there are no cases that non-agent template can see agent templates.

Some subtleties that might need adjustment:

Do we allow agent prompt to use the lexical context? (currently no)
Should Module code or Agent code appear first in the system prompt (right now: module -> agent).
Agent class in notebook might not have access to module docstring.

eb8680 · 2026-02-23T16:08:29Z

effectful/handlers/llm/completions.py

+        template,
+        "__system_prompt__",
+        textwrap.dedent(f"""
+        SYSTEM: You are a helpful LLM assistant named {template.__name__}.


Should we just remove this hard-coded default given the degradation it seems to have caused in #567 ? We want the system message to be fully modifiable by the user, so falling back to the module docstring in Template.define might be better.

While updating it I realize that this might not work in notebook as notebook would not have module doc. For Agent, it's fine as Template can still use Agent's docstring, but not normal templates, so I used another prompt instead.

eb8680 · 2026-02-23T16:10:58Z

effectful/handlers/llm/template.py

+        """Build an Agent system prompt from module and class docstrings."""
+        parts: list[str] = []
+
+        mod = inspect.getmodule(cls)


This part with the module-level docstring should probably happen in Template.define, so that mod is the module of the Template and this covers non-method Templates as well.

I moved it into Template.define

eb8680 · 2026-02-23T16:13:53Z

effectful/handlers/llm/template.py

+
+        mod = inspect.getmodule(cls)
+        if mod is not None and mod.__doc__:
+            parts.append(inspect.cleandoc(mod.__doc__))


I'm not sure if it makes a difference for modules, but you might want to use inspect.getdoc here instead of mod.__doc__.

I used inspect.getdoc now.

eb8680 · 2026-02-23T16:15:46Z

effectful/handlers/llm/template.py

+        if mod is not None and mod.__doc__:
+            parts.append(inspect.cleandoc(mod.__doc__))
+
+        class_doc = cls.__dict__.get("__doc__")


If you use inspect.getdoc here instead of just looking at the __doc__ attribute, it will automatically go up the inheritance hierarchy to find the first non-empty class docstring. Note that that would in some cases be the docstring of Agent itself, which we may or may not want (or we may want to change the Agent docstring so that it is more amenable to being used this way).

I updated to use inspect.getdoc here as well.

eb8680 · 2026-02-23T16:16:58Z

effectful/handlers/llm/template.py

+        class_doc = cls.__dict__.get("__doc__")
+        if class_doc is None or not inspect.cleandoc(class_doc):
+            raise ValueError(
+                "Agent subclasses must define a non-empty class docstring."
+            )


I'm not sure we want this to be an error. As I wrote in one of my other comments, using inspect.getdoc rather than cls.__doc__ would look at the inheritance hierarchy of cls and get the first non-empty docstring, which would always include Agent as a fallback.

I also updated it, as it has fallback, I dont need to throw ValueError anymore.

eb8680 · 2026-02-23T16:19:40Z

effectful/handlers/llm/template.py

+    __system_prompt__: str
+
+    @classmethod
+    def _build_system_prompt(cls) -> str:


I would just inline this logic into Agent.__init_subclass__ / Template.define so that it's not something that can be overridden in subclasses. There are already two ways of customizing the system prompt (docstrings and handling call_system), having a third is unnecessarily confusing.

I inlined it.

eb8680 · 2026-02-23T16:33:16Z

effectful/handlers/llm/template.py

            prop.__set_name__(cls, "__history__")
            cls.__history__ = prop
+        if not hasattr(cls, "__system_prompt__"):
+            sp = functools.cached_property(


In this PR, __system_prompt__ is actually fixed at the class rather than instance level, despite being an instance-level cached_property. Are there cases where we might want __system_prompt__ to vary across instances of the same Agent class? If so, how?

That's an interesting question. I think we might actually want to have agent of same class but behave slightly differently.
For example, agents Alice and Bob communicating with each other might want to know who they are. We could make Alice and Bob subclass. But that stop us from mass-producing these agents.
We could also move __system_prompt__ to be at init level or a property definable by user.

Bound method Templates already have access to Agent instance-level data in user messages via self, so I'm not sure identifiers like "alice"/"bob" are the best example. It seems to me that something should only go in the system message if it satisfies some or ideally all of the following properties:

Constant and immutable at an Agent class or instance level (so that the system message can't change during the lifetime of an Agent instance)

Naturally associated with an Agent, Template or module rather than a data type or Tool (so that we aren't duplicating structured input/output specifications already handled by Encodable or Tool)

Identifiable mechanically from runtime introspection (so that we aren't making arbitrary internal choices for the system prompt that lead to repeats of Call_system should be more emphasized by Template docstring. #567 )

Large or complicated (so that repeatedly encoding the same information across user messages would be too wasteful)

Unconditionally useful for any call to any method Template (so that the system prompt isn't inlining something that should be gated behind a tool call)

There are a number of things that are not instance-specific that satisfy these properties (e.g. module source code), but the only specifically instance-level things that come to mind immediately are:

Annotated fields that are not explicitly accessed by any Template and are known somehow to be immutable e.g. a tuple[Image, ...] field on a dataclass. More generally I guess you could imagine an Annotation for use on Agent class definitions that asserts that some instance-level field should be appended to the system message of an instance. This seems likely to be especially useful for non-textual data.

Compactified message history - as with ordinary coding agents or chatbots, once an Agent's __history__ gets to be too long for the underlying model, it has to be compressed somehow to continue using it. One way to do this is by having the model summarize it and append the resulting constant-size summary to the system message before clearing __history__.

Both of those are speculative and out of scope for this PR, though.

eb8680 · 2026-02-23T16:56:24Z

Do we allow agent prompt to use the lexical context? (currently no)

I'd say no, since we don't want the system message to change once it's been set. Template methods should already be able to refer to variables in the enclosing scope around the class definition. In #545 I included the source of the module via inspect.getsource(inspect.getmodule(template)), which gives some information about the lexical context, but I think there are too many open design questions around doing that to include it in this more conservative PR.

Should Module code or Agent code appear first in the system prompt (right now: module -> agent).

Module before Agent seems logical. Would you expect this to matter much in practice?

Agent class in notebook might not have access to module docstring.

This seems fine, not all regular modules have docstrings either.

eb8680 · 2026-02-23T17:05:43Z

there are no cases that non-agent template can see agent templates.

I think this should just work as expected with the design in this PR (because Templates shouldn't ever see message histories other than their own or their Agent's) but this seems like an important edge case to include in the tests.

…ompt construction to template

datvo06 · 2026-02-23T22:06:43Z

Should Module code or Agent code appear first in the system prompt (right now: module -> agent).

Module before Agent seems logical. Would you expect this to matter much in practice?

I think it might, but we don't know until we have cases that break because of wrong order, so keeping this until then

datvo06 · 2026-02-23T22:08:07Z

Agent class in notebook might not have access to module docstring.

This seems fine, not all regular modules have docstrings either.

Got it. It's fine for the agent. But I think Template should still have some system prompt if it's in REPL or notebook environment. So I added the fallback in call_system:

"You are a helpful assistant, you need to follow user's instruction"

datvo06 · 2026-02-23T22:11:02Z

I think this should just work as expected with the design in this PR (because Templates shouldn't ever see message histories other than their own or their Agent's) but this seems like an important edge case to include in the tests.

I added test to make sure that this is true.
Update: after adding the test, I realized that the current logic allow Template to collect agent tools in scope as well. So I'm adding a guard condition. Will think again tonight.

eb8680 · 2026-02-23T23:31:40Z

Update: after adding the test, I realized that the current logic allow Template to collect agent tools in scope as well. So I'm adding a guard condition. Will think again tonight.

Maybe I misunderstood, but I interpreted your earlier comment about this to be about histories rather than tools? I don't think we'd want to change the current tool collection behavior to make the new test you added pass.

eb8680 · 2026-02-23T23:33:21Z

effectful/handlers/llm/template.py


            # Collect tools as methods on Agent instances in context
-            elif isinstance(obj, Agent):
+            elif include_agent_instance_tools and isinstance(obj, Agent):


This should not be changed, the existing behavior seems correct to me.

Got it. Reverted!

eb8680 · 2026-02-23T23:36:59Z

tests/test_handlers_llm_template.py

+            return standalone, captured_agent
+
+        standalone, _ = make_template()
+        assert "helper" not in standalone.tools


The only problem I see here is that the name should be "helper_agent.helper" or "captured_agent.helper" rather than "helper". Otherwise I don't think this test should pass, the Agent's methods should be visible to standalone.

Understood. I removed this test.

eb8680 · 2026-02-23T23:38:54Z

tests/test_handlers_llm_template.py

+        assert MissingDocAgent.__doc__ is None
+        prompt = MissingDocAgent().__system_prompt__
+        assert prompt
+        assert "persistent LLM message history" in prompt


This hardcoded substring of the current Agent.__doc__ will cause the test to break immediately upon changes to Agent.__doc__.

eb8680 · 2026-02-23T23:40:42Z

tests/test_handlers_llm_template.py

+        assert_single_system_message_first(mock.received_messages[0])
+        assert (
+            mock.received_messages[0][0]["content"]
+            == "You are a helpful assistant, you need to follow user's instruction"


This hardcoded value will break this test immediately upon any changes to default behavior of call_system.

eb8680 · 2026-02-24T04:20:42Z

tests/test_handlers_llm_template.py

+        assert agent_doc is not None
+        assert prompt == agent_doc
+
+    def test_blank_docstring_uses_inherited_doc(self):


This test is failing, but it's also just a very weak and brittle test and should be deleted entirely. Not all tests are created equal.

Good to merge once this is addressed.

datvo06 added 4 commits February 22, 2026 22:37

Agent prompt and test

4b1e32a

Lint

94079f4

More lint

66d2812

Leave independent template system prompt untouched

6cbee55

datvo06 requested review from eb8680 and kiranandcode February 23, 2026 14:17

eb8680 reviewed Feb 23, 2026

View reviewed changes

eb8680 added the module:llm label Feb 23, 2026

datvo06 added 6 commits February 23, 2026 16:44

Remove the default template that causes regression and move system pr…

9b0c38c

…ompt construction to template

Lint

dee9e04

Add fall back

19dac93

Inlining

142139f

Inline build_system_prompt

614c92b

Lint

4256520

Add test to check that template does not see agent template

7e8c981

eb8680 requested changes Feb 23, 2026

View reviewed changes

datvo06 added 2 commits February 23, 2026 21:27

Reverse template guard

a850c14

Refractoring tests

7571e60

eb8680 mentioned this pull request Feb 24, 2026

What instance-level data should be incorporated into system messages? #586

Open

eb8680 approved these changes Feb 24, 2026

View reviewed changes

eb8680 mentioned this pull request Feb 24, 2026

CI build times are getting too long again #587

Open

eb8680 reviewed Feb 24, 2026

View reviewed changes

Removing brittle test

753b96c

eb8680 merged commit befba34 into master Feb 24, 2026
6 checks passed

eb8680 deleted the dn-agent-systemprompt branch February 24, 2026 05:15

This was referenced Feb 24, 2026

Call_system should be more emphasized by Template docstring. #567

Closed

Agent method Tools should have correct name prefixes #588

Closed

Add system prompt and tools for reading lexical variables #545

Closed

Conversation

datvo06 commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eb8680 Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eb8680 commented Feb 23, 2026

Uh oh!

eb8680 commented Feb 23, 2026

Uh oh!

datvo06 commented Feb 23, 2026

Uh oh!

datvo06 commented Feb 23, 2026

Uh oh!

datvo06 commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eb8680 commented Feb 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

datvo06 commented Feb 23, 2026 •

edited

Loading

eb8680 Feb 23, 2026 •

edited

Loading

datvo06 commented Feb 23, 2026 •

edited

Loading