Lightweight Mock Executor for Deterministic Agent Testing

[AlejandroBlanco2001]

Please make sure you read the contribution guide and file issues in the appropriate repository.

Contribution Guide

🔴 Required Information

Please ensure all items in this section are completed to allow for efficient triaging. Requests without complete information may be rejected or deprioritized. If an item is not applicable, please mark it as N/A.

Is your feature request related to a specific problem?

I was building an agent for personal use that is evaluated by executing commands on my machine. Although the Eval framework is quite powerful, I think LLM-as-a-Judge has some inherent limitations when one AI is responsible for evaluating another AI.

Because of that, I started building a small end-to-end testing framework that allows me to send a prompt to an agent, execute the resulting tool calls or commands, and then evaluate the outcome deterministically.

However, implementing this requires understanding several internal concepts of the framework, such as the Runner, sessions, and state management. While this is certainly possible, it introduces unnecessary complexity, especially for developers who are more familiar with building REST APIs than with the internals of the ADK framework.

Describe the Solution You'd Like

It would be helpful to provide a lightweight testing utility (or a PyTest helper) that allows developers to execute an agent programmatically and retrieve the conversation turns and tool calls without having to manually configure the Runner, sessions, or state.

For example:

"""
With this approach, I don't need to worry about setting up the Runner,
sessions, or state. I can simply execute the agent and programmatically
assert the expected behavior.
"""

tools, turns = MockExecutor(
    agent=agent,
    prompt="some_random_message",
    state=some_random_state,
)

This would make it much easier to write deterministic unit and integration tests using plain Python assertions.

Impact on your work

This feature would open up new ways to test agents outside of the Eval framework, making it easier to write deterministic tests for tool execution, conversation flows, and other agent behaviors.

Willingness to contribute

Yes. If this feature aligns with the project's direction, I would be happy to implement it and submit a PR.

---

🟡 Recommended Information

Describe Alternatives You've Considered

Currently, this can be achieved by manually setting up the Runner, creating sessions, managing state, and invoking the agent directly. While this works, it requires understanding several internal APIs and adds boilerplate to every test. A dedicated testing utility would significantly reduce this complexity and make agent testing more accessible.

Proposed API / Implementation

A possible API could look like this:

result = MockExecutor(
    agent=agent,
    prompt="Open Google Chrome",
    state=initial_state,
)

assert result.tools_called == ["execute_command"]
assert result.turns[-1].role == "model"
assert "chrome" in result.tools_called[0].args["command"]

Alternatively, a PyTest fixture could be provided:

def test_agent(mock_executor):
    result = mock_executor.run(
        agent=agent,
        prompt="List all files"
    )

    assert result.tools_called

Additional Context

This feature would complement the existing Eval framework rather than replace it. The Eval framework is excellent for evaluating agent quality, but a lightweight testing utility would make it easier to write deterministic tests focused on execution behavior, similar to how unit tests complement end-to-end tests.