Pre- and post-hooks for specific cases

[CantLooseToAMoose]

Description

I’m currently working on evaluating my agent implementation.

The agent interacts with an external configuration service through a CRUD-based API.

As part of the evaluation workflow, I want to perform setup steps before executing specific test cases and run teardown routines afterward to revert any state changes introduced by the agent. At the moment, I don’t see a mechanism or hook available for executing custom code before task or after running tasks on individual cases.

References

No response

[dmontagu]

Can you share an example of what you want the code to look like? My gut reaction is that it makes more sense to bake this logic into the task execution and define the task in a way that it can do your cleanup etc., possibly wrapping a task function from your code in a wrapper that does the cleanup etc.

But I could be convinced this is valuable with a better understanding of what the infrastructure/API might look like, and maybe an example showing why it's painful to do in other ways.

[CantLooseToAMoose]

What I did in my code was implement a CaseFactory together with a ConversationTurns builder.

With these, I want to create complete conversations (a list of either ModelRequests or ModelResponses) and generate cases along the way. Essentially, the goal is to simulate multi-turn interactions for evaluation purposes.

The code for building such cases looks something like this:

convo = ConversationTurns()
factory= CaseFactory()

convo.add_user_message("Hey")

case_1_expected_response ="Hello! How can I help you?"

factory.create_conversation_case(
    name="Initializing Conversation",
    conversation_turns=convo,
    expected_final_response=case1_expected_response,
    description="Saying hello",
    metadata={"some" : "metadata"}
)

conva.add_model_message(case_1_expected_response)

convo.add("I want you to create [something] for me")

case_2_expected_response="I did create [something] for you. Do you want me to configure it for your specific needs?"

factory.create_conversation_case(
    name="Create [Something] for user",
    conversation_turns=convo,
    expected_final_response=case2_expected_response,
    expected_final_tool_calls=[
        create_tool_call_part(
            tool_name="create",
            args={
                "arg":"something"
            },
            tool_call_id="some_id"
        )
    ],
    metadata={"some" : "metadata"}
    description="Creating something"
)
convo.add_tool_call(tool_name="create",
            args={
                "arg":"something"
            },
            tool_call_id="some_id"
)
convo.add_tool_response(
    tool_call_id="some_id",
    tool_name="create",
    content="success"
)

convo.add_model_message(case_2_expected_response)

convo.add("Yes, I want you to configure [something] for me")

case_3_expected_response="I did configure [something] for you. Do you want me to create [something_else] for you that uses [something] internally?"

# At this point it is possible that the previous case did not create the [something] and the Model will get lost.
# So being able to call a pre hook that lets me create the [something] first would be really helpful.
factory.create_conversation_case(
    name="Configure [Something] for user",
    conversation_turns=convo,
    expected_final_response=case_3_expected_response,
    expected_final_tool_calls=[
        create_tool_call_part(
            tool_name="configure",
            args={
                "arg":"something",
                "arg2":"something_to_configure"
            },
            tool_call_id="some_id_2"
        )
    ],
    metadata={"some" : "metadata"}
    description="Configuring something"
)
convo.add_tool_call(tool_name="configure",
            args={
                "arg":"something",
                "arg2":"something_to_configure"
            },
            tool_call_id="some_id_2"
)
convo.add_tool_response(
    tool_call_id="some_id_2",
    tool_name="configure",
    content="success"
)

convo.add_model_message(case_3_expected_response)



factory.build_dataset("Dataset Name")

If I were to add hooks directly into the task definitions, I would need a way to reference case names — and I’m not even sure whether that would be feasible. It would also lead to a lot of case-specific handling: depending on the scenario, I might need to ensure that a given [something] exists or doesn’t exist, or that it comes with a specific configuration. In some cases, I might need an additional [something_else] with its own configuration, and so on.

Overall, I don’t think this approach would stay very clean or maintainable.