AgentExperience.NET: an AIContextProvider that learns from verified runs, with recall on tool failure (preview.18)

[Fabbrik (fabbrik)]

Hi all. AgentExperience.NET is an open-source memory layer for MAF agents (.NET, with Python and Go ports on the same language-neutral contract). 0.1.0-preview.18 is out. I wanted to share the intent, not only the feature list.

An agent built on Agent Framework already has chat history and, with an AIContextProvider, whatever retrieval the app wires in. What it usually lacks is memory of its own outcomes: this tool call failed like this, and in an earlier run this sequence fixed it and the tests passed. Without that, each run rediscovers the same fixes.

UseAgentExperience(provider) adds capture and injection beside whatever the agent already has, and nothing reads another provider's store. A lesson is stored only after the host's own check passes; the model's claim about its own success is never enough. It does not train anything, does not make the model more capable, and does not replace retrieval for written knowledge.

What's new, and where it touches the framework:

Recall at the moment of failure. Before-run injection is keyed on the request text, which is the wrong key for most lessons. The new opt-in Recall runs inside capture's function-invocation middleware: when a tool call fails (thrown, or classified failed by ClassifyToolResult) with a failure signature that a verified lesson resolved, the next model call in the same run gets a short labelled hint. I evaluated three mechanisms against FunctionInvokingChatClient (1.22.0 and 1.27.0):

• adding a content item to the tool result (rejected: for a thrown call, FICC builds the error result after the middleware rethrows);
• a chat-client layer below function invocation (rejected: needs the host to rebuild its client or turn on EnableMessageInjection with a session);
• a message in context.Messages (rejected: lands between the tool call and its result, which OpenAI-style APIs refuse).

I settled on appending to the run's ChatOptions.Instructions, which MAF clones once per run and FICC carries across iterations. Proven by a compatibility test at both versions. It's system-role text for the rest of the run, which the docs state as a trust trade-off. If anyone from the MAF team sees a cleaner seam for "add context before the next model call, mid-run, without a session," I'd like to hear it.

Failure signatures. Tool name + error class + an ASCII-normalized fingerprint of the first error line, indexed in every store (in-memory, PostgreSQL, SQLite), with a new optional IExperienceSignatureSource port. IExperienceCandidateSource is unchanged.

The recipe. ExperienceCaptureOptions.RecipeResultExcerpt lets the host vouch for a short excerpt of a returned call's result; it renders on the working attempt for allowlisted tools. AgentExperienceOptions.Recipe(tool, arguments, resultExcerpt) sets all consents in one place.

Measured injection floor, shipped off. ExperienceInjectionOptions.MinimumMatch omits weakly matching records before the record limit. I fixed the selection rule before the sweep (FP@8 <= 0.15 on both adapters, no distractor noise); no value qualified, so it's opt-in, with the table in docs/benchmarks.md. Sharing this because "any three terms" matching (which raised recall a lot) has a precision cost that a floor can't buy back without losing long realistic requests.

A built-in holdout. A share of runs gets no lessons, and the verified outcomes of the two groups are compared (helps, hurts, or no evidence yet, per model, with intervals). So far: on a synthetic task lessons help; on realistic tasks there is no overall benefit shown yet, and wrong lessons did worse than none. Reports are under experiments/ in the repo.

Beyond MAF. A dotnet tool, aexp, exposes the same loop as a CLI and MCP server over a local SQLite file; internally it drives a probe ChatClientAgent wired with UseAgentExperience, so the block and hints are exactly what a MAF agent would get. Plus a Claude Code hooks recipe.

Status: preview, Apache-2.0, 0.1.0-preview.18 (NuGet, dotnet tool install --global AgentExperience.Cli --prerelease for the tool). Supported: Microsoft.Agents.AI 1.22.0 and later 1.x, .NET 10, PostgreSQL 15 to 18. https://github.com/fabbrik/AgentExperience.NET

Fabricio Zoff, zoff.tech