Putting a dollar cap on one Semantic Kernel agent run (Python)

[domondi1]

With auto function calling, one Semantic Kernel invocation can loop through many model calls, and the cost of a single request isn't bounded. Here's one way to give each run a hard dollar budget without changing your kernel or plugins.

Give the chat service an AsyncOpenAI client that carries the run's id and budget as headers, pointed at a small gateway running in the same process:

import inferrail
base_url = inferrail.start()  # gateway on a background thread, no second process

from openai import AsyncOpenAI
from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion

def service_for_run(run_id: str, budget_usd: str) -> OpenAIChatCompletion:
    return OpenAIChatCompletion(
        ai_model_id="gpt-4o-mini",
        async_client=AsyncOpenAI(base_url=base_url, api_key="unused", default_headers={
            "X-Inferrail-Attribute-Work-Id": run_id,
            "X-Inferrail-Budget-Usd": budget_usd}))

Each call reserves its worst-case cost before it's sent, so parallel function calls in one run can't overspend together. A call that doesn't fit is refused with a 402 before the provider and shows up as ServiceResponseException. inferrail work <run id> shows what the run cost.

Caveats: Chat Completions only, set max tokens in your execution settings, and only calls through that service count. Tested with semantic-kernel 1.44.1 and inferrail 0.4.10 or later (earlier versions conflicted with SK's mcp pin).

Full setup: guide. I maintain Inferrail (open source, Apache-2.0), so take the recommendation with that in mind. Curious how others bound the cost of a single run here.

[domondi1]

The same works from .NET (tested with Semantic Kernel 1.80.1 on .NET 10). Run the gateway as a separate process (pip install inferrail, then inferrail serve --quickstart --app-mode, which listens on http://127.0.0.1:8000) and give SK an OpenAIClient whose HTTP transport adds the run's headers:

var http = new HttpClient(new RunBudgetHandler { InnerHandler = new HttpClientHandler() });
var openai = new OpenAIClient(new ApiKeyCredential("unused"), new OpenAIClientOptions
{
    Endpoint = new Uri("http://127.0.0.1:8000/v1"),
    Transport = new HttpClientPipelineTransport(http),
});
builder.AddOpenAIChatCompletion("gpt-4o-mini", openai);

RunBudgetHandler is a small DelegatingHandler that reads the current run's id and budget from an AsyncLocal and sets X-Inferrail-Attribute-Work-Id and X-Inferrail-Budget-Usd. So parallel InvokePromptAsync calls inside using (RunBudget.Begin("run-42", "0.50")) { ... } share one budget. Set MaxTokens in OpenAIPromptExecutionSettings. A refused call surfaces as HttpOperationException with StatusCode 402, and inferrail work run-42 shows the run's cost.

The full handler and scope code is in dotnet/extensions#7802.