Batch inference for LLM calls in bedrock

[edvanr]

Hi there,
I know this is probably not the most common use case for Strands Agends, however I am using strands to process millions of organisational meetings to produce structured output on things like the context of the meeting, it is work-related, desired outcomes, agenda overview...etc

This task can happen in the background so i'm wondering if there is any way i can take advantage of the reduced LLM costs by processing it using Batch Inference in AWS...

https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html

[yonib05]

Hey @edvanr, fun use case! Apologies for leaving this hanging so long.

Strands doesn't support Bedrock batch inference today. BedrockModel drives the real-time Converse APIs, and the agent loop is inherently interactive: the model responds, the SDK runs tools, and the loop continues. Batch inference is a one-request-in, one-response-out job format, so there's no way to run the agent loop through it.

That said, your workload might not need the agent loop for every record. If each meeting is a single-shot extraction (prompt in, structured JSON out, no tool calls), you can get the batch discount by building the Converse request payloads yourself, submitting them as a Bedrock batch job, and processing the outputs outside Strands. You could still use a Strands agent upstream or downstream, for example to triage which meetings need the full interactive treatment.

If you want to stay entirely in Strands, the practical lever is concurrency rather than batch pricing: create one Agent per meeting and fan out with asyncio.gather, using invoke_async(..., structured_output_model=YourModel) if you want typed extraction. One instance per task matters, since a single agent rejects concurrent invocations by default (and shares history besides). That's billed at on-demand rates though.

Native batch support would be a real feature, and I checked, nothing tracks it yet. If it would move the needle for you, please open a feature request so the team can gauge demand and scope it.

[NovaRouteAI]

@edvanr since batch is one-request-in, one-response-out, the pragmatic split is partitioning the workload by latency tolerance: keep the interactive agent loop on the real-time path, and push evals, labeling, and backfill jobs to batch endpoints.

The economics make it worth the plumbing — batch endpoints are commonly priced around half of on-demand, so anything that can wait hours should never pay the interactive rate. One ledger tracking both paths keeps the per-workflow cost honest.

Cost-aware routing between real-time and batch paths is one of the things we've been testing in an OpenAI-compatible multi-model API layer with usage/cost visibility.

What were you planning to batch — evaluation runs, or the main agent workload?