Repository navigation
v0.24.0
Media generation: images, video, speech, transcription, voice cloning and talking-head video,
from a JSON style instead of provider code. Until now toolnexus could carry an image or audio
part but not make one; every provider (a2e, Replicate, fal, OpenAI images/TTS, ComfyUI…) made you
hand-write a request and a polling loop, per endpoint, per language. Now one engine does it:
- Typed verbs for your code —
createMedia(style)givesimage,video,speech,
transcribe,cloneVoiceandtalking, with normalised arguments (prompt,seconds,
voice, …) and a result of{files, text?, jobId?, status}. - Tools for your agent — the same verbs become
generate_image,generate_video,
generate_speech,transcribe_audio,clone_voiceandtalking_video;createToolkit({ media })
registers them. Tool names never change when you switch provider, so prompts and skills keep working.
Media tools are concurrency-safe: two media calls in one response run side by side (up to
maxToolConcurrency) instead of waiting for each other like unlabelled tools. - Long jobs just work — async styles start the job and poll for you (default every 5 s, up to
10 min), report progress, honour cancellation, and keep polling through statuses a provider
never documented. - Providers are data — a style is a JSON file (endpoints, dot-paths,
${ENV}headers, an
optional error envelope).examples/media-styles/a2e.jsonships first, live-verified for image,
video and speech. The library names no vendor. RunResult.files— every image, video or audio a run's tools produced, in order, tagged with
the tool name and call id;toolCalls[].partscarries them per call. The answer text never gets
markup — you decide how to render. An agent with a completion gate keeps the files of every
attempt, exactly as it keeps their tool calls.
It is not media-only: image or file parts returned by MCP tools and by built-in file reads land
inRunResult.filestoo, and each entry keeps the part'snamewhen it had one.- Bounded, safe requests — every request (start and each poll) is bounded by the time left
until the job's deadline (timeoutMs, counted from the start; sync verbs default to 10 min), so
a server that accepts and never answers can no longer hang a call. Style headers (your API key)
are never sent to a different origin thanbaseUrl's, even across a redirect. Redirects follow
one rule set in every port: 301/302/303/307/308 with aLocation, at most 10 hops (then
<verb>: too many redirects), 301/302/303 after a POST become a body-less GET, 307/308 keep the
method and body (Elixir now follows same-origin redirects too instead of refusing them). Error bodies
echoed inHTTP <status>: …are capped at 1000 characters. Python has no separatetimeout
option oncreate_media; the style'stimeoutMsis the one knob.
Every port ships a runnable media example (an image through the typed call, speech through the
generate_speech tool; without A2E_API_KEY it prints a hint and exits), and each language's API
reference page has a Media generation section.
Contract: SPEC §7G, ADR 0036, conformance/media/ (every port replays the same 34 cases and schema set).
Not done yet (tracked in openspec/changes/add-media-generation): downloading expiring result
URLs (a2e's last 3–30 days — save them yourself for now), cancelling the job at the provider
(cancel stops polling; the provider's job keeps running), uploading local files, ComfyUI workflow
injection, webhooks, price quotes, pausing a long job through waitFor, and streaming file
events. Realtime/live voice conversation is deliberately out of scope.
Known redirect gaps: on Clojure's cljgo host the Go HTTP client follows redirects itself, because
koine has no no-redirect option yet: custom style headers reach another origin (and so does
Authorization when only the port differs), and a redirect loop stops after 10 requests instead
of 11 (same too many redirects error); tracked in #130. The JVM build follows the §7G rules. An HTTP client or transport you inject yourself
(Java, C#, Rust, Python, and JS's opts.fetch, which may ignore redirect: "manual") may follow
redirects before the engine sees them, so it must not forward headers cross-origin.
Tool scheduling: reads run together, writes run alone (all eight ports)
The tool calls of one model response no longer all run at once. Before, every call started
immediately with no limit, so two write / edit calls on one file could race and a response
with forty reads opened forty at once. Now each call is safe or unsafe: a tool's own
concurrencySafe(args) predicate decides, otherwise only a tool whose hints say readOnly is
safe. Calls run in emitted order: a run of safe calls overlaps, an unsafe call runs alone after
everything before it and before anything after it. Results and the transcript keep emitted
order, as before. New client option maxToolConcurrency (default 10; env
TOOLNEXUS_MAX_TOOL_CONCURRENCY overrides the default, the option overrides both) caps how many
safe calls run at once. It applies to run and stream, both styles, resume paths and agent runs,
and to the in-process client. Relay tools and the subagent task tool always run concurrently.
SPEC §8 "Tool scheduling"; change openspec/changes/add-tool-concurrency; recipe page
cookbook/tool-scheduling.
- Behaviour change: a tool without hints (most
defineTool/ HTTP tools, MCP tools
whose server sends noreadOnlyHint) is now unsafe and runs alone. Mark it read-only or give
it a predicate to keep it parallel. - rust only, source break:
Toolgained aconcurrency_safefield, so aTool { .. }
struct literal must addconcurrency_safe: None(or use..from another tool).
define_toolcallers are unaffected. - Not done: the cap is per response, not per tool;
todowritekeeps itsreadOnlyhint, so
twotodowritecalls in one response may still overlap (last writer wins, as before). On Clojure, when
the cap is full a new safe call waits for the oldest running call rather than the first to finish
(#128). Only Go's tests are mutation-checked so far (#129).
Background subagents: a documented recipe (all eight ports)
Running a subagent in the background — an id back at once, a callback when it finishes — needs
no new API: spawn returns the id, wake returns before the turn ends, and a wait in a
background task delivers the result. SPEC §7D "Background runs" documents it and each port now
pins it with a test.