Skip to content

v0.24.0

Choose a tag to compare

@muthuishere muthuishere released this 03 Nov 20:22
· 486 commits to main since this release
8178d09

Media generation: images, video, speech, transcription, voice cloning and talking-head video,
from a JSON style instead of provider code.
Until now toolnexus could carry an image or audio
part but not make one; every provider (a2e, Replicate, fal, OpenAI images/TTS, ComfyUI…) made you
hand-write a request and a polling loop, per endpoint, per language. Now one engine does it:

  • Typed verbs for your code — createMedia(style) gives image, video, speech,
    transcribe, cloneVoice and talking, with normalised arguments (prompt, seconds,
    voice, …) and a result of {files, text?, jobId?, status}.
  • Tools for your agent — the same verbs become generate_image, generate_video,
    generate_speech, transcribe_audio, clone_voice and talking_video; createToolkit({ media })
    registers them. Tool names never change when you switch provider, so prompts and skills keep working.
    Media tools are concurrency-safe: two media calls in one response run side by side (up to
    maxToolConcurrency) instead of waiting for each other like unlabelled tools.
  • Long jobs just work — async styles start the job and poll for you (default every 5 s, up to
    10 min), report progress, honour cancellation, and keep polling through statuses a provider
    never documented.
  • Providers are data — a style is a JSON file (endpoints, dot-paths, ${ENV} headers, an
    optional error envelope). examples/media-styles/a2e.json ships first, live-verified for image,
    video and speech. The library names no vendor.
  • RunResult.files — every image, video or audio a run's tools produced, in order, tagged with
    the tool name and call id; toolCalls[].parts carries them per call. The answer text never gets
    markup — you decide how to render. An agent with a completion gate keeps the files of every
    attempt, exactly as it keeps their tool calls.
    It is not media-only: image or file parts returned by MCP tools and by built-in file reads land
    in RunResult.files too, and each entry keeps the part's name when it had one.
  • Bounded, safe requests — every request (start and each poll) is bounded by the time left
    until the job's deadline (timeoutMs, counted from the start; sync verbs default to 10 min), so
    a server that accepts and never answers can no longer hang a call. Style headers (your API key)
    are never sent to a different origin than baseUrl's, even across a redirect. Redirects follow
    one rule set in every port: 301/302/303/307/308 with a Location, at most 10 hops (then
    <verb>: too many redirects), 301/302/303 after a POST become a body-less GET, 307/308 keep the
    method and body (Elixir now follows same-origin redirects too instead of refusing them). Error bodies
    echoed in HTTP <status>: … are capped at 1000 characters. Python has no separate timeout
    option on create_media; the style's timeoutMs is the one knob.

Every port ships a runnable media example (an image through the typed call, speech through the
generate_speech tool; without A2E_API_KEY it prints a hint and exits), and each language's API
reference page has a Media generation section.

Contract: SPEC §7G, ADR 0036, conformance/media/ (every port replays the same 34 cases and schema set).

Not done yet (tracked in openspec/changes/add-media-generation): downloading expiring result
URLs (a2e's last 3–30 days — save them yourself for now), cancelling the job at the provider
(cancel stops polling; the provider's job keeps running), uploading local files, ComfyUI workflow
injection, webhooks, price quotes, pausing a long job through waitFor, and streaming file
events. Realtime/live voice conversation is deliberately out of scope.

Known redirect gaps: on Clojure's cljgo host the Go HTTP client follows redirects itself, because
koine has no no-redirect option yet: custom style headers reach another origin (and so does
Authorization when only the port differs), and a redirect loop stops after 10 requests instead
of 11 (same too many redirects error); tracked in #130. The JVM build follows the §7G rules. An HTTP client or transport you inject yourself
(Java, C#, Rust, Python, and JS's opts.fetch, which may ignore redirect: "manual") may follow
redirects before the engine sees them, so it must not forward headers cross-origin.

Tool scheduling: reads run together, writes run alone (all eight ports)

The tool calls of one model response no longer all run at once. Before, every call started
immediately with no limit, so two write / edit calls on one file could race and a response
with forty reads opened forty at once. Now each call is safe or unsafe: a tool's own
concurrencySafe(args) predicate decides, otherwise only a tool whose hints say readOnly is
safe. Calls run in emitted order: a run of safe calls overlaps, an unsafe call runs alone after
everything before it and before anything after it. Results and the transcript keep emitted
order, as before. New client option maxToolConcurrency (default 10; env
TOOLNEXUS_MAX_TOOL_CONCURRENCY overrides the default, the option overrides both) caps how many
safe calls run at once. It applies to run and stream, both styles, resume paths and agent runs,
and to the in-process client. Relay tools and the subagent task tool always run concurrently.
SPEC §8 "Tool scheduling"; change openspec/changes/add-tool-concurrency; recipe page
cookbook/tool-scheduling.

  • Behaviour change: a tool without hints (most defineTool / HTTP tools, MCP tools
    whose server sends no readOnlyHint) is now unsafe and runs alone. Mark it read-only or give
    it a predicate to keep it parallel.
  • rust only, source break: Tool gained a concurrency_safe field, so a Tool { .. }
    struct literal must add concurrency_safe: None (or use .. from another tool).
    define_tool callers are unaffected.
  • Not done: the cap is per response, not per tool; todowrite keeps its readOnly hint, so
    two todowrite calls in one response may still overlap (last writer wins, as before). On Clojure, when
    the cap is full a new safe call waits for the oldest running call rather than the first to finish
    (#128). Only Go's tests are mutation-checked so far (#129).

Background subagents: a documented recipe (all eight ports)

Running a subagent in the background — an id back at once, a callback when it finishes — needs
no new API: spawn returns the id, wake returns before the turn ends, and a wait in a
background task delivers the result. SPEC §7D "Background runs" documents it and each port now
pins it with a test.