v0.10.0

Highlights

• Custom (string-id) tools now type-check. register_tool()'d tools carry a plain-string id, and the runtime always accepted them -- but the public annotations named only the built-in enum, forcing casts under pyright/basedpyright. BrainSpec.tools is now tuple[ToolId | str, ...] and ToolSpec.id is ToolId | str (the MCP pool's typing was widened to match), so declaring a custom spec and selecting it from a brain type-checks cleanly.
• Custom tools can be enabled via config. AgentConfig.enabled_tools (and setup_tools/warmup_tools' enabled_tool_names) now accepts a registered custom tool's id alongside the built-in ToolNames, so consumers no longer need a custom BrainSpec just to turn a registered tool on. A name that is neither a configurable ToolName nor a registered tool id raises a clear ValueError.
• Fixed a latent crash on string ids. Error and log paths that did spec.id.value (the MCP "not in the pool" setup error, warmup logging/failure messages, pool shutdown logging) would raise AttributeError on a string id and mask the real error; ids are now formatted enum/str-agnostically via str(spec.id).

No behavior change for built-in tools: ToolId is a StrEnum, so enum and string ids already interoperate as dict keys and in comparisons. No breaking changes.

Install

pip install 'openlily @ git+https://github.com/getlark/openlily.git@v0.10.0#subdirectory=server'

v0.9.0

Highlights

• Tool prompt snippets are now rendered as a named <ToolGuidance> block. Each active tool's guidance is paired with the LLM-visible function name(s) it covers and rendered as <Tool name="...">text</Tool> entries wrapped in one <ToolGuidance> block inside the system prompt's <Tools> section -- so the model knows exactly which functions each snippet is about, instead of reading an anonymous bullet list. Local tools name their direct functions (end_session, web_search/web_fetch, send_email_to_user); MCP tools (browser, Notion, X) pair their snippet with every function their server exposes.
• Breaking: the AgentConfig.system_instruction callable contract changed. A callable override now receives the single pre-rendered <ToolGuidance> block as a str (previously a list[str] of raw snippets), and "" when no tools contribute guidance -- so you can embed it unconditionally. build_system_instruction takes the same pre-rendered block.
• New public API. ToolGuidance and render_tool_guidance are exported from the package root; ToolBundle.instructions now carries ToolGuidance entries instead of plain strings (breaking for custom tools that construct bundles -- see the updated examples/custom_brain_and_tool.py).
• Fixed two typos in the end-session prompt snippet, and synced the drifted openlily.__version__ with pyproject.

Upgrading: if you pass a callable system_instruction or build ToolBundles with prompt snippets, update to the new shapes: the callable receives one pre-rendered str, and bundle snippets become ToolGuidance(tool_names=(...), text=...). String/None system_instruction configs are unaffected.

Install

pip install 'openlily @ git+https://github.com/getlark/openlily.git@v0.9.0#subdirectory=server'

v0.8.0

Highlights

• Spoken fallback for empty user turns with nothing to recover. When VAD detects speech but STT produces no transcript for the turn -- and the context holds no unanswered user message to re-run (the very first turn of a session is the primary case) -- the bot previously stayed completely silent while the working-sound cue looped forever, until the user gave up and the idle timeout ended the session. This hits real users: degraded audio that trips VAD but defeats STT (Bluetooth HFP codec + wind, noisy mics) is common on mobile. Now the bot audibly says it couldn't make out what the user said and asks them to repeat, within a few seconds of the turn ending. The resulting speech naturally stops the working sound and resets the idle timer.
• New AgentConfig.empty_turn_fallback (default True). Set False to disable, or pass a string to override the injected instruction and control the tone of the "could you repeat that?" response.
• Loop-guarded. The fallback never fires while a response is in flight, the existing empty-turn recovery (unanswered question re-run, from v0.6.0) keeps priority, and consecutive fallbacks are capped at 2 -- a noisy mic repeatedly tripping VAD gets at most two apologies, then silence until a turn with real content. A turn where STT returns only whitespace counts as empty.
• Cascade brains only. Realtime (speech-to-speech) services don't implement mid-session context message appends in Pipecat 1.4.0, so the fallback is automatically disabled for realtime brains (logged at build time).

Upgrading: no config changes required -- the fallback is on by default. Pass empty_turn_fallback=False to restore the previous behavior.

Install

pip install 'openlily @ git+https://github.com/getlark/openlily.git@v0.8.0#subdirectory=server'

v0.7.0

Highlights

• Fast barge-in is now the default (cascade brains). v0.6.0 gated turn starts on transcription so VAD false-triggers couldn't cancel an in-flight response — but that made barge-in wait for a finalized STT transcript, which some services only produce after an endpoint, i.e. seconds after the user starts talking over the bot. Cascade brains now use Pipecat's default start pair (VAD or interim transcription), so barge-in fires the moment the VAD does. Empty-turn recovery (from v0.6.0) remains the safety net: a ghost turn that cancels a response automatically re-runs the LLM.
• New AgentConfig.user_turn_strategies. Pass a UserTurnStrategies to take full control of turn start/stop behavior — e.g. UserTurnStrategies(start=[TranscriptionUserTurnStartStrategy()]) restores the v0.6.0 transcription-gated behavior. Cascade brains only: combining it with a realtime (speech-to-speech) brain raises, since Pipecat's external-strategy swap for realtime services only happens when no custom strategies are passed.
• VAD/STT disagreement warning. ConversationLogObserver now logs a WARNING when STT transcribes speech during sustained (>1.5s) VAD silence — the previously invisible signature of VAD thresholds (confidence/min_volume) rejecting a quiet mic's audio, which force-ends turns mid-sentence. One line per silence stretch, greppable, with a pointer to user_vad_params.
• Relaxed default VAD confidence 0.8 → 0.7 (Pipecat's default). The stricter threshold guarded against noise-burst STT hallucinations, but demonstrably rejected real speech from quieter mics; that failure mode is now mitigated by empty-turn recovery and surfaced by the new warning. start_secs=0.3 and min_volume=0.5 are unchanged.

Upgrading: no config changes required. If you relied on the v0.6.0 turn-start behavior, pass user_turn_strategies=UserTurnStrategies(start=[TranscriptionUserTurnStartStrategy()]).

Install

pip install 'openlily @ git+https://github.com/getlark/openlily.git@v0.7.0#subdirectory=server'

v0.6.0

Highlights

• Fixed: the bot could get permanently stuck after a question. A raw VAD false-trigger (breath, cough, background noise) while the bot was thinking would start a user turn and broadcast an interruption, cancelling the in-flight LLM generation. Since no transcript ever arrived, the turn force-ended empty, the LLM never re-ran, and the question sat unanswered while the working-sound blips played forever. Two changes fix this:
  • Transcription-gated turn starts (cascade brains). Turn starts now require an (interim) transcription instead of a raw VAD trigger, so noise that never produces a transcript can no longer start a turn or cancel an in-flight response. Trade-off: barge-in waits for the first interim transcript (a few hundred ms). Realtime brains keep Pipecat's service-driven turn detection, unchanged.
  • Empty-turn recovery. If a user turn still ends with no content while the conversation ends in an unanswered user message (or a tool call the interruption cancelled), the LLM is automatically re-run so the question gets answered. Guarded against loops: no re-run while a response is in flight, and a small cap on consecutive recovery attempts.

No config changes required -- both behaviors are on by default.

Install

pip install 'openlily @ git+https://github.com/getlark/openlily.git@v0.6.0#subdirectory=server'

v0.5.0

Highlights

• Control barge-in via config. New AgentConfig.allow_interruptions (default True, so existing behavior is unchanged). Set it to False to disallow barge-in while the bot is speaking.
  • Implemented with Pipecat's AlwaysUserMuteStrategy: the user's mic is muted while the bot speaks, so captured audio is dropped before it reaches STT (cascade brains) or the realtime LLM (speech-to-speech brains). Neither can be interrupted mid-utterance, and the user is heard again the moment the bot stops.

Usage

config = openlily.AgentConfig(
    brain="cartesia_openai",
    allow_interruptions=False,  # disallow barge-in while the bot speaks
)

Install

pip install 'openlily @ git+https://github.com/getlark/openlily.git@v0.5.0#subdirectory=server'

v0.4.0

Highlights

• Leaner base install. Optional dependencies are now split into extras, so import openlily no longer pulls in PyAudio, LiveKit, wake-word, the email/web tool SDKs, or torch/CUDA. A plain install is just cloud brains, the tool runtime, and the dev-runner/WebRTC transport.
• New extras:
  • local — terminal CLI: mic transport (PyAudio), AEC (LiveKit APM), and openWakeWord.
  • web — Exa-backed web_search / web_fetch tools (exa-py).
  • email — email tool (resend, markdown).
  • all — everything (incl. local-models, unchanged).
• Lazy tool-provider imports. The Exa and Resend providers are imported on demand inside their setup paths, so the tool registry lists specs without importing the SDKs. A missing extra only errors — with an actionable 'install the extra' message — when you actually enable that tool. The local CLI likewise exits with a clear message when the local extra is absent.

Upgrade note

If you run the terminal CLI or use the web/email tools, add the matching extra, e.g.:

pip install 'openlily[local,web,email] @ git+https://github.com/getlark/openlily.git@v0.4.0#subdirectory=server'

Install (base)

pip install 'openlily @ git+https://github.com/getlark/openlily.git@v0.4.0#subdirectory=server'

v0.3.0

Highlights

• Idle timeout now off by default. AgentConfig.idle_timeout_secs defaults to None, so an AgentConfig built without it never idles out. Set a float (e.g. DEFAULT_IDLE_TIMEOUT_SECS) to end the session after that much silence. The idle keep-alive heartbeat is skipped when no timeout is set.
• The CLI is unchanged: IDLE_TIMEOUT_SECS still defaults to 30s when unset, and 0 (or any non-positive value) disables the timeout.

Install

pip install 'openlily @ git+https://github.com/getlark/openlily.git@v0.3.0#subdirectory=server'

v0.2.0

Highlights

• Optional idle timeout. AgentConfig.idle_timeout_secs now accepts None, which disables the idle timeout entirely so a session is never ended on silence. When no timeout is set, the idle keep-alive heartbeat is skipped since it only exists to protect the timeout.
• CLI: IDLE_TIMEOUT_SECS of 0 (or any non-positive value) now disables the idle timeout. Unset still uses the default; invalid values still fall back to the default.

Install

pip install 'openlily @ git+https://github.com/getlark/openlily.git@v0.2.0#subdirectory=server'

openlily 0.1.0

First tagged release of openlily as an installable, reusable Pipecat voice-agent library.

Highlights

• Installable openlily package (src/openlily/) usable in any Pipecat agent and deployable to Pipecat Cloud, alongside the existing local CLI.
• Single AgentConfig injection point plus factory API: create_agent / build_pipeline / build_worker / warmup.
• Swappable brains (OpenAI, Cartesia, OpenAI Realtime, local Whisper/Ollama/Kokoro) and a central tool registry; extend at runtime with register_tool.
• Optional audio cues (working-sound + readiness chime) and an always-on idle keep-alive heartbeat.
• Local run modes preserved: uv run bot.py (local / local-with-wake-word / webrtc).

Install

pip install "openlily @ git+https://github.com/getlark/openlily.git@v0.1.0#subdirectory=server"

The on-device brain needs the optional model runtimes: append [local-models] to the package name. See the README install section for uv and upgrade instructions.