-
Notifications
You must be signed in to change notification settings - Fork 128
Plugin Development
This guide explains how to develop and install third-party plugins for the Whisplay AI Chatbot. The plugin system supports six types: ASR (Speech Recognition), LLM (Large Language Model), TTS (Text-to-Speech), IMAGE_GENERATION (Image Generation), VISION (Image Understanding), and LLM_TOOLS (Function-Calling Tools).
- Architecture Overview
- Plugin Types
- Quick Start
- Plugin Interface Specification
- Installing Third-Party Plugins
- Plugin Development Templates
- Type Reference
- FAQ
┌────────────────────────────────────────────────────────┐
│ Plugin Registry │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ ASR │ │ LLM │ │ TTS │ │LLM-Tools │ │
│ │ Plugins │ │ Plugins │ │ Plugins │ │ Plugins │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
├────────────────────────────────────────────────────────┤
│ Plugin Loader │
│ ┌──────────────┐ ┌────────────────────────┐ │
│ │ Built-in │ │ External │ │
│ │ Plugins │ │ plugins/ directory │ │
│ │ │ │ whisplay-plugin-* npm │ │
│ └──────────────┘ └────────────────────────┘ │
└────────────────────────────────────────────────────────┘
The plugin system is automatically initialized at application startup:
- Registers all built-in plugins (lazy-loaded; unused modules are not loaded)
- Scans the
plugins/directory for local third-party plugins - Scans
node_modulesfor npm packages with thewhisplay-plugin-*prefix - Activates the corresponding plugin based on environment variables (e.g.,
ASR_SERVER=openai) - Injects a
PluginContext(including a snapshot ofprocess.envand host-managed directories likeimageDirandttsDir) into each plugin'sactivate(ctx)call
Important: Plugins should read configuration from the injected
ctx.envrather than accessingprocess.envdirectly. This ensures proper isolation and testability.
| Type | Environment Variable | Description |
|---|---|---|
asr |
ASR_SERVER |
Speech Recognition: audio file → text |
llm |
LLM_SERVER |
Large Language Model: conversation, summarization |
tts |
TTS_SERVER |
Text-to-Speech: text → audio |
image-generation |
IMAGE_GENERATION_SERVER |
Image Generation: text prompt → image |
vision |
VISION_SERVER |
Image Understanding: image → text description |
llm-tools |
(all activated) | Function-Calling Tools: contribute tools to LLM |
The fastest way to create a new plugin is using the CLI:
whisplay plugin createThis will interactively ask for the plugin name, type, and language (TypeScript/JavaScript), then generate a complete project scaffold with the correct interfaces, package.json, tsconfig.json (for TS), .env.template, and .gitignore.
If you prefer to create a plugin manually:
mkdir -p plugins/my-custom-tts// plugins/my-custom-tts/index.js
module.exports = {
name: "my-custom-tts", // Unique identifier, used in .env config
displayName: "My Custom TTS", // Human-readable name
version: "1.0.0", // Semantic version
type: "tts", // Plugin type
description: "My custom text-to-speech plugin",
activate(ctx) {
// Read config from injected ctx.env (NOT process.env)
const apiKey = ctx.env.MY_TTS_API_KEY;
return {
async ttsProcessor(text) {
// Your TTS implementation
const buffer = await myTTSApi.synthesize(text, { apiKey });
const duration = calculateDuration(buffer);
return { buffer, duration };
}
};
}
};# .env
TTS_SERVER=my-custom-ttsnpm run build && npm startEvery plugin must export an object that conforms to the following base structure:
interface PluginBase {
name: string; // Unique identifier, must match the value in .env
displayName: string; // Display name
version: string; // Semantic version (e.g., "1.0.0")
type: PluginType; // "asr" | "llm" | "tts" | "image-generation" | "vision"
description?: string; // Optional description
activate(ctx: PluginContext): Provider | Promise<Provider>; // Activation function, returns a Provider
}
/** Context injected by the host process */
interface PluginContext {
env: Record<string, string | undefined>; // Snapshot of environment variables
imageDir: string; // Host-managed directory for generated images
ttsDir: string; // Host-managed directory for TTS temp/output files
}The ctx.env object is a snapshot of process.env at activation time. Plugins should always read configuration from ctx.env for proper isolation.
For image-related plugins, use ctx.imageDir as the output directory instead of hard-coding paths.
For TTS plugins that need temp/output files, use ctx.ttsDir instead of hard-coding paths.
Formats are defined by plugin metadata (not environment variables):
- ASR plugins should set
audioFormatto declare expected recording input ("wav"or"mp3"). - TTS plugins should set
audioFormatto declare playback decoding format forbase64/bufferoutput ("wav"or"mp3"). If you return afilePath, it only supports "wav". - If metadata is omitted, the system falls back to built-in compatibility defaults.
Purpose: Convert audio files to text.
interface ASRPlugin {
name: string;
displayName: string;
version: string;
type: "asr";
audioFormat?: "wav" | "mp3";
activate(ctx: PluginContext): ASRProvider | Promise<ASRProvider>;
}
interface ASRProvider {
/**
* Recognize speech from an audio file
* @param audioPath - Absolute path to the audio file (WAV format)
* @returns Recognized text, or empty string on failure
*/
recognizeAudio(audioPath: string): Promise<string>;
}Full Example:
// plugins/my-asr/index.js
const fs = require("fs");
const axios = require("axios");
module.exports = {
name: "my-asr",
displayName: "My ASR Service",
version: "1.0.0",
type: "asr",
audioFormat: "wav",
description: "Custom ASR using my API",
activate(ctx) {
const apiKey = ctx.env.MY_ASR_API_KEY;
const apiUrl = ctx.env.MY_ASR_API_URL || "https://api.example.com/asr";
return {
async recognizeAudio(audioPath) {
if (!fs.existsSync(audioPath)) {
console.error("Audio file not found:", audioPath);
return "";
}
try {
const audioBuffer = fs.readFileSync(audioPath);
const response = await axios.post(apiUrl, audioBuffer, {
headers: {
"Content-Type": "audio/wav",
"Authorization": `Bearer ${apiKey}`,
},
});
return response.data.text || "";
} catch (error) {
console.error("ASR recognition failed:", error.message);
return "";
}
}
};
}
};Purpose: Implement streaming conversation and text summarization.
interface LLMPlugin {
name: string;
displayName: string;
version: string;
type: "llm";
activate(ctx: PluginContext): LLMProvider | Promise<LLMProvider>;
}
interface LLMProvider {
/**
* Streaming chat
* @param inputMessages - Array of input messages
* @param partialCallback - Called with each partial text chunk
* @param endCallBack - Called when generation is complete
* @param partialThinkingCallback - Thinking process callback (optional, for chain-of-thought models)
* @param invokeFunctionCallback - Function call callback (optional, for tool invocations)
*/
chatWithLLMStream: (
inputMessages: Message[],
partialCallback: (partialAnswer: string) => void,
endCallBack: () => void,
partialThinkingCallback?: (partialThinking: string) => void,
invokeFunctionCallback?: (functionName: string, result?: string) => void,
) => Promise<any>;
/** Reset conversation history */
resetChatHistory: () => void;
/**
* Text summarization (optional)
* If not provided, the system will return the original text as-is
*/
summaryTextWithLLM?: (text: string, promptPrefix: string) => Promise<string>;
}
/** Message type definition */
interface Message {
role: "system" | "user" | "assistant" | "tool";
content: string;
tool_calls?: FunctionCall[];
tool_call_id?: string;
}Full Example:
// plugins/my-llm/index.js
module.exports = {
name: "my-llm",
displayName: "My LLM Service",
version: "1.0.0",
type: "llm",
activate(ctx) {
const messages = [
{ role: "system", content: ctx.env.SYSTEM_PROMPT || "You are a helpful assistant." }
];
return {
async chatWithLLMStream(inputMessages, partialCallback, endCallback) {
messages.push(...inputMessages);
try {
// Your streaming API call
const stream = await myLLMApi.chat(messages, { stream: true });
let fullResponse = "";
for await (const chunk of stream) {
fullResponse += chunk.text;
partialCallback(fullResponse);
}
messages.push({ role: "assistant", content: fullResponse });
endCallback();
} catch (error) {
console.error("LLM chat failed:", error);
endCallback();
}
},
resetChatHistory() {
messages.length = 1; // Keep system prompt
},
async summaryTextWithLLM(text, promptPrefix) {
const response = await myLLMApi.chat([
{ role: "system", content: promptPrefix },
{ role: "user", content: text }
]);
return response.text;
}
};
}
};Purpose: Convert text to audio.
interface TTSPlugin {
name: string;
displayName: string;
version: string;
type: "tts";
audioFormat?: "wav" | "mp3";
activate(ctx: PluginContext): TTSProvider | Promise<TTSProvider>;
}
interface TTSProvider {
/**
* Text-to-speech synthesis
* @param text - Text to synthesize
* @returns TTSResult object
*/
ttsProcessor(text: string): Promise<TTSResult>;
}
/** TTS return result */
interface TTSResult {
filePath?: string; // Audio file path (one of three)
base64?: string; // Base64-encoded audio data (one of three)
buffer?: Buffer; // Audio Buffer (one of three)
duration: number; // Audio duration in milliseconds
}Full Example:
// plugins/my-tts/index.js
const fs = require("fs");
const path = require("path");
const axios = require("axios");
const mp3Duration = require("mp3-duration");
module.exports = {
name: "my-tts",
displayName: "My TTS Service",
version: "1.0.0",
type: "tts",
audioFormat: "mp3",
activate(ctx) {
const apiKey = ctx.env.MY_TTS_API_KEY;
fs.mkdirSync(ctx.ttsDir, { recursive: true });
return {
async ttsProcessor(text) {
const tempFilePath = path.join(ctx.ttsDir, `my-tts-${Date.now()}.mp3`);
try {
const response = await axios.post(
"https://api.example.com/tts",
{ text, voice: "default" },
{
headers: { Authorization: `Bearer ${apiKey}` },
responseType: "arraybuffer",
}
);
fs.writeFileSync(tempFilePath, Buffer.from(response.data));
const buffer = fs.readFileSync(tempFilePath);
const duration = await mp3Duration(buffer);
return {
buffer,
duration: duration * 1000, // Convert to milliseconds
};
} catch (error) {
console.error("TTS synthesis failed:", error.message);
return { duration: 0 };
} finally {
if (fs.existsSync(tempFilePath)) {
fs.unlinkSync(tempFilePath);
}
}
}
};
}
};Purpose: Provide image generation capabilities through the LLM tool-calling mechanism.
interface ImageGenerationPlugin {
name: string;
displayName: string;
version: string;
type: "image-generation";
activate(ctx: PluginContext): ImageGenerationProvider | Promise<ImageGenerationProvider>;
}
interface ImageGenerationProvider {
/**
* Add image generation tools to the tool list
* @param tools - LLM tool array; the plugin should push tool definitions into it
*/
addImageGenerationTools(tools: LLMTool[]): void;
}
/** LLM tool definition */
interface LLMTool {
type: "function";
function: {
name: string;
description: string;
parameters: {
type?: string;
properties?: Record<string, any>;
required?: string[];
};
};
func: (params: any) => Promise<string>;
}Full Example:
// plugins/my-image-gen/index.js
const fs = require("fs");
const path = require("path");
module.exports = {
name: "my-image-gen",
displayName: "My Image Generator",
version: "1.0.0",
type: "image-generation",
activate(ctx) {
return {
addImageGenerationTools(tools) {
tools.push({
type: "function",
function: {
name: "generateImage",
description: "Generate an image from a text prompt",
parameters: {
type: "object",
properties: {
prompt: {
type: "string",
description: "The text prompt to generate the image from",
},
},
required: ["prompt"],
},
},
func: async (params) => {
try {
const imageBuffer = await myImageApi.generate(params.prompt);
const fileName = `generated-${Date.now()}.png`;
const imagePath = path.join(ctx.imageDir, fileName);
fs.writeFileSync(imagePath, imageBuffer);
return "[success]Image generated successfully.";
} catch (error) {
return `[error]Image generation failed: ${error.message}`;
}
},
});
}
};
}
};Purpose: Provide image understanding capabilities through the LLM tool-calling mechanism.
interface VisionPlugin {
name: string;
displayName: string;
version: string;
type: "vision";
activate(ctx: PluginContext): VisionProvider | Promise<VisionProvider>;
}
interface VisionProvider {
/**
* Add vision analysis tools to the tool list
* @param tools - LLM tool array; the plugin should push tool definitions into it
*/
addVisionTools(tools: LLMTool[]): void;
}Full Example:
// plugins/my-vision/index.js
module.exports = {
name: "my-vision",
displayName: "My Vision Analyzer",
version: "1.0.0",
type: "vision",
activate(ctx) {
return {
addVisionTools(tools) {
tools.push({
type: "function",
function: {
name: "analyzeImage",
description: "Analyze and describe the content of an image",
parameters: {
type: "object",
properties: {
imagePath: {
type: "string",
description: "Path to the image file",
},
question: {
type: "string",
description: "Question about the image",
},
},
required: ["imagePath"],
},
},
func: async (params) => {
const description = await myVisionApi.analyze(
params.imagePath,
params.question
);
return `[response]${description}`;
},
});
}
};
}
};Purpose: Contribute custom function-calling tools to the LLM. Unlike other plugin types where only one plugin is active at a time, all registered llm-tools plugins are activated simultaneously, and their tools are merged into the LLM tool list.
No environment variable required. All
llm-toolsplugins are activated automatically on startup.
interface LLMToolsPlugin {
name: string;
displayName: string;
version: string;
type: "llm-tools";
activate(ctx: PluginContext): LLMToolsProvider | Promise<LLMToolsProvider>;
}
interface LLMToolsProvider {
/** Return the tool definitions this plugin contributes */
getTools(): LLMTool[];
}
/** LLM tool definition */
interface LLMTool {
type: "function";
function: {
name: string;
description: string;
parameters: {
type?: string;
properties?: Record<string, any>;
required?: string[];
};
};
func: (params: any) => Promise<string>;
}Full Example:
// plugins/smart-home/index.js
const net = require("net");
module.exports = {
name: "smart-home",
displayName: "Smart Home Tools",
version: "1.0.0",
type: "llm-tools",
description: "Smart home control tools for lights and switches",
activate(ctx) {
const host = ctx.env.SMART_HOME_HOST || "192.168.1.100";
const port = parseInt(ctx.env.SMART_HOME_PORT || "8888");
return {
getTools() {
return [
{
type: "function",
function: {
name: "switchLight",
description: "Switch the light on or off",
parameters: {
type: "object",
properties: {
action: {
type: "string",
description: "Action to perform on the light",
enum: ["start", "stop"],
},
},
required: ["action"],
},
},
func: async (params) => {
if (params.action !== "start" && params.action !== "stop") {
return "[error]Invalid action. Please specify 'start' or 'stop'.";
}
return new Promise((resolve) => {
const client = new net.Socket();
client.connect(port, host, () => {
client.write(
JSON.stringify({ action: params.action, effect: "rainbow" })
);
client.end();
resolve(`[success]Light switched ${params.action}`);
});
client.on("error", (err) => {
resolve(`[error]Failed to switch light: ${err.message}`);
});
});
},
},
{
type: "function",
function: {
name: "getTemperature",
description: "Get the current room temperature",
parameters: {},
},
func: async () => {
// ...your implementation
return "[response]The current room temperature is 23°C.";
},
},
];
}
};
}
};Use the whisplay CLI to install a plugin directly from a GitHub repository:
# Install a plugin
whisplay plugin install https://github.com/user/whisplay-plugin-azure-tts.git
# List installed plugins
whisplay plugin list
# Update a plugin
whisplay plugin update whisplay-plugin-azure-tts
# Update all plugins
whisplay plugin update --all
# Remove a plugin
whisplay plugin remove whisplay-plugin-azure-ttsThe CLI will clone the repo into plugins/, automatically install dependencies, and run npm run build if a build script is defined.
Note: The
whisplayCLI is installed automatically duringinstall_dependencies.sh. You can also run it directly withbin/whisplayfrom the project root.
Place the plugin in the plugins/ folder at the project root:
whisplay-ai-chatbot/
├── plugins/
│ ├── my-custom-asr/
│ │ ├── index.js # Entry file
│ │ └── package.json # Optional
│ └── my-custom-tts/
│ ├── index.js
│ └── package.json
├── src/
└── ...
Each subdirectory is a plugin. The system automatically loads all subdirectories under plugins/.
Automatic Dependency Installation: If a plugin directory contains a
package.json, the system will automatically runnpm install --productionbefore loading the plugin (skipped ifnode_modulesalready exists and is up-to-date). This means plugins can declare their own dependencies in theirpackage.jsonand they will be installed automatically on first launch.
Each plugin can include its own .env file in its directory for plugin-specific configuration. These variables are scoped to the plugin and never pollute process.env or leak to other plugins.
plugins/
└── my-custom-tts/
├── index.js
├── package.json
└── .env # Plugin-scoped environment variables
Example plugin .env:
MY_TTS_API_KEY=sk-xxxx
MY_TTS_ENDPOINT=https://api.example.com/v1/tts
MY_TTS_VOICE=en-US-Standard-AThe scoped variables are accessible inside the plugin's activate() function via the context object:
module.exports = {
name: "my-custom-tts",
type: "tts",
// ...
activate(ctx) {
// ctx.pluginEnv — ONLY this plugin's .env variables
const apiKey = ctx.pluginEnv.MY_TTS_API_KEY;
// ctx.env — merged: global process.env + this plugin's .env
// (plugin vars override global vars with the same key)
const endpoint = ctx.env.MY_TTS_ENDPOINT;
return { /* provider implementation */ };
},
};| Property | Description |
|---|---|
ctx.pluginEnv |
Only variables from this plugin's .env file |
ctx.env |
Global process.env merged with the plugin's .env (plugin vars take precedence) |
Security Note: Plugin
.envvariables are isolated. Plugin A cannot read Plugin B's.envvariables throughctx.pluginEnvorctx.env.
Example plugin package.json:
{
"name": "my-custom-tts",
"version": "1.0.0",
"main": "index.js",
"dependencies": {
"axios": "^1.6.0",
"mp3-duration": "^1.1.0"
}
}Install npm packages with the whisplay-plugin- prefix:
npm install whisplay-plugin-azure-ttsThe system automatically discovers and loads all packages with the whisplay-plugin-* prefix.
Third-party plugins can use the same name as a built-in plugin to override the built-in implementation:
// plugins/better-openai-tts/index.js
module.exports = {
name: "openai", // Same name as the built-in OpenAI TTS
displayName: "Better OpenAI TTS",
version: "2.0.0",
type: "tts",
activate(ctx) {
// Your improved implementation
return { ttsProcessor: myBetterTTS };
}
};Note: Third-party plugins are loaded after built-in plugins, so they will override built-in plugins with the same name.
If you prefer developing plugins with TypeScript, you need to compile to JavaScript first:
my-plugin/
├── src/
│ └── index.ts
├── dist/
│ └── index.js ← Compiled output, used as plugin entry
├── package.json
└── tsconfig.json
package.json:
{
"name": "whisplay-plugin-my-service",
"version": "1.0.0",
"main": "dist/index.js",
"scripts": {
"build": "tsc"
}
}src/index.ts:
import type {
TTSPlugin,
TTSProvider,
TTSResult,
PluginContext,
} from "whisplay-ai-chatbot/dist/plugin/types";
const plugin: TTSPlugin = {
name: "my-service",
displayName: "My Service TTS",
version: "1.0.0",
type: "tts",
description: "Custom TTS implementation",
activate(ctx: PluginContext): TTSProvider {
// Read config from ctx.env
const apiKey = ctx.env.MY_SERVICE_API_KEY;
return {
async ttsProcessor(text: string): Promise<TTSResult> {
// Implementation...
return { buffer: Buffer.alloc(0), duration: 0 };
},
};
},
};
export default plugin;The activate() function supports returning a Promise, suitable for scenarios that require async initialization (e.g., connecting to a database, loading a model):
module.exports = {
name: "my-local-asr",
displayName: "My Local ASR",
version: "1.0.0",
type: "asr",
async activate(ctx) {
// Read config from ctx.env
const modelPath = ctx.env.MY_ASR_MODEL_PATH || "./models/asr-model.bin";
// Asynchronously load the model
const model = await loadASRModel(modelPath);
return {
async recognizeAudio(audioPath) {
return model.transcribe(audioPath);
}
};
}
};Note: Async
activate()plugins cannot be activated viaactivatePluginSync(). The system uses synchronous activation by default, so ensure your plugin works in synchronous mode, or coordinate with the project maintainers to use async activation.
All plugin-related TypeScript type definitions are located in src/plugin/types.ts. Key types:
| Type | Description |
|---|---|
PluginType |
"asr" | "llm" | "tts" | "image-generation" | "vision" | "llm-tools" |
AudioFormat |
"wav" | "mp3" |
PluginContext |
Context object injected into activate(ctx) containing env, imageDir, and ttsDir
|
PluginBase |
Base plugin interface (name, displayName, version, type) |
ASRPlugin / ASRProvider
|
ASR plugin and provider interfaces (audioFormat metadata) |
LLMPlugin / LLMProvider
|
LLM plugin and provider interfaces |
TTSPlugin / TTSProvider
|
TTS plugin and provider interfaces (audioFormat metadata) |
ImageGenerationPlugin / ImageGenerationProvider
|
Image generation plugin and provider interfaces |
VisionPlugin / VisionProvider
|
Vision plugin and provider interfaces |
LLMToolsPlugin / LLMToolsProvider
|
LLM tools plugin and provider interfaces |
Message |
LLM conversation message type |
LLMTool |
LLM tool definition type |
TTSResult |
TTS return result type |
ToolReturnTag |
Tool return tag enum (Success / Error / Response) |
In IMAGE_GENERATION, VISION, and LLM_TOOLS plugins, tool function return values use special prefix tags:
-
[success]— Operation succeeded -
[error]— Operation failed -
[response]— Used directly as assistant reply content
The name is the unique identifier within its plugin type and must exactly match (lowercase) the corresponding environment variable value in the .env file. For example, TTS_SERVER=my-custom-tts corresponds to name: "my-custom-tts".
You can inspect via the plugin registry API:
import { pluginRegistry } from "./plugin";
// List all plugins
console.log(pluginRegistry.listPlugins());
// List plugins of a specific type
console.log(pluginRegistry.getPluginsOfType("tts"));- Built-in plugins are registered first
- Plugins in the
plugins/directory are loaded in alphabetical order by folder name -
whisplay-plugin-*npm packages are loaded in alphabetical order by package name - Later-loaded plugins with the same name override earlier ones
Yes. During activate(ctx), plugins should read environment variables from ctx.env (injected by the host process) and can use require() to load any Node.js module. Avoid accessing process.env directly — use ctx.env instead for proper isolation.
Add logging in activate(ctx), then run the application and check the console:
activate(ctx) {
console.log("[MyPlugin] Initializing...");
console.log("[MyPlugin] API URL:", ctx.env.MY_PLUGIN_API_URL);
// ...
}