Skip to content

Extensions Overview

Jump to bottom
rookiemann edited this page Mar 15, 2026 · 1 revision

Part 6: Extensions — TTS, Music, and Image Generation

Hermes supports three powerful extension modules that add local AI generation capabilities. Each extension is a separate server that runs alongside Hermes.

How Extensions Work

Each extension is:

  • Portable — nothing is installed system-wide, everything stays in the extensions/ folder
  • Self-installing — downloads its own Python environment, dependencies, and model weights
  • GPU-accelerated — uses your NVIDIA GPU for fast generation
  • Server-based — runs as a local web server that Hermes talks to via tools

Managing Extensions

Go to Tools > Extensions in the menu bar to:

  • See which extensions are installed
  • Install new extensions
  • Start/stop extension servers
  • Check server status

Extension 1: TTS Server (Text-to-Speech)

The TTS Server converts text to natural-sounding speech. It supports 10 different AI voice models, voice cloning, multi-language support, and emotional expression.

Server port: 8200 (configurable via TTS_SERVER_URL environment variable)

Installation

  1. Go to Tools > Extensions
  2. Click Install next to "Text-to-Speech Server"
  3. Wait for the installation to complete (downloads ~2-5 GB depending on models)
  4. Click Start to launch the server

Or install manually:

cd extensions/tts-server
install.bat
launcher.bat api

Available TTS Models

Model Size Best For Voice Cloning Languages GPU VRAM
kokoro 82M Fast English TTS, 54 built-in voices No (preset voices only) English ~1 GB
xtts 500M General purpose, best multilingual Yes (from WAV/MP3 reference) 17 languages ~2 GB
dia 1.6B Two-speaker dialogue with emotions No (uses [S1]/[S2] tags) English ~4 GB
bark 1B Expressive speech with laughter, music, non-verbal sounds No (preset voices) Multilingual ~3 GB
fish 500M Fast voice cloning with minimal reference audio Yes (few seconds of audio) Multilingual ~2 GB
chatterbox 500M Conversational with emotion control Yes English ~2 GB
f5 300M Diffusion-based high-quality cloning Yes Multilingual ~2 GB
qwen 7B Chinese + English bilingual Yes Chinese, English ~8 GB
vibevoice 1.5B Wide emotional range Yes English ~4 GB
higgs 3B Works on CPU (no GPU needed) Yes English CPU or ~3 GB

How to Use TTS

Simple text-to-speech: Say: "Read this text aloud: Hello, welcome to Hermes Agent!" Or: "Generate speech saying 'The quick brown fox jumps over the lazy dog' using Kokoro"

With a specific voice: Say: "Generate speech with the af_heart voice: I love this application"

Dialogue with Dia model: Say: "Generate a dialogue: [S1] Hey, how's it going? [S2] (laughs) I'm doing great! [S1] That's wonderful to hear."

The [S1] and [S2] tags mark different speakers. Dia automatically uses different voices for each speaker and can express emotions in parentheses.

Voice cloning (xtts, fish, f5):

  1. Place a reference audio file (WAV or MP3, 5-30 seconds of clear speech) in: 
    extensions/tts-server/voices/
  2. Say: "Generate speech using xtts with voice 'my_voice.wav': Hello, this is my cloned voice"

TTS Tools Reference

Tool How to Use It
tts_server_status "Is the TTS server running?"
tts_server_generate "Generate speech: [your text]" with optional model and voice
tts_server_models "What TTS models are available?"
tts_server_model_load "Load the xtts model"
tts_server_model_unload "Unload the TTS model to free GPU memory"
tts_server_voices "List available voices for Kokoro"
tts_server_jobs "Show TTS generation jobs"

TTS Troubleshooting

"TTS server unreachable":

  • Make sure the TTS server is running (check Tools > Extensions)
  • Start it with launcher.bat api in the extensions/tts-server/ folder
  • Check that the port (8200) isn't being used by another application

"Model not loaded":

  • The server needs to load a model before generating speech
  • Say "Load the kokoro model" or use the tts_server_model_load tool
  • First-time loading downloads model weights (~1-8 GB depending on model)

"Out of GPU memory":

  • Unload other models (LM Studio models, other TTS models)
  • Use a smaller model (kokoro is smallest at 82M)
  • Use higgs if you want to run on CPU without GPU

Audio sounds robotic/bad:

  • Try a different model — each model has different strengths
  • For English, kokoro and xtts produce the best quality
  • For emotional expression, dia or bark are best
  • Make sure the text is clear and properly punctuated

Extension 2: Music Server

The Music Server generates music, songs, and sound effects from text descriptions. Describe what you want to hear, and it creates an audio file.

Server port: 9150

Installation

  1. Go to Tools > Extensions
  2. Click Install next to "Music Generation Server"
  3. Wait for the installation to complete
  4. Click Start to launch the server

Or install manually:

cd extensions/music-server
install.bat
launcher.bat api

Available Music Models

Model Best For Duration GPU VRAM
musicgen General music from text prompts 5-30 sec ~4 GB
musicgen-large Higher quality music 5-30 sec ~8 GB
stable_audio High-quality, longer music 5-90 sec ~6 GB
ace_step Fast music with style control 5-60 sec ~4 GB
riffusion Spectrogram-based (unique sound) 5-15 sec ~4 GB
audioldm2 Music + sound effects 5-30 sec ~6 GB

How to Generate Music

Simple prompt: Say: "Generate 30 seconds of upbeat electronic dance music with synths and a driving beat"

Specific genre: Say: "Generate a calm acoustic guitar melody with light percussion, 20 seconds"

Sound effects: Say: "Generate the sound of rain falling on a tin roof with distant thunder"

With model selection: Say: "Generate music using stable_audio: epic orchestral film soundtrack, 60 seconds"

Music Tools Reference

Tool How to Use It
music_status "Is the music server running?"
music_generate "Generate music: [description]" with optional model and duration
music_models "What music models are available?"
music_model_load "Load the musicgen model"
music_model_unload "Unload the music model"
music_outputs "Show my generated music files"
music_install "Install the stable_audio model"

Music Tips

  • Be descriptive — "upbeat jazz with saxophone solo and walking bass" works better than "jazz music"
  • Specify instruments — mention specific instruments for better results
  • Set duration — start with 10-15 seconds to test, then increase
  • Try different models — each model has a different sound character
  • Generated audio files are saved in extensions/music-server/output/

Music Troubleshooting

"Music server unreachable":

  • Start the server with launcher.bat api in extensions/music-server/
  • Check that port 9150 isn't in use

"No music model loaded":

  • Say "Load the musicgen model" before generating
  • First-time loading downloads model weights

Generated music sounds bad:

  • Try a more specific prompt
  • Try a different model
  • Some models work better for certain genres

Extension 3: ComfyUI (Image Generation)

ComfyUI is a powerful image generation system based on Stable Diffusion. It can create images, edit photos, generate video frames, and more using a visual workflow editor.

Management API port: 5000 ComfyUI instance ports: 8188+

Installation

  1. Go to Tools > Extensions
  2. Click Install next to "ComfyUI Image Generator"
  3. Wait for the installation to complete (downloads ~3-5 GB for base install)
  4. Click Start to launch the management server

Or install manually:

cd extensions/comfyui
install.bat
launcher.bat

How ComfyUI Works

ComfyUI has two layers:

  1. Management Server (port 5000) — manages instances, models, and custom nodes
  2. ComfyUI Instances (port 8188+) — the actual image generation servers

Each GPU can run its own instance. If you have two GPUs, you can have two instances generating images simultaneously.

Available Model Types

Category Examples What It Does
Checkpoints SD 1.5, SDXL, Flux Base image generation models
LoRAs Style LoRAs, Character LoRAs Fine-tuned additions for specific styles
VAEs SDXL VAE, SD VAE Encoder/decoder for better image quality
ControlNet Canny, Depth, Pose Guide generation with reference images
Upscalers RealESRGAN, SwinIR Increase image resolution

The model registry has 100+ pre-defined models you can download with one click.

How to Generate Images

Simple prompt: Say: "Generate an image of a sunset over mountains with dramatic clouds"

With settings: Say: "Generate a 1024x768 image of a cyberpunk city at night, 30 steps, high detail"

Through the web UI:

  1. Open your browser to http://localhost:8188
  2. You'll see the visual workflow editor
  3. Build or load a workflow
  4. Click "Queue Prompt" to generate

ComfyUI Tools Reference

Tool How to Use It
comfyui_status "Is ComfyUI running?"
comfyui_instances "List ComfyUI instances"
comfyui_instance_start "Start a ComfyUI instance on GPU 0"
comfyui_instance_stop "Stop the ComfyUI instance"
comfyui_generate "Generate an image: [description]"
comfyui_models "List available ComfyUI models"
comfyui_nodes "List installed ComfyUI nodes"

ComfyUI Troubleshooting

"ComfyUI unreachable":

  • Start the management server with launcher.bat in extensions/comfyui/
  • Check that ports 5000 and 8188 aren't in use

"No checkpoints installed":

  • You need to download at least one checkpoint model
  • Say "List ComfyUI models" to see what's available in the registry
  • Download via the management API or web UI

Images look bad or distorted:

  • Try different prompts — be more specific
  • Increase step count (20-30 is usually good)
  • Try a different checkpoint model
  • Adjust CFG scale (7-9 is typical)

Clone this wiki locally