-
Notifications
You must be signed in to change notification settings - Fork 149
Writing Plugins
Pallaidium discovers AI models at startup from a folder of plain Python files.
Adding a new model means dropping one .py file in the right folder — no
registration code, no __init__.py edits, no restart of the registry.
- How Discovery Works
- Quick Start
- File and Folder Layout
- Class Attributes Reference
- Implementing load()
- Implementing generate()
- ModelInputs Fields
- Available Helpers
- Custom UI with draw_custom_ui()
- Multiple Variants from One Base Class
- Complete Examples
- Testing and Debugging
When Blender loads Pallaidium, models/__init__.py calls discover().
It scans every .py file under models_plugins/, imports each one, and
instantiates every class that:
- subclasses
ModelPlugin - has a non-empty
MODEL_ID
Each instance is stored in PLUGIN_REGISTRY[MODEL_ID] and added to the
dropdown for its MODEL_TYPE. Files or directories whose names start with
_ are silently skipped (use this for shared base classes or drafts).
You never need to edit any existing file. Drop your
.pyin the right folder and restart Blender.
- Copy
models_plugins/_template.pyto the correct sub-folder. - Fill in the four identity attributes and implement
load()+generate(). - Restart Blender (or use Reload Scripts in the Text Editor).
models_plugins/
image/my_cool_model.py ← new file, done
models_plugins/
_template.py ← starter template (ignored by loader)
image/ → plugins that produce images (.png / .jpg)
video/ → plugins that produce video (.mp4)
audio/ → plugins that produce audio (.wav)
text/ → plugins that produce a text strip
Put your file in the folder that matches what your model outputs.
One file may define multiple plugin classes (e.g. a base class plus
txt2vid / img2vid sub-classes that share a load() implementation).
Files named _something.py or inside directories named _something/
are ignored by the loader — use this convention for shared helpers or
work-in-progress files.
MODEL_ID = "author/my-model" # unique key — HuggingFace repo ID is ideal
DISPLAY_NAME = "Image: My Model (1024)" # shown in the dropdown
MODEL_TYPE = "image" # "image" | "video" | "audio" | "text"
DESCRIPTION = "Short tooltip text" # shown as a tooltip in the UIMODEL_ID must be unique across all plugins. Duplicate IDs are skipped
with a warning in the Blender console.
INPUTS is a bitflag that tells the framework which fields to populate in
the ModelInputs object passed to generate().
from ...models.base import InputSpec
INPUTS = InputSpec.PROMPT | InputSpec.IMAGE| Flag | Field populated in ModelInputs
|
Notes |
|---|---|---|
PROMPT |
inputs.prompt |
Main text prompt |
NEG_PROMPT |
inputs.neg_prompt |
Negative prompt |
IMAGE |
inputs.image |
PIL.Image from the selected strip |
MULTI_IMAGE |
inputs.images |
List of PIL.Image; count set by PARAMS.max_multi_images
|
AUDIO_REF |
inputs.audio_ref |
Path to a reference .wav / .mp3 (optional) |
AUDIO_REF_REQ |
inputs.audio_ref |
Same, but UI marks it required |
TEXT_REF |
inputs.text_ref |
Reference transcription string |
VIDEO |
inputs.video_path |
Path to an input video file |
FACE_FOLDER |
inputs.face_folder |
Path for IP-Adapter face images |
STYLE_FOLDER |
inputs.style_folder |
Path for IP-Adapter style images |
LORA |
inputs.lora_files |
List of (path, weight) tuples |
API_KEY |
(read manually) | Signals that an external API key is required |
HF_TOKEN |
(read from prefs) | Shows the HuggingFace token field; call login() in load()
|
UI_SECTIONS is a list of UISection values. The panel renders exactly
these sections, in the order listed.
from ...models.base import UISection
UI_SECTIONS = [
UISection.PROMPT,
UISection.NEG_PROMPT,
UISection.IMAGE_STRIP,
UISection.RESOLUTION,
UISection.STEPS,
UISection.GUIDANCE,
UISection.SEED,
]| Value | What it renders |
|---|---|
PROMPT |
Main prompt textarea |
NEG_PROMPT |
Negative prompt textarea |
IMAGE_STRIP |
Single image strip eyedropper |
MULTI_IMAGES |
Dynamic add/remove image strip pickers |
TRIPLE_IMAGE |
Three fixed image strip pickers |
TRIPLE_PROMPT_IMG |
Three (prompt textarea + image picker) pairs |
AUDIO_REF |
Speaker reference file picker |
TEXT_REF |
Reference text input |
VIDEO_STRIP |
Video strip eyedropper |
RESOLUTION |
Width × Height dropdowns |
FRAMES |
Frame count slider |
STEPS |
Inference steps slider |
GUIDANCE |
Guidance / word-power slider |
IMAGE_STRENGTH |
img2img / inpaint strength slider |
SEED |
Seed input + randomise toggle |
LORA |
LoRA folder + weighted file list |
IP_ADAPTER |
Face folder + style folder pickers |
AUDIO_DURATION |
Duration slider |
SPEED |
Playback speed / CPS slider |
CHAT_PARAMS |
Exaggeration, pace, temperature sliders |
ILLUMINATION |
Lighting style + direction dropdowns |
POSE_TOGGLE |
"Read as OpenPose Rig Image" checkbox |
SCRIBBLE_TOGGLE |
"Read as Scribble Image" checkbox |
ENHANCE |
Quality / Speed / Faces / Upscale toggles |
Sections not in UI_SECTIONS are hidden — the user never sees controls
they cannot use with your model.
Override only the fields that differ from the generic defaults.
from ...models.base import ParamSpec
PARAMS = ParamSpec(
width=1024,
height=1024,
steps=20,
guidance=7.5,
strength=0.8, # img2img / inpaint strength
max_multi_images=3, # for MULTI_IMAGE input
)| Field | Default | Description |
|---|---|---|
width |
1024 | Output width in pixels |
height |
576 | Output height in pixels |
frames |
49 | Video frame count |
steps |
25 | Inference steps |
guidance |
7.5 | Guidance / CFG scale |
strength |
0.8 | img2img / inpaint strength |
audio_length |
5.0 | Audio duration in seconds |
max_multi_images |
1 | Maximum strips for MULTI_IMAGE mode |
These four booleans tell the framework what modes the model supports.
The defaults are permissive; set False only when a feature truly does
not apply.
supports_inpaint: bool = True # set False → inpaint mode never activated
supports_img2img: bool = True # set False → img2img conversion never activated
requires_input_strip: bool = False # set True → always requires a selected strip
uses_standard_input_strip: bool = True # set False → plugin draws its own strip UICommon patterns:
# ControlNet / conditioning model — always needs an image, no inpaint/img2img
supports_inpaint = False
supports_img2img = False
requires_input_strip = True
# Background remover — processes a strip, no conversion modes
supports_inpaint = False
supports_img2img = False
uses_standard_input_strip = False # plugin's draw_custom_ui() handles the UI
# Cloud API — no local strip selection at all
uses_standard_input_strip = FalseA list of Python package names checked by is_available(). If any is
missing the plugin is still registered, but Blender will prompt the user
to install it before generating.
REQUIRED_PACKAGES = ["torch", "diffusers", "transformers"]Use the top-level importable name (e.g. "PIL" not "Pillow").
def load(self, prefs, scene, **kwargs) -> dict:
"""Load and return the model pipeline.
Called once; the result is cached by MODEL_ID for the lifetime of
the Blender session.
"""Parameters:
| Name | Type | Description |
|---|---|---|
prefs |
AddonPreferences |
Add-on preferences (HuggingFace token, output folder, etc.) |
scene |
bpy.types.Scene |
The active scene (rarely needed in load) |
**kwargs |
dict | Extra context; kwargs.get("mode") is "txt2img" / "img2img" / "inpaint"
|
Return value: any object — it is passed back as the first argument to
generate(). Conventionally a dict:
return {"pipe": pipe, "converter": pipe, "refiner": None, "preprocessor": None}Memory management: use enable_model_cpu_offload() or
enable_sequential_cpu_offload() so GPU memory is freed between runs.
def load(self, prefs, scene, **kw):
import torch
from diffusers import FluxPipeline
pipe = FluxPipeline.from_pretrained(self.MODEL_ID, torch_dtype=torch.bfloat16)
if gfx_device == "mps":
pipe.to("mps")
elif low_vram():
pipe.enable_sequential_cpu_offload()
else:
pipe.enable_model_cpu_offload()
return {"pipe": pipe, "converter": pipe, "refiner": None, "preprocessor": None}def generate(self, pipe_obj, inputs: ModelInputs, scene, prefs):
"""Run inference and return a PIL.Image (for image plugins) or a
file path string (for video/audio/text plugins)."""Parameters:
| Name | Type | Description |
|---|---|---|
pipe_obj |
any | Whatever load() returned |
inputs |
ModelInputs |
Collected inputs (only fields in INPUTS are populated) |
scene |
bpy.types.Scene |
Active scene |
prefs |
AddonPreferences |
Add-on preferences |
Return value:
-
Image plugins: return a
PIL.Image.Image— the framework saves it and adds it to the sequencer. - Video / Audio / Text plugins: return an absolute file path as a string.
Only fields declared in INPUTS are guaranteed to be populated.
Everything else stays at its default.
# Text
inputs.prompt # str
inputs.neg_prompt # str
inputs.text_ref # str — reference transcription (Qwen3-TTS)
# Mode (set by the dispatcher)
inputs.mode # "txt2img" | "img2img" | "inpaint"
# Media
inputs.image # PIL.Image or None — single image from strip
inputs.inpaint_mask # PIL.Image or None — white = paint here
inputs.images # list of PIL.Image — MULTI_IMAGE
inputs.audio_ref # str path or None
inputs.video_path # str path or None
# LoRA
inputs.lora_files # list of (path, weight) tuples
# IP Adapter
inputs.face_folder # str path or None
inputs.style_folder # str path or None
# Generation parameters
inputs.width # int
inputs.height # int
inputs.frames # int
inputs.steps # int
inputs.guidance # float
inputs.strength # float
inputs.seed # int
# Audio
inputs.audio_length # float (seconds)
inputs.speed # float
inputs.exaggeration # float
inputs.pace # float
inputs.temperature # float
# Lighting (Kontext Relight)
inputs.illumination_style # str
inputs.light_direction # strImport from ...utils.helpers:
from ...utils.helpers import gfx_device, low_vram, solve_path, clean_filename, \
find_strip_by_name, get_strip_path, load_first_frame| Helper | Signature | Description |
|---|---|---|
gfx_device |
str |
"cuda" / "mps" / "cpu"
|
low_vram() |
() → bool |
True when VRAM < 8 GB |
solve_path(filename) |
str → str |
Builds an absolute output path inside the user's Pallaidium media folder |
clean_filename(text) |
str → str |
Strips characters that are invalid in file names |
find_strip_by_name(scene, name) |
(scene, str) → Strip|None |
Finds a sequencer strip by name |
get_strip_path(strip) |
Strip → str |
Returns the absolute file path of an image or movie strip |
load_first_frame(path) |
str → PIL.Image |
Opens the first frame of an image or video file as a PIL.Image
|
If your plugin needs controls that no standard UISection covers, override
draw_custom_ui().
def draw_custom_ui(self, col, context) -> bool:
"""
col — a Blender UILayout column inside the input-selector box.
context — bpy.context
Return True if you completely replaced the standard input_strips
dropdown (like OmniGen's triple-prompt layout).
Return False if you only added extra controls below the dropdown
(or added nothing at all).
"""
scene = context.scene
if scene.sequence_editor is None:
return False
row = col.row(align=True)
row.prop_search(
scene, "my_custom_strip",
scene.sequence_editor, "strips",
text="My Strip", icon="FILE_IMAGE",
)
row.operator("sequencer.strip_picker", text="", icon="EYEDROPPER").action = "my_select"
return FalseWhen to return True: only when your UI entirely replaces the
Input mode selector (the txt2img / img2img / input strips
dropdown). Returning True suppresses that dropdown.
For video plugins with uses_standard_input_strip = False, the
framework calls draw_custom_ui() in the strip-selector area of the
panel (below the prompt). Return value is ignored for video plugins.
When a service offers text-to-video, image-to-video, and subject-to-video
variants, share the load() logic in a private base class and override
only what differs. Prefix the base class name with _ so the loader
skips it (it has no MODEL_ID anyway, but the underscore makes intent
clear).
class _MyModelBase(ModelPlugin):
MODEL_TYPE = "video"
INPUTS = InputSpec.PROMPT
UI_SECTIONS = [UISection.PROMPT, UISection.SEED]
PARAMS = ParamSpec(steps=1)
REQUIRED_PACKAGES = ["torch", "diffusers"]
def load(self, prefs, scene, **kw):
# shared loading logic
...
return {"pipe": pipe}
class MyModelTxt2VidPlugin(_MyModelBase):
MODEL_ID = "author/my-model-txt2vid"
DISPLAY_NAME = "Video: My Model txt2vid"
DESCRIPTION = "Text to video"
def generate(self, pipe_obj, inputs, scene, prefs):
...
class MyModelImg2VidPlugin(_MyModelBase):
MODEL_ID = "author/my-model-img2vid"
DISPLAY_NAME = "Video: My Model img2vid"
DESCRIPTION = "Image to video"
INPUTS = InputSpec.PROMPT | InputSpec.IMAGE
def generate(self, pipe_obj, inputs, scene, prefs):
...Both sub-classes appear in the video dropdown independently.
"""Text-to-image via my-org/my-model."""
import torch
from ...models.base import ModelPlugin, InputSpec, UISection, ParamSpec, ModelInputs
from ...utils.helpers import gfx_device, low_vram
class MyModelPlugin(ModelPlugin):
MODEL_ID = "my-org/my-model"
DISPLAY_NAME = "Image: My Model"
MODEL_TYPE = "image"
DESCRIPTION = "Text-to-image via my-org/my-model"
INPUTS = InputSpec.PROMPT | InputSpec.NEG_PROMPT
UI_SECTIONS = [
UISection.PROMPT, UISection.NEG_PROMPT,
UISection.RESOLUTION, UISection.STEPS, UISection.GUIDANCE, UISection.SEED,
]
PARAMS = ParamSpec(steps=20, guidance=7.5)
REQUIRED_PACKAGES = ["torch", "diffusers"]
# No inpaint or img2img — this is a pure text-to-image model
supports_inpaint = False
supports_img2img = False
def load(self, prefs, scene, **kw):
from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained(
self.MODEL_ID, torch_dtype=torch.float16,
)
if gfx_device == "mps":
pipe.to("mps")
elif low_vram():
pipe.enable_sequential_cpu_offload()
else:
pipe.enable_model_cpu_offload()
return {"pipe": pipe, "converter": None, "refiner": None, "preprocessor": None}
def generate(self, pipe_obj, inputs: ModelInputs, scene, prefs):
pipe = pipe_obj["pipe"]
seed = inputs.seed
generator = (
torch.Generator("cuda").manual_seed(seed)
if torch.cuda.is_available() and seed != 0 else None
)
return pipe(
prompt=inputs.prompt,
negative_prompt=inputs.neg_prompt,
num_inference_steps=inputs.steps,
guidance_scale=inputs.guidance,
height=inputs.height,
width=inputs.width,
generator=generator,
).images[0]"""Text-to-image, img2img, and inpaint via my-org/my-model."""
import torch
from ...models.base import ModelPlugin, InputSpec, UISection, ParamSpec, ModelInputs
from ...utils.helpers import gfx_device, low_vram
class MyInpaintPlugin(ModelPlugin):
MODEL_ID = "my-org/my-inpaint-model"
DISPLAY_NAME = "Image: My Model (inpaint)"
MODEL_TYPE = "image"
DESCRIPTION = "Text-to-image with img2img and inpaint support"
INPUTS = InputSpec.PROMPT | InputSpec.NEG_PROMPT | InputSpec.IMAGE
UI_SECTIONS = [
UISection.PROMPT, UISection.NEG_PROMPT, UISection.IMAGE_STRIP,
UISection.RESOLUTION, UISection.STEPS, UISection.GUIDANCE,
UISection.IMAGE_STRENGTH, UISection.SEED,
]
PARAMS = ParamSpec(steps=30, guidance=7.5, strength=0.75)
REQUIRED_PACKAGES = ["torch", "diffusers"]
def load(self, prefs, scene, **kw):
from diffusers import (
StableDiffusionPipeline,
StableDiffusionImg2ImgPipeline,
StableDiffusionInpaintPipeline,
)
mode = kw.get("mode", "txt2img")
kwargs = dict(pretrained_model_name_or_path=self.MODEL_ID, torch_dtype=torch.float16)
if mode == "inpaint":
pipe = StableDiffusionInpaintPipeline.from_pretrained(**kwargs)
elif mode == "img2img":
pipe = StableDiffusionImg2ImgPipeline.from_pretrained(**kwargs)
else:
pipe = StableDiffusionPipeline.from_pretrained(**kwargs)
if gfx_device == "mps":
pipe.to("mps")
elif low_vram():
pipe.enable_model_cpu_offload()
else:
pipe.to(gfx_device)
return {"pipe": pipe, "converter": pipe, "refiner": None, "preprocessor": None}
def generate(self, pipe_obj, inputs: ModelInputs, scene, prefs):
pipe = pipe_obj["pipe"]
seed = inputs.seed
generator = (
torch.Generator("cuda").manual_seed(seed)
if torch.cuda.is_available() and seed != 0 else None
)
common = dict(
prompt=inputs.prompt,
negative_prompt=inputs.neg_prompt,
num_inference_steps=inputs.steps,
guidance_scale=inputs.guidance,
generator=generator,
)
if inputs.mode == "inpaint" and inputs.image and inputs.inpaint_mask:
return pipe(
**common,
image=inputs.image,
mask_image=inputs.inpaint_mask,
height=inputs.height,
width=inputs.width,
).images[0]
if inputs.mode == "img2img" and inputs.image:
return pipe(
**common,
image=inputs.image,
strength=1.0 - inputs.strength,
).images[0]
return pipe(
**common,
height=inputs.height,
width=inputs.width,
).images[0]"""Text-to-video via my-org/my-video-model."""
import shutil
import torch
from diffusers.utils import export_to_video
from ...models.base import ModelPlugin, InputSpec, UISection, ParamSpec, ModelInputs
from ...utils.helpers import gfx_device, low_vram, solve_path, clean_filename
class MyVideoPlugin(ModelPlugin):
MODEL_ID = "my-org/my-video-model"
DISPLAY_NAME = "Video: My Model"
MODEL_TYPE = "video"
DESCRIPTION = "Text-to-video"
INPUTS = InputSpec.PROMPT
UI_SECTIONS = [
UISection.PROMPT,
UISection.RESOLUTION, UISection.FRAMES, UISection.STEPS,
UISection.GUIDANCE, UISection.SEED,
]
PARAMS = ParamSpec(width=848, height=480, frames=49, steps=25, guidance=6.0)
REQUIRED_PACKAGES = ["torch", "diffusers"]
supports_inpaint = False
supports_img2img = False
def load(self, prefs, scene, **kw):
from diffusers import CogVideoXPipeline # replace with your pipeline class
pipe = CogVideoXPipeline.from_pretrained(self.MODEL_ID, torch_dtype=torch.bfloat16)
if gfx_device == "mps":
pipe.to("mps")
elif low_vram():
pipe.enable_sequential_cpu_offload()
else:
pipe.enable_model_cpu_offload()
return {"pipe": pipe, "refiner": None, "last_model_card": self.MODEL_ID}
def generate(self, pipe_obj, inputs: ModelInputs, scene, prefs):
import bpy
pipe = pipe_obj["pipe"]
seed = inputs.seed
generator = (
torch.Generator("cuda").manual_seed(seed)
if torch.cuda.is_available() and seed != 0 else None
)
output = pipe(
prompt=inputs.prompt,
num_inference_steps=inputs.steps,
guidance_scale=inputs.guidance,
height=inputs.height,
width=inputs.width,
num_frames=inputs.frames,
generator=generator,
)
render = bpy.context.scene.render
fps = round(render.fps / render.fps_base, 3)
tmp_path = export_to_video(output.frames[0], fps=fps)
dst_path = solve_path(clean_filename(str(seed) + "_" + inputs.prompt) + ".mp4")
shutil.move(tmp_path, dst_path)
return dst_path"""Text-to-speech via my-org/my-tts-model."""
import torch
from ...models.base import ModelPlugin, InputSpec, UISection, ParamSpec, ModelInputs
from ...utils.helpers import solve_path, clean_filename
class MyTTSPlugin(ModelPlugin):
MODEL_ID = "my-org/my-tts"
DISPLAY_NAME = "TTS: My TTS Model"
MODEL_TYPE = "audio"
DESCRIPTION = "Text-to-speech via my-org/my-tts"
INPUTS = InputSpec.PROMPT | InputSpec.AUDIO_REF
UI_SECTIONS = [
UISection.PROMPT,
UISection.AUDIO_REF,
UISection.SEED,
]
PARAMS = ParamSpec()
REQUIRED_PACKAGES = ["torch", "torchaudio"]
def load(self, prefs, scene, **kw):
# Load and return your model — the return value is cached.
from my_tts_lib import TTSModel
model = TTSModel.from_pretrained(self.MODEL_ID)
return {"model": model}
def generate(self, pipe_obj, inputs: ModelInputs, scene, prefs) -> str:
import torchaudio as ta
model = pipe_obj["model"]
torch.manual_seed(inputs.seed)
wav = model.generate(
text=inputs.prompt,
speaker_wav=inputs.audio_ref, # optional; None if no reference
)
out_path = solve_path(clean_filename(inputs.prompt[:30]) + ".wav")
ta.save(out_path, wav, model.sample_rate)
return out_pathConsole output: every plugin load attempt is logged to the Blender
system console (Window → Toggle System Console on Windows). Look for:
[Pallaidium] Registered image plugin: my-org/my-model
If the file fails to import, the full traceback is printed there.
Skipping on error: a broken plugin does not crash the add-on. The registry skips it and continues loading the rest.
Re-loading during development: open the Blender Text Editor, create a new script, and run:
import importlib, sys
# Remove cached module so discover() re-imports your file
for key in list(sys.modules.keys()):
if "pallaidium" in key and "my_model_name" in key:
del sys.modules[key]
from bl_ext.user_default.pallaidium_generative_ai.models import discover
discover()Checking registration:
from bl_ext.user_default.pallaidium_generative_ai.models import PLUGIN_REGISTRY
print(list(PLUGIN_REGISTRY.keys()))Common mistakes:
| Symptom | Likely cause |
|---|---|
| Plugin doesn't appear in dropdown |
MODEL_ID is empty, or there is a syntax / import error — check the console |
generate() receives None for a field |
That InputSpec flag was not added to INPUTS
|
| UI section not shown | That UISection value was not added to UI_SECTIONS
|
| Inpaint / img2img activates unexpectedly | Set supports_inpaint = False / supports_img2img = False
|
| Strip selector shown even though model doesn't need one | Set uses_standard_input_strip = False and implement draw_custom_ui()
|
Duplicate MODEL_ID warning |
Two plugins share the same MODEL_ID string |