media-scribe

A Python framework for generating text, images, and short videos by combining large language models with diffusion models. Built on Meta's Llama 3 and the Stable Diffusion family via HuggingFace Transformers and Diffusers.

Features

Interactive text generation via Llama 3 (8B Instruct) with configurable chat history, temperature, and sampling
LLaMA-assisted prompt refinement — chat with Llama 3 to iteratively improve your image prompt before generation
Image generation with Stable Diffusion 3, SDXL, CivitAI models, and pix2pix
Image-to-image transformation using InstructPix2Pix and SD 1.5 img2img pipelines
Frame-by-frame video — produce an animated WebP from a sequence of prompts (each prompt = one frame)
SVD video generation — coherent video from a single text prompt via Stable Video Diffusion (text → anchor image → 14/25-frame clip)
Unified YAML config for all model settings
MPS (Apple Silicon), CUDA, and CPU device support

Prerequisites

Requirement	Notes
Python >= 3.10	Tested on 3.11
PyTorch >= 2.4	MPS (Apple Silicon), CUDA, or CPU
HuggingFace account	Required for gated models (Llama 3, SD3, SVD)

Installation

# 1. Clone the repository
git clone https://github.com/ahestevenz/media-scribe.git
cd media-scribe

# 2. Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate

# 3. Install the package
pip install -e .

# 4. (Optional) Install dev tools including pre-commit hooks
pip install -e ".[dev]"
pre-commit install

Downloading Models

All models are stored under root_models_path (default ~/.cache/huggingface/hub). Run huggingface-cli login once before downloading any gated model.

pip install huggingface_hub
huggingface-cli login

Getting a token: go to huggingface.co/settings/tokens, click New token, choose Read access, and paste the token when prompted.

Gated models (Llama 3, SD3, SVD): a token alone is not enough. You must also visit each model page on HuggingFace and click Agree and access repository to accept the licence before the download will succeed.

1. Llama 3 8B Instruct (gated)

Request access at meta-llama/Meta-Llama-3-8B-Instruct, accept the licence, then:

huggingface-cli download meta-llama/Meta-Llama-3-8B-Instruct

The model is loaded by name at runtime via transformers; no path entry in config.yml is needed.

2. Stable Diffusion 3 Medium (gated)

Request access at stabilityai/stable-diffusion-3-medium, accept the licence, then:

huggingface-cli download stabilityai/stable-diffusion-3-medium \
  sd3_medium_incl_clips_t5xxlfp16.safetensors \
  --local-dir ~/.cache/huggingface/hub/models--stabilityai--stable-diffusion-3-medium-diffusers

Config path: models--stabilityai--stable-diffusion-3-medium-diffusers/sd3_medium_incl_clips_t5xxlfp16.safetensors

3. Stable Diffusion XL Base 1.0

huggingface-cli download stabilityai/stable-diffusion-xl-base-1.0 \
  sd_xl_base_1.0.safetensors \
  --local-dir ~/.cache/huggingface/hub/models--Stable-Diffusion-XL

Config path: models--Stable-Diffusion-XL/sd_xl_base_1.0.safetensors

4. Stable Diffusion XL Refiner 1.0

Used as the refiner for sd_3, sd_xl, and civitai model types.

huggingface-cli download stabilityai/stable-diffusion-xl-refiner-1.0 \
  sd_xl_refiner_1.0.safetensors \
  --local-dir ~/.cache/huggingface/hub/models--stabilityai--stable-diffusion-xl-refiner-1.0

Config path: models--stabilityai--stable-diffusion-xl-refiner-1.0/sd_xl_refiner_1.0.safetensors

5. JuggernautXL by RunDiffusion (CivitAI)

This model is distributed via CivitAI. Download juggernautXL_juggXIByRundiffusion.safetensors from the site, then move it:

mkdir -p ~/.cache/huggingface/hub/models--civitai
mv ~/Downloads/juggernautXL_juggXIByRundiffusion.safetensors \
   ~/.cache/huggingface/hub/models--civitai/

Config path: models--civitai/juggernautXL_juggXIByRundiffusion.safetensors

6. InstructPix2Pix

huggingface-cli download timbrooks/instruct-pix2pix

The HuggingFace cache manager places this at the exact snapshot path expected by the config automatically.

Config path: models--timbrooks--instruct-pix2pix/snapshots/31519b5cb02a7fd89b906d88731cd4d6a7bbf88d

7. SD 1.5 img2img

huggingface-cli download radames/stable-diffusion-v1-5-img2img

Config path: models--radames--stable-diffusion-v1-5-img2img/snapshots/cac24f945bd8f5604e243550bb3092be11964805

8. Stable Video Diffusion img2vid-xt (gated)

Used by bn-run-scribe-video. Request access at stabilityai/stable-video-diffusion-img2vid-xt, accept the licence, then:

huggingface-cli download stabilityai/stable-video-diffusion-img2vid-xt

The pipeline is downloaded automatically on first run if svd_model_path is null in config.yml. To pin a local copy, set:

sd_config:
  svd_model_path: null   # or an absolute/relative local path

Configuration

Copy config.example.yml to config.yml and edit the key fields:

verbose: true
device: "mps"           # "mps" | "cuda" | "cpu"

llama_config:
  model_name: "meta-llama/Meta-Llama-3-8B-Instruct"
  system_prompt: "You are a helpful assistant"
  max_num_historical_messages: 6
  max_tokens: 256
  temperature: 0.7
  top_p: 0.9

sd_config:
  root_models_path: ~/.cache/huggingface/hub
  root_output_dir: ~/Desktop/media-scribe-outputs
  model_type: "sd_xl"   # active model — see Supported Models below
  load_refiner: true
  num_inference_steps: 50
  guidance_scale: 7.5
  svd_model_path: null  # null = auto-download SVD from HF on first use
  model_paths:
    sd_3:
      - models--stabilityai--stable-diffusion-3-medium-diffusers/sd3_medium_incl_clips_t5xxlfp16.safetensors
      - models--stabilityai--stable-diffusion-xl-refiner-1.0/sd_xl_refiner_1.0.safetensors
    sd_xl:
      - models--Stable-Diffusion-XL/sd_xl_base_1.0.safetensors
      - models--Stable-Diffusion-XL/sd_xl_refiner_1.0.safetensors
    civitai:
      - models--civitai/juggernautXL_juggXIByRundiffusion.safetensors
      - models--Stable-Diffusion-XL/sd_xl_refiner_1.0.safetensors
    pix_2_pix:
      - models--timbrooks--instruct-pix2pix/snapshots/31519b5cb02a7fd89b906d88731cd4d6a7bbf88d
      - null
    sd_1_5_img_2_img:
      - models--radames--stable-diffusion-v1-5-img2img/snapshots/cac24f945bd8f5604e243550bb3092be11964805
      - null

Each entry in model_paths is [base_model, refiner]. Use null for the refiner when the model does not have one. All paths are relative to root_models_path. Tilde (~) is expanded automatically.

Usage

CLI

Text generation

bn-run-scribe-text -c config.yml

You: Write me a haiku about diffusion models
LLaMA:
 Noise fades to signal,
 Guided by a gentle hand—
 Art blooms from chaos.
You: exit

Image generation

By default, a LLaMA chat session opens first so you can refine your prompt. Type exit when satisfied and generation starts:

bn-run-scribe-image -c config.yml

Skip LLaMA and type a prompt directly:

bn-run-scribe-image -c config.yml --no-llama

Image-to-image (requires pix_2_pix or sd_1_5_img_2_img as model_type in config):

bn-run-scribe-image -c config.yml --no-llama \
  --path-image path/to/source.png \
  --strength 0.75

Flag	Short	Default	Description
`--conf`	`-c`	required	Path to YAML config file
`--no-llama`	`-nl`	false	Skip LLaMA prompt refinement
`--path-image`	`-i`	none	Source image for img2img
`--strength`	`-s`	0.75	img2img denoising strength (0–1)
`--verbose`	`-v`	INFO	Repeat for DEBUG (`-vv`)
`--profile`	`-p`	none	Write cProfile output to file

SVD video generation

Generates a temporally coherent video from a single text prompt. Internally: prompt → anchor image (base model) → 14/25-frame video (Stable Video Diffusion).

bn-run-scribe-video -c config.yml --no-llama

With LLaMA-assisted prompt refinement:

bn-run-scribe-video -c config.yml

Flag	Short	Default	Description
`--conf`	`-c`	required	Path to YAML config file
`--no-llama`	`-nl`	false	Skip LLaMA prompt refinement
`--num-frames`	`-n`	25	Frames to generate (14 or 25 for SVD)
`--fps`		7	Output frames per second
`--motion-bucket-id`		127	Motion amount (0–255); higher = more motion
`--noise-aug-strength`		0.02	Anchor frame noise (0.0–1.0)
`--num-inference-steps`		25	SVD denoising steps
`--decode-chunk-size`		4	VAE frames decoded at once; lower = less peak memory
`--verbose`	`-v`	INFO	Repeat for DEBUG (`-vv`)
`--profile`	`-p`	none	Write cProfile output to file

Memory note (MPS / Apple Silicon): SVD is memory-intensive. On MPS the base model is automatically offloaded to CPU before SVD runs, and the anchor image is resized to 512×320 (instead of 1024×576) to fit within device limits. If you still run out of memory, lower --num-frames to 14 and --decode-chunk-size to 2.

Python API

from media_scribe.media_scribe_config import MediaScribeConfig
from media_scribe.llama_text_scribe import LlamaTextScribe
from media_scribe.image_video_scribe import ImageVideoScribe

config = MediaScribeConfig.from_yaml("config.yml")

# Text generation
llama = LlamaTextScribe(config)
response = llama.generate_text("Explain diffusion models in simple terms")
print(response)

# Image generation — returns Path to the saved PNG
scribe = ImageVideoScribe(config)
image_path = scribe.generate_image(
    prompt="A futuristic city at sunset, cinematic lighting",
    negative_prompt="blurry, low quality",
)

# Image-to-image — requires pix_2_pix or sd_1_5_img_2_img model type
from pathlib import Path
image_path = scribe.generate_image_from_image(
    prompt="Turn this into a watercolour painting",
    img_path=Path("source.png"),
    strength=0.75,
)

# Frame-by-frame video — each prompt becomes one frame, returns animated WebP
video_path = scribe.generate_video(
    prompts=[
        "a lone wolf standing on a snowy hill at dusk",
        "the wolf lifts its head and howls",
        "the wolf turns and disappears into the forest",
    ],
    fps=8,
    num_inference_steps=20,
)

# SVD video — single prompt → anchor image → coherent video clip
video_path = scribe.generate_video_svd(
    prompt="A majestic eagle soaring over mountain peaks at golden hour",
    negative_prompt="blurry, low quality",
    num_frames=14,       # 14 or 25
    fps=7,
    motion_bucket_id=127,
    decode_chunk_size=4, # lower to reduce peak memory on MPS
)

Examples

Text generation

> bn-run-scribe-text -c config.yml
2026-05-25 12:12:05.988 | INFO     | media_scribe.scripts.run_scribe_text:main:94 - Running without profiling
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:12<00:00,  3.19s/it]
2026-05-25 12:12:23.276 | INFO     | media_scribe.utils:start_text_interaction:49 - Welcome to the LLaMA 3 Interactive Text Generator!
2026-05-25 12:12:23.276 | INFO     | media_scribe.utils:start_text_interaction:50 - Type 'exit' to quit the program and use the last LLaMA response as the prompt.

You:
 Write a short story featuring a shark and a whale as the main characters.
LLaMA:
 Here is a short story featuring a shark and a whale as the main characters:

---

The ocean was a vast and mysterious place, full of wonders and dangers. But for Finley the shark and Wally the whale, it was home.

Finley was a curious and adventurous shark, always looking for the next big thrill. He loved to explore the depths of the ocean, discovering hidden caves and shipwrecks. But despite his bravery, Finley had a secret: he was terrified of the dark.

Wally, on the other hand, was a gentle giant. He was a humpback whale, with a heart as big as the sea itself. Wally loved to sing, and his deep, rumbling voice could be heard for miles. He was a natural performer, and his concerts drew in all sorts of sea creatures from far and wide.

One day, Finley and Wally met on the ocean floor. Finley was exploring a shipwreck, and Wally was singing a solo in the nearby kelp forest. They locked eyes, and Finley felt a spark of recognition. He had never seen a whale up close before, and Wally had never seen a shark that wasn't trying to eat him.

The two became fast friends,
You:

Image generation

> bn-run-scribe-image -c config.yml
2026-05-25 12:00:22.918 | INFO     | media_scribe.scripts.run_scribe_image:main:145 - Running without profiling
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:15<00:00,  3.99s/it]
2026-05-25 12:00:44.902 | INFO     | media_scribe.utils:start_text_interaction:35 - Introduce your prompt to generate the image:
You:
 Generate an image showing a sunset between mountain peaks
Do you want to improve the prompt? (yes/no): no
2026-05-25 12:02:34.368 | INFO     | media_scribe.utils:start_text_interaction:45 - Returning the original prompt...
2026-05-25 12:02:34.368 | INFO     | media_scribe.scripts.run_scribe_image:_main:48 - Enter a negative prompt for image generation (leave blank if none):
Negative Prompt:

Fetching 21 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 21/21 [00:00<00:00, 8786.95it/s]
Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:05<00:00,  1.52it/s]
Fetching 10 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 4094.80it/s]
Loading pipeline components...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00,  4.28it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [02:21<00:00,  2.83s/it]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:31<00:00,  2.09s/it]

Supported Models

Type	Model	`model_type` key	Gated
LLM	Meta Llama 3 8B Instruct	—	yes
Text-to-image	Stable Diffusion 3 Medium	`sd_3`	yes
Text-to-image	Stable Diffusion XL Base + Refiner	`sd_xl`	no
Text-to-image	JuggernautXL (CivitAI)	`civitai`	no
Image-to-image	InstructPix2Pix	`pix_2_pix`	no
Image-to-image	SD 1.5 img2img	`sd_1_5_img_2_img`	no
Text-to-video	Stable Video Diffusion img2vid-xt	SVD (via `generate_video_svd`)	yes

Text-to-image models (sd_3, sd_xl, civitai) support generate_image(), generate_video(), and generate_video_svd(). Image-to-image models support generate_image_from_image() only.

Project Structure

media-scribe/
├── src/
│   └── media_scribe/
│       ├── __init__.py
│       ├── image_video_scribe.py   # ImageVideoScribe
│       ├── llama_text_scribe.py    # LlamaTextScribe
│       ├── media_scribe_config.py  # Pydantic config models
│       ├── utils.py                # Interactive CLI loop
│       └── scripts/
│           ├── run_scribe_image.py
│           ├── run_scribe_text.py
│           └── run_scribe_video.py
├── config.example.yml
├── pyproject.toml
├── .pre-commit-config.yaml
└── README.md

Development

# Run all linters and type checks
pre-commit run --all-files

The pre-commit pipeline runs: autopep8, flake8 (max line length 120), mypy, and isort.

Known Issues

Apple Silicon (MPS) — SVD video generation: `RuntimeError: Invalid buffer size`

Status: open — not yet resolved. Planned for a future release.

Affected command:

bn-run-scribe-video -c config.yml --num-frames 14 --fps 5 --decode-chunk-size 1 --no-llama

Symptom: Generation fails at the first SVD denoising step with:

RuntimeError - Invalid buffer size: 22.15 GB

The error size doubles to 44.30 GB if the SVD model is loaded in float32.

Investigated and ruled out:

Reducing --num-frames to 14 and --decode-chunk-size to 1 — no effect on the buffer error.
Reducing the SVD input resolution to 256×144 without passing height/width to the pipeline — no effect; the pipeline was silently running at its default 1024×576, not the resized image size.
Passing height=144, width=256 explicitly to override the pipeline default — the Invalid buffer size error resolved but exposed a new UNet skip-connection mismatch (Expected size 6 but got size 5). The root cause: both image dimensions must be divisible by 64 (VAE scale 8 × UNet 3-stage scale 8) for skip connections to match. 144 fails this check — latent height 18 halves as 18→9→5→3; the decoder upsamples 3→6 but the encoder skip is 5. The resolution was changed to 320×192 (both divisible by 64), which resolved this secondary error.
Deleting the SD3 base model and forcing gc.collect() + torch.mps.empty_cache() before SVD loads — reduces peak MPS usage but does not affect SVD inference.
Loading SVD in float32 instead of float16 — the error doubled to 44.30 GB proportionally, confirming the buffer size scales with dtype and is a real tensor allocation, not a spurious MPS error.
Dropping the SD3 T5-XXL text encoder on MPS (text_encoder_3=None) — resolved the SD3 loading issue independently but has no effect on SVD.

Likely root causes (under investigation):

MPS per-buffer allocation ceiling — PyTorch's MPS backend enforces a hard limit on the size of a single Metal buffer. On Apple Silicon, this ceiling is tied to the total unified memory of the device. SVD's UNet spatial self-attention at the first resolution level materialises a [frames × heads, seq, seq] matrix that appears to exceed this ceiling for the specific combination of frame count, spatial resolution, and attention head count in stable-video-diffusion-img2vid-xt.
No flash attention on MPS — Unlike CUDA (which uses memory-efficient FlashAttention), MPS computes the full attention matrix in a single operation. There is currently no xformers-style memory-efficient attention backend available for MPS in PyTorch 2.4.x, and enable_attention_slicing does not reduce the allocation for all attention layer types in UNetSpatioTemporalConditionModel.
Possible diffusers 0.30.x incompatibility with MPS for spatiotemporal UNets — UNetSpatioTemporalConditionModel (used exclusively by SVD) may have MPS-specific paths that do not respect attention slicing or that trigger large intermediate allocations not present on CUDA.

License

MIT — see LICENSE for details.

Name		Name	Last commit message	Last commit date
Latest commit History 31 Commits
assets		assets
src/media_scribe		src/media_scribe
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
LICENSE		LICENSE
README.md		README.md
ci.yml		ci.yml
config.example.yml		config.example.yml
config.yml		config.yml
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

media-scribe

Features

Prerequisites

Installation

Downloading Models

1. Llama 3 8B Instruct (gated)

2. Stable Diffusion 3 Medium (gated)

3. Stable Diffusion XL Base 1.0

4. Stable Diffusion XL Refiner 1.0

5. JuggernautXL by RunDiffusion (CivitAI)

6. InstructPix2Pix

7. SD 1.5 img2img

8. Stable Video Diffusion img2vid-xt (gated)

Configuration

Usage

CLI

Text generation

Image generation

SVD video generation

Python API

Examples

Text generation

Image generation

Supported Models

Project Structure

Development

Known Issues

Apple Silicon (MPS) — SVD video generation: `RuntimeError: Invalid buffer size`

License

About

Releases

Packages

Used by

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

media-scribe

Features

Prerequisites

Installation

Downloading Models

1. Llama 3 8B Instruct (gated)

2. Stable Diffusion 3 Medium (gated)

3. Stable Diffusion XL Base 1.0

4. Stable Diffusion XL Refiner 1.0

5. JuggernautXL by RunDiffusion (CivitAI)

6. InstructPix2Pix

7. SD 1.5 img2img

8. Stable Video Diffusion img2vid-xt (gated)

Configuration

Usage

CLI

Text generation

Image generation

SVD video generation

Python API

Examples

Text generation

Image generation

Supported Models

Project Structure

Development

Known Issues

Apple Silicon (MPS) — SVD video generation: RuntimeError: Invalid buffer size

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages

Apple Silicon (MPS) — SVD video generation: `RuntimeError: Invalid buffer size`