Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions built-in-nodes/BasicScheduler.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
---
title: "BasicScheduler - ComfyUI Built-in Node Documentation"
description: "The BasicScheduler node is used to compute a sequence of sigma values for diffusion models based on the provided scheduler, model, and denoising parameters."
sidebarTitle: "BasicScheduler"
icon: "circle"
---

The `BasicScheduler` node is designed to compute a sequence of sigma values for diffusion models based on the provided scheduler, model, and denoising parameters. It dynamically adjusts the total number of steps based on the denoise factor to fine-tune the diffusion process, providing precise "recipes" for different stages in advanced sampling processes that require fine control (such as multi-stage sampling).

## Inputs

| Parameter | Data Type | Input Type | Default | Range | Metaphor Description | Technical Purpose |
| ----------- | ------------- | ---------- | ------- | --------- | ------------------------------ | ---------------------------- |
| `model` | MODEL | Input | - | - | **Canvas Type**: Different canvas materials need different paint formulas | Diffusion model object, determines sigma calculation basis |
| `scheduler` | COMBO[STRING] | Widget | - | 9 options | **Mixing Technique**: Choose how paint concentration changes | Scheduling algorithm, controls noise decay mode |
| `steps` | INT | Widget | 20 | 1-10000 | **Mixing Count**: 20 mixes vs 50 mixes precision difference | Sampling steps, affects generation quality and speed |
| `denoise` | FLOAT | Widget | 1.0 | 0.0-1.0 | **Creation Intensity**: Control level from fine-tuning to repainting | Denoising strength, supports partial repainting scenarios |

### Scheduler Types

Based on source code `comfy.samplers.SCHEDULER_NAMES`, supports the following 9 schedulers:

| Scheduler Name | Characteristics | Use Cases | Noise Decay Pattern |
| -------------------- | -------------------- | ---------------------------- | ---------------------------- |
| **normal** | Standard linear | General scenarios, balanced | Uniform decay |
| **karras** | Smooth transition | High quality, detail-rich | Smooth non-linear decay |
| **exponential** | Exponential decay | Fast generation, efficiency | Exponential rapid decay |
| **sgm_uniform** | SGM uniform | Specific model optimization | SGM optimized decay |
| **simple** | Simple scheduling | Quick testing, basic use | Simplified decay |
| **ddim_uniform** | DDIM uniform | DDIM sampling optimization | DDIM specific decay |
| **beta** | Beta distribution | Special distribution needs | Beta function decay |
| **linear_quadratic** | Linear quadratic | Complex scenario optimization| Quadratic function decay |
| **kl_optimal** | KL optimal | Theoretical optimization | KL divergence optimized decay|

## Outputs

| Parameter | Data Type | Output Type | Metaphor Description | Technical Meaning |
| --------- | --------- | ----------- | ---------------------- | -------------------------------- |
| `sigmas` | SIGMAS | Output | **Paint Recipe Chart**: Detailed paint concentration list for step-by-step use | Noise level sequence, guides diffusion model denoising process |

## Node Role: Artist's Color Mixing Assistant

Imagine you are an artist creating a clear image from a chaotic mixture of paint (noise). `BasicScheduler` acts like your **professional color mixing assistant**, whose job is to prepare a series of precise paint concentration recipes:

### Workflow

- **Step 1**: Use 90% concentration paint (high noise level)
- **Step 2**: Use 80% concentration paint
- **Step 3**: Use 70% concentration paint
- **...**
- **Final Step**: Use 0% concentration (clean canvas, no noise)

### Color Assistant's Special Skills

**Different mixing methods (scheduler)**:

- **"karras" mixing method**: Paint concentration changes very smoothly, like professional artist's gradient technique
- **"exponential" mixing method**: Paint concentration decreases rapidly, suitable for quick creation
- **"linear" mixing method**: Paint concentration decreases uniformly, stable and controllable

**Fine control (steps)**:

- **20 mixes**: Quick painting, efficiency priority
- **50 mixes**: Fine painting, quality priority

**Creation intensity (denoise)**:

- **1.0 = Complete new creation**: Start completely from blank canvas
- **0.5 = Half transformation**: Keep half of original painting, transform half
- **0.2 = Fine adjustment**: Only make subtle adjustments to original painting

### Collaboration with Other Nodes

`BasicScheduler` (Color Assistant) → Prepare Recipe → `SamplerCustom` (Artist) → Actual Painting → Completed Work
47 changes: 47 additions & 0 deletions built-in-nodes/Canny.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
---
title: "Canny - ComfyUI Built-in Node Documentation"
description: "The Canny node used to extract edge lines from photos."
sidebarTitle: "Canny"
icon: "circle"
---

Extract all edge lines from photos, like using a pen to outline a photo, drawing out the contours and detail boundaries of objects.

## Working Principle

Imagine you are an artist who needs to use a pen to outline a photo. The Canny node acts like an intelligent assistant, helping you decide where to draw lines (edges) and where not to.

This process is like a screening job:

- **High threshold** is the "must draw line standard": only very obvious and clear contour lines will be drawn, such as facial contours of people and building frames
- **Low threshold** is the "definitely don't draw line standard": edges that are too weak will be ignored to avoid drawing noise and meaningless lines
- **Middle area**: edges between the two standards will be drawn together if they connect to "must draw lines", but won't be drawn if they are isolated

The final output is a black and white image, where white parts are detected edge lines and black parts are areas without edges.

## Inputs

| Parameter Name | Data Type | Input Type | Default | Range | Function Description |
|------------------|-----------|------------|---------|-----------|----------------------|
| `image` | IMAGE | Input | - | - | Original photo that needs edge extraction |
| `low_threshold` | FLOAT | Widget | 0.4 | 0.01-0.99 | Low threshold, determines how weak edges to ignore. Lower values preserve more details but may produce noise |
| `high_threshold` | FLOAT | Widget | 0.8 | 0.01-0.99 | High threshold, determines how strong edges to preserve. Higher values only keep the most obvious contour lines |

## Outputs

| Output Name | Data Type | Description |
|-------------|-----------|-------------|
| `image` | IMAGE | Black and white edge image, white lines are detected edges, black areas are parts without edges |

## Parameter Comparison

![Original Image](/images/built-in-nodes/canny/input.webp)

![Parameter Comparison](/images/built-in-nodes/canny/compare.webp)

**Common Issues:**

- Broken edges: Try lowering high threshold
- Too much noise: Raise low threshold
- Missing important details: Lower low threshold
- Edges too rough: Check input image quality and resolution
28 changes: 28 additions & 0 deletions built-in-nodes/CheckpointLoaderSimple.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
title: "CheckpointLoaderSimple - ComfyUI Built-in Node Documentation"
description: "The CheckpointLoaderSimple node is used to load model files from specified locations and decompose them into three core components: the main model, text encoder, and image encoder/decoder."
sidebarTitle: "CheckpointLoaderSimple"
icon: "circle"
---

This is a model loader node that loads model files from specified locations and decomposes them into three core components: the main model, text encoder, and image encoder/decoder.

This node automatically detects all model files in the `ComfyUI/models/checkpoints` folder, as well as additional paths configured in your `extra_model_paths.yaml` file.

1. **Model Compatibility**: Ensure the selected model is compatible with your workflow. Different model types (such as SD1.5, SDXL, Flux, etc.) need to be paired with corresponding samplers and other nodes
2. **File Management**: Place model files in the `ComfyUI/models/checkpoints` folder, or configure other paths through extra_model_paths.yaml
3. **Interface Refresh**: If new model files are added while ComfyUI is running, you need to refresh the browser (Ctrl+R) to see the new files in the dropdown list

## Inputs

| Parameter | Data Type | Input Type | Default | Range | Description |
|----------------|-----------|------------|---------|-------|-------------|
| `ckpt_name` | STRING | Widget | null | All model files in checkpoints folder | Select the checkpoint model file name to load, which determines the AI model used for subsequent image generation |

## Outputs

| Output Name | Data Type | Description |
|-------------|-----------|-------------|
| `MODEL` | MODEL | The main diffusion model used for image denoising generation, the core component of AI image creation |
| `CLIP` | CLIP | The model used for encoding text prompts, converting text descriptions into information that AI can understand |
| `VAE` | VAE | The model used for image encoding and decoding, responsible for converting between pixel space and latent space |
37 changes: 37 additions & 0 deletions built-in-nodes/CheckpointSave.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
---
title: "CheckpointSave - ComfyUI Built-in Node Documentation"
description: "The CheckpointSave node is used to save the complete Stable Diffusion model (including UNet, CLIP, and VAE components) as a **.safetensors** format checkpoint file."
sidebarTitle: "CheckpointSave"
icon: "circle"
---

The `Save Checkpoint` node is designed to save a complete Stable Diffusion model (including UNet, CLIP, and VAE components) as a **.safetensors** format checkpoint file.

The Save Checkpoint is primarily used in model merging workflows. After creating a new merged model through nodes like `ModelMergeSimple`, `ModelMergeBlocks`, etc., you can use this node to save the result as a reusable checkpoint file.

## Inputs

| Parameter | Data Type | Description |
|-----------|-----------|-------------|
| `model` | MODEL | The model parameter represents the primary model whose state is to be saved. It is essential for capturing the current state of the model for future restoration or analysis. |
| `clip` | CLIP | The clip parameter is intended for the CLIP model associated with the primary model, allowing its state to be saved alongside the main model. |
| `vae` | VAE | The vae parameter is for the Variational Autoencoder (VAE) model, enabling its state to be saved for future use or analysis alongside the main model and CLIP. |
| `filename_prefix` | STRING | This parameter specifies the prefix for the filename under which the checkpoint will be saved. |

Additionally, the node has two hidden inputs for metadata:

**prompt (PROMPT)**: Workflow prompt information
**extra_pnginfo (EXTRA_PNGINFO)**: Additional PNG information

## Outputs

This node will output a checkpoint file, and the corresponding output file path is `output/checkpoints/` directory

## Architecture Compatibility

- Currently fully supported: SDXL, SD3, SVD and other mainstream architectures, see [source code](https://github.com/comfyanonymous/ComfyUI/blob/master/comfy_extras/nodes_model_merging.py#L176-L189)
- Basic support: Other architectures can be saved but without standardized metadata information

## Related Links

Related source code: [nodes_model_merging.py#L227](https://github.com/comfyanonymous/ComfyUI/blob/master/comfy_extras/nodes_model_merging.py#L227)
79 changes: 79 additions & 0 deletions built-in-nodes/ClipLoader.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
title: "ClipLoader - ComfyUI Built-in Node Documentation"
description: "The ClipLoader node is used to load CLIP text encoder models independently."
sidebarTitle: "ClipLoader"
icon: "circle"
---

This node is primarily used for loading CLIP text encoder models independently.
The model files can be detected in the following paths:

- "ComfyUI/models/text_encoders/"
- "ComfyUI/models/clip/"

> If you save a model after ComfyUI has started, you'll need to refresh the ComfyUI frontend to get the latest model file path list

Supported model formats:

- `.ckpt`
- `.pt`
- `.pt2`
- `.bin`
- `.pth`
- `.safetensors`
- `.pkl`
- `.sft`

For more details on the latest model file loading, please refer to [folder_paths](https://github.com/comfyanonymous/ComfyUI/blob/master/folder_paths.py)

## Inputs

| Parameter | Data Type | Description |
|---------------|---------------|-------------|
| `clip_name` | COMBO[STRING] | Specifies the name of the CLIP model to be loaded. This name is used to locate the model file within a predefined directory structure. |
| `type` | COMBO[STRING] | Determines the type of CLIP model to load. As ComfyUI supports more models, new types will be added here. Please check the `CLIPLoader` class definition in [node.py](https://github.com/comfyanonymous/ComfyUI/blob/master/nodes.py) for details. |
| `device` | COMBO[STRING] | Choose the device for loading the CLIP model. `default` will run the model on GPU, while selecting `CPU` will force loading on CPU. |

### Device Options Explained

**When to choose "default":**

- Have sufficient GPU memory
- Want the best performance
- Let the system optimize memory usage automatically

**When to choose "cpu":**

- Insufficient GPU memory
- Need to reserve GPU memory for other models (like UNet)
- Running in a low VRAM environment
- Debugging or special purpose needs

**Performance Impact**

Running on CPU will be much slower than GPU, but it can save valuable GPU memory for other more important model components. In memory-constrained environments, putting the CLIP model on CPU is a common optimization strategy.

### Supported Combinations

| Model Type | Corresponding Encoder |
|------------|---------------------|
| stable_diffusion | clip-l |
| stable_cascade | clip-g |
| sd3 | t5 xxl/ clip-g / clip-l |
| stable_audio | t5 base |
| mochi | t5 xxl |
| cosmos | old t5 xxl |
| lumina2 | gemma 2 2B |
| wan | umt5 xxl |

As ComfyUI updates, these combinations may expand. For details, please refer to the `CLIPLoader` class definition in [node.py](https://github.com/comfyanonymous/ComfyUI/blob/master/nodes.py)

## Outputs

| Parameter | Data Type | Description |
|-----------|-----------|-------------|
| `clip` | CLIP | The loaded CLIP model, ready for use in downstream tasks or further processing. |

## Additional Notes

CLIP models play a core role as text encoders in ComfyUI, responsible for converting text prompts into numerical representations that diffusion models can understand. You can think of them as translators, responsible for translating your text into a language that large models can understand. Of course, different models have their own "dialects," so different CLIP encoders are needed between different architectures to complete the text encoding process.
47 changes: 47 additions & 0 deletions built-in-nodes/ClipMergeSimple.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
---
title: "ClipMergeSimple - ComfyUI Built-in Node Documentation"
description: "The ClipMergeSimple node is used to combine two CLIP text encoder models based on a specified ratio."
sidebarTitle: "ClipMergeSimple"
icon: "circle"
---

`CLIPMergeSimple` is an advanced model merging node used to combine two CLIP text encoder models based on a specified ratio.

This node specializes in merging two CLIP models based on a specified ratio, effectively blending their characteristics. It selectively applies patches from one model to another, excluding specific components like position IDs and logit scale, to create a hybrid model that combines features from both source models.

## Inputs

| Parameter | Data Type | Description |
|-----------|-----------|-------------|
| `clip1` | CLIP | The first CLIP model to be merged. It serves as the base model for the merging process. |
| `clip2` | CLIP | The second CLIP model to be merged. Its key patches, except for position IDs and logit scale, are applied to the first model based on the specified ratio. |
| `ratio` | FLOAT | Range `0.0 - 1.0`, determines the proportion of features from the second model to blend into the first model. A ratio of 1.0 means fully adopting the second model's features, while 0.0 retains only the first model's features. |

## Outputs

| Parameter | Data Type | Description |
|-----------|-----------|-------------|
| `clip` | CLIP | The resulting merged CLIP model, incorporating features from both input models according to the specified ratio. |

## Merging Mechanism Explained

### Merging Algorithm

The node uses weighted averaging to merge the two models:

1. **Clone Base Model**: First clones `clip1` as the base model
2. **Get Patches**: Obtains all key patches from `clip2`
3. **Filter Special Keys**: Skips keys ending with `.position_ids` and `.logit_scale`
4. **Apply Weighted Merge**: Uses the formula `(1.0 - ratio) * clip1 + ratio * clip2`

### Ratio Parameter Explained

- **ratio = 0.0**: Fully uses clip1, ignores clip2
- **ratio = 0.5**: 50% contribution from each model
- **ratio = 1.0**: Fully uses clip2, ignores clip1

## Use Cases

1. **Model Style Fusion**: Combine characteristics of CLIP models trained on different data
2. **Performance Optimization**: Balance strengths and weaknesses of different models
3. **Experimental Research**: Explore combinations of different CLIP encoders
38 changes: 38 additions & 0 deletions built-in-nodes/ClipSave.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
title: "ClipSave - ComfyUI Built-in Node Documentation"
description: "The ClipSave node is used to save CLIP text encoder models in SafeTensors format."
sidebarTitle: "ClipSave"
icon: "circle"
---

The `CLIPSave` node is designed for saving CLIP text encoder models in SafeTensors format. This node is part of advanced model merging workflows and is typically used in conjunction with nodes like `CLIPMergeSimple` and `CLIPMergeAdd`. The saved files use the SafeTensors format to ensure security and compatibility.

## Inputs

| Parameter | Data Type | Required | Default Value | Description |
|-----------|-----------|----------|---------------|-------------|
| clip | CLIP | Yes | - | The CLIP model to be saved |
| filename_prefix | STRING | Yes | "clip/ComfyUI" | The prefix path for the saved file |
| prompt | PROMPT | Hidden | - | Workflow prompt information (for metadata) |
| extra_pnginfo | EXTRA_PNGINFO | Hidden | - | Additional PNG information (for metadata) |

## Outputs

This node has no defined output types. It saves the processed files to the `ComfyUI/output/` folder.

### Multi-file Saving Strategy

The node saves different components based on the CLIP model type:

| Prefix Type | File Suffix | Description |
|------------|-------------|-------------|
| `clip_l.` | `_clip_l` | CLIP-L text encoder |
| `clip_g.` | `_clip_g` | CLIP-G text encoder |
| Empty prefix | No suffix | Other CLIP components |

## Usage Notes

1. **File Location**: All files are saved in the `ComfyUI/output/` directory
2. **File Format**: Models are saved in SafeTensors format for security
3. **Metadata**: Includes workflow information and PNG metadata if available
4. **Naming Convention**: Uses the specified prefix plus appropriate suffixes based on model type
Loading