Advanced

Fetching External Resources

Models, Adapters, Checkpoints etc

Files

Online Images and Videos

Structuring Arguments

Multiplexing Results

Customizing Pipeline Components

Model Quantization

The quantization configuration allows you to specify how model components should be quantized to reduce memory usage and potentially improve performance. This is defined in the workflow JSON schema and processed by the pipeline system.

Schema Definition

The quantization config can be specified for pipeline components like transformers, text encoders, etc. Here's the relevant schema structure:

"quantization_config": {
    "type": "object",
    "properties": {
        "configuration": {
            "type": "object",
            "properties": {
                "config_type": {
                    "type": "string"
                }
            },
            "required": ["config_type"]
        },
        "arguments": {
            "$ref": "#/$defs/arguments"
        }
    },
    "required": ["configuration", "arguments"]
}

Usage Example

Here's an example of using quantization config in a workflow file:

"transformer": {
    "configuration": {
        "pipeline_type": "SD3Transformer2DModel"
    },
    "quantization_config": {
        "configuration": {
            "config_type": "BitsAndBytesConfig"
        },
        "arguments": {
            "load_in_4bit": true,
            "bnb_4bit_quant_type": "nf4",
            "bnb_4bit_compute_dtype": "torch.bfloat16"
        }
    },
    "from_pretrained_arguments": {
        "model_name": "stabilityai/stable-diffusion-3.5-large",
        "subfolder": "transformer",
        "torch_dtype": "torch.bfloat16"
    }
}

Common Config Types

BitsAndBytesConfig

The BitsAndBytesConfig is used for 4-bit and 8-bit quantization using the bitsandbytes library. Common arguments include:

• load_in_4bit: Enable 4-bit quantization
• bnb_4bit_quant_type: Quantization type (e.g., "nf4")
• bnb_4bit_compute_dtype: Computation data type

TorchAoConfig

GGUFQuantizationConfig

Notes

• Quantization config is optional for pipeline components
• The config_type should be a valid configuration class (typically from transformers or diffusers)
• Arguments are passed directly to the config_type constructor
• Improper configuration may result in runtime errors when loading the model