Advanced

Fetching External Resources

Models, Adapters, Checkpoints etc

Files

Online Images and Videos

Structuring Arguments

Multiplexing Results

Customizing Pipeline Components

Model Quantization

The quantization configuration allows you to specify how model components should be quantized to reduce memory usage and potentially improve performance. This is defined in the workflow JSON schema and processed by the pipeline system.

Schema Definition

The quantization config can be specified for pipeline components like transformers, text encoders, etc. Here's the relevant schema structure:

"quantization_config": {
    "type": "object",
    "properties": {
        "configuration": {
            "type": "object",
            "properties": {
                "config_type": {
                    "type": "string"
                }
            },
            "required": ["config_type"]
        },
        "arguments": {
            "$ref": "#/$defs/arguments"
        }
    },
    "required": ["configuration", "arguments"]
}

Usage Example

Here's an example of using quantization config in a workflow file:

"transformer": {
    "configuration": {
        "pipeline_type": "SD3Transformer2DModel"
    },
    "quantization_config": {
        "configuration": {
            "config_type": "BitsAndBytesConfig"
        },
        "arguments": {
            "load_in_4bit": true,
            "bnb_4bit_quant_type": "nf4",
            "bnb_4bit_compute_dtype": "torch.bfloat16"
        }
    },
    "from_pretrained_arguments": {
        "model_name": "stabilityai/stable-diffusion-3.5-large",
        "subfolder": "transformer",
        "torch_dtype": "torch.bfloat16"
    }
}

Common Config Types

BitsAndBytesConfig

The BitsAndBytesConfig is used for 4-bit and 8-bit quantization using the bitsandbytes library. Common arguments include:

• load_in_4bit: Enable 4-bit quantization
• bnb_4bit_quant_type: Quantization type (e.g., "nf4")
• bnb_4bit_compute_dtype: Computation data type

TorchAoConfig

The TorchAoConfig is used for 4-bit and 8-bit quantization using the torchao library. Common arguments include:

• quant_type: The type of quantization we want to use (e.g., "int4wo", "int4dq", "int8wo", "int8dq")

GGUFQuantizationConfig

The GGUFQuantizationConfig is used for 4-bit and 8-bit quantization using the gguf file format. Common arguments include:

• compute_dtype: This sets the computational type which might be different than the input type (e.g., torch.float32)

Notes

• Quantization config is optional for pipeline components
• The config_type should be a valid configuration class (typically from transformers or diffusers)
• Arguments are passed directly to the config_type constructor
• Improper configuration may result in runtime errors when loading the model