Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 1 addition & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ limitations under the License.

## Installation

We recommend installing 🤗 Diffusers in a virtual environment from PyPI or Conda. For more details about installing [PyTorch](https://pytorch.org/get-started/locally/) and [Flax](https://flax.readthedocs.io/en/latest/#installation), please refer to their official documentation.
We recommend installing 🤗 Diffusers in a virtual environment from PyPI or Conda. For more details about installing [PyTorch](https://pytorch.org/get-started/locally/), please refer to their official documentation.

### PyTorch

Expand All @@ -53,14 +53,6 @@ With `conda` (maintained by the community):
conda install -c conda-forge diffusers
```

### Flax

With `pip` (official package):

```bash
pip install --upgrade diffusers[flax]
```

### Apple Silicon (M1/M2) support

Please refer to the [How to use Stable Diffusion in Apple Silicon](https://huggingface.co/docs/diffusers/optimization/mps) guide.
Expand Down
2 changes: 0 additions & 2 deletions docs/source/en/_toctree.yml
Original file line number Diff line number Diff line change
Expand Up @@ -194,8 +194,6 @@
- title: Model accelerators and hardware
isExpanded: false
sections:
- local: using-diffusers/stable_diffusion_jax_how_to
title: JAX/Flax
- local: optimization/onnx
title: ONNX
- local: optimization/open_vino
Expand Down
12 changes: 0 additions & 12 deletions docs/source/en/api/models/autoencoderkl.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,15 +44,3 @@ model = AutoencoderKL.from_single_file(url)
## DecoderOutput

[[autodoc]] models.autoencoders.vae.DecoderOutput

## FlaxAutoencoderKL

[[autodoc]] FlaxAutoencoderKL

## FlaxAutoencoderKLOutput

[[autodoc]] models.vae_flax.FlaxAutoencoderKLOutput

## FlaxDecoderOutput

[[autodoc]] models.vae_flax.FlaxDecoderOutput
8 changes: 0 additions & 8 deletions docs/source/en/api/models/controlnet.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,3 @@ pipe = StableDiffusionControlNetPipeline.from_single_file(url, controlnet=contro
## ControlNetOutput

[[autodoc]] models.controlnets.controlnet.ControlNetOutput

## FlaxControlNetModel

[[autodoc]] FlaxControlNetModel

## FlaxControlNetOutput

[[autodoc]] models.controlnets.controlnet_flax.FlaxControlNetOutput
4 changes: 0 additions & 4 deletions docs/source/en/api/models/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,6 @@ All models are built from the base [`ModelMixin`] class which is a [`torch.nn.Mo
## ModelMixin
[[autodoc]] ModelMixin

## FlaxModelMixin

[[autodoc]] FlaxModelMixin

## PushToHubMixin

[[autodoc]] utils.PushToHubMixin
6 changes: 0 additions & 6 deletions docs/source/en/api/models/unet2d-cond.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,3 @@ The abstract from the paper is:

## UNet2DConditionOutput
[[autodoc]] models.unets.unet_2d_condition.UNet2DConditionOutput

## FlaxUNet2DConditionModel
[[autodoc]] models.unets.unet_2d_condition_flax.FlaxUNet2DConditionModel

## FlaxUNet2DConditionOutput
[[autodoc]] models.unets.unet_2d_condition_flax.FlaxUNet2DConditionOutput
4 changes: 0 additions & 4 deletions docs/source/en/api/outputs.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,6 @@ To check a specific pipeline or model output, refer to its corresponding API doc

[[autodoc]] pipelines.ImagePipelineOutput

## FlaxImagePipelineOutput

[[autodoc]] pipelines.pipeline_flax_utils.FlaxImagePipelineOutput

## AudioPipelineOutput

[[autodoc]] pipelines.AudioPipelineOutput
Expand Down
8 changes: 0 additions & 8 deletions docs/source/en/api/pipelines/controlnet.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,11 +72,3 @@ Make sure to check out the Schedulers [guide](../../using-diffusers/schedulers)

## StableDiffusionPipelineOutput
[[autodoc]] pipelines.stable_diffusion.StableDiffusionPipelineOutput

## FlaxStableDiffusionControlNetPipeline
[[autodoc]] FlaxStableDiffusionControlNetPipeline
- all
- __call__

## FlaxStableDiffusionControlNetPipelineOutput
[[autodoc]] pipelines.stable_diffusion.FlaxStableDiffusionPipelineOutput
4 changes: 0 additions & 4 deletions docs/source/en/api/pipelines/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,10 +106,6 @@ The table below lists all the pipelines currently available in 🤗 Diffusers an

[[autodoc]] pipelines.StableDiffusionMixin.disable_freeu

## FlaxDiffusionPipeline

[[autodoc]] pipelines.pipeline_flax_utils.FlaxDiffusionPipeline

## PushToHubMixin

[[autodoc]] utils.PushToHubMixin
Expand Down
10 changes: 0 additions & 10 deletions docs/source/en/api/pipelines/stable_diffusion/img2img.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,13 +47,3 @@ Make sure to check out the Stable Diffusion [Tips](overview#tips) section to lea
## StableDiffusionPipelineOutput

[[autodoc]] pipelines.stable_diffusion.StableDiffusionPipelineOutput

## FlaxStableDiffusionImg2ImgPipeline

[[autodoc]] FlaxStableDiffusionImg2ImgPipeline
- all
- __call__

## FlaxStableDiffusionPipelineOutput

[[autodoc]] pipelines.stable_diffusion.FlaxStableDiffusionPipelineOutput
10 changes: 0 additions & 10 deletions docs/source/en/api/pipelines/stable_diffusion/inpaint.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,13 +49,3 @@ If you're interested in using one of the official checkpoints for a task, explor
## StableDiffusionPipelineOutput

[[autodoc]] pipelines.stable_diffusion.StableDiffusionPipelineOutput

## FlaxStableDiffusionInpaintPipeline

[[autodoc]] FlaxStableDiffusionInpaintPipeline
- all
- __call__

## FlaxStableDiffusionPipelineOutput

[[autodoc]] pipelines.stable_diffusion.FlaxStableDiffusionPipelineOutput
10 changes: 0 additions & 10 deletions docs/source/en/api/pipelines/stable_diffusion/text2img.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,13 +51,3 @@ If you're interested in using one of the official checkpoints for a task, explor
## StableDiffusionPipelineOutput

[[autodoc]] pipelines.stable_diffusion.StableDiffusionPipelineOutput

## FlaxStableDiffusionPipeline

[[autodoc]] FlaxStableDiffusionPipeline
- all
- __call__

## FlaxStableDiffusionPipelineOutput

[[autodoc]] pipelines.stable_diffusion.FlaxStableDiffusionPipelineOutput
25 changes: 2 additions & 23 deletions docs/source/en/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ specific language governing permissions and limitations under the License.

# Installation

Diffusers is tested on Python 3.8+, PyTorch 1.4+, and Flax 0.4.1+. Follow the installation instructions for the deep learning library you're using, [PyTorch](https://pytorch.org/get-started/locally/) or [Flax](https://flax.readthedocs.io/en/latest/).
Diffusers is tested on Python 3.8+ and PyTorch 1.4+. Install [PyTorch](https://pytorch.org/get-started/locally/) according to your system and setup.

Create a [virtual environment](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/) for easier management of separate projects and to avoid compatibility issues between dependencies. Use [uv](https://docs.astral.sh/uv/), a Rust-based Python package and project manager, to create a virtual environment and install Diffusers.

Expand All @@ -32,12 +32,6 @@ PyTorch only supports Python 3.8 - 3.11 on Windows.
uv pip install diffusers["torch"] transformers
```

Use the command below for Flax.

```bash
uv pip install diffusers["flax"] transformers
```

</hfoption>
<hfoption id="conda">

Expand Down Expand Up @@ -71,27 +65,12 @@ An editable install is recommended for development workflows or if you're using

Clone the repository and install Diffusers with the following commands.

<hfoptions id="editable">
<hfoption id="PyTorch">

```bash
git clone https://github.com/huggingface/diffusers.git
cd diffusers
uv pip install -e ".[torch]"
```

</hfoption>
<hfoption id="Flax">

```bash
git clone https://github.com/huggingface/diffusers.git
cd diffusers
uv pip install -e ".[flax]"
```

</hfoption>
</hfoptions>

> [!WARNING]
> You must keep the `diffusers` folder if you want to keep using the library with the editable install.

Expand Down Expand Up @@ -140,7 +119,7 @@ For more details about managing and cleaning the cache, take a look at the [Unde
## Telemetry logging

Diffusers gathers telemetry information during [`~DiffusionPipeline.from_pretrained`] requests.
The data gathered includes the Diffusers and PyTorch/Flax version, the requested model or pipeline class,
The data gathered includes the Diffusers and PyTorch version, the requested model or pipeline class,
and the path to a pretrained checkpoint if it is hosted on the Hub.

This usage data helps us debug issues and prioritize new features.
Expand Down
83 changes: 2 additions & 81 deletions docs/source/en/training/controlnet.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ specific language governing permissions and limitations under the License.

[ControlNet](https://hf.co/papers/2302.05543) models are adapters trained on top of another pretrained model. It allows for a greater degree of control over image generation by conditioning the model with an additional input image. The input image can be a canny edge, depth map, human pose, and many more.

If you're training on a GPU with limited vRAM, you should try enabling the `gradient_checkpointing`, `gradient_accumulation_steps`, and `mixed_precision` parameters in the training command. You can also reduce your memory footprint by using memory-efficient attention with [xFormers](../optimization/xformers). JAX/Flax training is also supported for efficient training on TPUs and GPUs, but it doesn't support gradient checkpointing or xFormers. You should have a GPU with >30GB of memory if you want to train faster with Flax.
If you're training on a GPU with limited vRAM, you should try enabling the `gradient_checkpointing`, `gradient_accumulation_steps`, and `mixed_precision` parameters in the training command. You can also reduce your memory footprint by using memory-efficient attention with [xFormers](../optimization/xformers).

This guide will explore the [train_controlnet.py](https://github.com/huggingface/diffusers/blob/main/examples/controlnet/train_controlnet.py) training script to help you become familiar with it, and how you can adapt it for your own use-case.

Expand All @@ -28,45 +28,10 @@ pip install .

Then navigate to the example folder containing the training script and install the required dependencies for the script you're using:

<hfoptions id="installation">
<hfoption id="PyTorch">
```bash
cd examples/controlnet
pip install -r requirements.txt
```
</hfoption>
<hfoption id="Flax">

If you have access to a TPU, the Flax training script runs even faster! Let's run the training script on the [Google Cloud TPU VM](https://cloud.google.com/tpu/docs/run-calculation-jax). Create a single TPU v4-8 VM and connect to it:

```bash
ZONE=us-central2-b
TPU_TYPE=v4-8
VM_NAME=hg_flax

gcloud alpha compute tpus tpu-vm create $VM_NAME \
--zone $ZONE \
--accelerator-type $TPU_TYPE \
--version tpu-vm-v4-base

gcloud alpha compute tpus tpu-vm ssh $VM_NAME --zone $ZONE -- \
```

Install JAX 0.4.5:

```bash
pip install "jax[tpu]==0.4.5" -f https://storage.googleapis.com/jax-releases/libtpu_releases.html
```

Then install the required dependencies for the Flax script:

```bash
cd examples/controlnet
pip install -r requirements_flax.txt
```

</hfoption>
</hfoptions>

<Tip>

Expand Down Expand Up @@ -120,7 +85,7 @@ Many of the basic and important parameters are described in the [Text-to-image](

### Min-SNR weighting

The [Min-SNR](https://huggingface.co/papers/2303.09556) weighting strategy can help with training by rebalancing the loss to achieve faster convergence. The training script supports predicting `epsilon` (noise) or `v_prediction`, but Min-SNR is compatible with both prediction types. This weighting strategy is only supported by PyTorch and is unavailable in the Flax training script.
The [Min-SNR](https://huggingface.co/papers/2303.09556) weighting strategy can help with training by rebalancing the loss to achieve faster convergence. The training script supports predicting `epsilon` (noise) or `v_prediction`, but Min-SNR is compatible with both prediction types. This weighting strategy is only supported by PyTorch.

Add the `--snr_gamma` parameter and set it to the recommended value of 5.0:

Expand Down Expand Up @@ -272,9 +237,6 @@ That's it! You don't need to add any additional parameters to your training comm
</hfoption>
</hfoptions>

<hfoptions id="training-inference">
<hfoption id="PyTorch">

```bash
export MODEL_DIR="stable-diffusion-v1-5/stable-diffusion-v1-5"
export OUTPUT_DIR="path/to/save/model"
Expand All @@ -292,47 +254,6 @@ accelerate launch train_controlnet.py \
--push_to_hub
```

</hfoption>
<hfoption id="Flax">

With Flax, you can [profile your code](https://jax.readthedocs.io/en/latest/profiling.html) by adding the `--profile_steps==5` parameter to your training command. Install the Tensorboard profile plugin:

```bash
pip install tensorflow tensorboard-plugin-profile
tensorboard --logdir runs/fill-circle-100steps-20230411_165612/
```

Then you can inspect the profile at [http://localhost:6006/#profile](http://localhost:6006/#profile).

<Tip warning={true}>

If you run into version conflicts with the plugin, try uninstalling and reinstalling all versions of TensorFlow and Tensorboard. The debugging functionality of the profile plugin is still experimental, and not all views are fully functional. The `trace_viewer` cuts off events after 1M, which can result in all your device traces getting lost if for example, you profile the compilation step by accident.

</Tip>

```bash
python3 train_controlnet_flax.py \
--pretrained_model_name_or_path=$MODEL_DIR \
--output_dir=$OUTPUT_DIR \
--dataset_name=fusing/fill50k \
--resolution=512 \
--learning_rate=1e-5 \
--validation_image "./conditioning_image_1.png" "./conditioning_image_2.png" \
--validation_prompt "red circle with blue background" "cyan circle with brown floral background" \
--validation_steps=1000 \
--train_batch_size=2 \
--revision="non-ema" \
--from_pt \
--report_to="wandb" \
--tracker_project_name=$HUB_MODEL_ID \
--num_train_epochs=11 \
--push_to_hub \
--hub_model_id=$HUB_MODEL_ID
```

</hfoption>
</hfoptions>

Once training is complete, you can use your newly trained model for inference!

```py
Expand Down
Loading