Skip to content

Commit d7c4ae6

Browse files
authored
Add SDXL long weighted prompt pipeline (replace pr:4629) (#4661)
* Add SDXL long weighted prompt pipeline * Add SDXL long weighted prompt pipeline usage sample in the readme document * Add SDXL long weighted prompt pipeline usage sample in the readme document, add result image
1 parent 67ea2b7 commit d7c4ae6

File tree

2 files changed

+1337
-1
lines changed

2 files changed

+1337
-1
lines changed

examples/community/README.md

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,8 @@ If a community doesn't work as expected, please open an issue and ping the autho
3939
| CLIP Guided Images Mixing Stable Diffusion Pipeline | Сombine images using usual diffusion models. | [CLIP Guided Images Mixing Using Stable Diffusion](#clip-guided-images-mixing-with-stable-diffusion) | - | [Karachev Denis](https://github.com/TheDenk) |
4040
| TensorRT Stable Diffusion Inpainting Pipeline | Accelerates the Stable Diffusion Inpainting Pipeline using TensorRT | [TensorRT Stable Diffusion Inpainting Pipeline](#tensorrt-inpainting-stable-diffusion-pipeline) | - | [Asfiya Baig](https://github.com/asfiyab-nvidia) |
4141
| IADB Pipeline | Implementation of [Iterative α-(de)Blending: a Minimalist Deterministic Diffusion Model](https://arxiv.org/abs/2305.03486) | [IADB Pipeline](#iadb-pipeline) | - | [Thomas Chambon](https://github.com/tchambon)
42-
| Zero1to3 Pipeline | Implementation of [Zero-1-to-3: Zero-shot One Image to 3D Object](https://arxiv.org/abs/2303.11328) | [Zero1to3 Pipeline](#Zero1to3-pipeline) | - | [Xin Kong](https://github.com/kxhit)
42+
| Zero1to3 Pipeline | Implementation of [Zero-1-to-3: Zero-shot One Image to 3D Object](https://arxiv.org/abs/2303.11328) | [Zero1to3 Pipeline](#Zero1to3-pipeline) | - | [Xin Kong](https://github.com/kxhit) |
43+
Stable Diffusion XL Long Weighted Prompt Pipeline | A pipeline support unlimited length of prompt and negative prompt, use A1111 style of prompt weighting | [Stable Diffusion XL Long Weighted Prompt Pipeline](#stable-diffusion-xl-long-weighted-prompt-pipeline) | - | [Andrew Zhu](https://xhinker.medium.com/) |
4344

4445

4546
To load a custom pipeline you just need to pass the `custom_pipeline` argument to `DiffusionPipeline`, as one of the files in `diffusers/examples/community`. Feel free to send a PR with your own pipelines, we will merge them quickly.
@@ -1529,6 +1530,44 @@ CLIP guided stable diffusion images mixing pipline allows to combine two images
15291530
This approach is using (optional) CoCa model to avoid writing image description.
15301531
[More code examples](https://github.com/TheDenk/images_mixing)
15311532

1533+
1534+
### Stable Diffusion XL Long Weighted Prompt Pipeline
1535+
1536+
This SDXL pipeline support unlimted length prompt and negative prompt, compatible with A1111 prompt weighted style.
1537+
1538+
You can provide both `prompt` and `prompt_2`. if only one prompt is provided, `prompt_2` will be a copy of the provided `prompt`. Here is a sample code to use this pipeline.
1539+
1540+
```python
1541+
from diffusers import DiffusionPipeline
1542+
import torch
1543+
1544+
pipe = DiffusionPipeline.from_pretrained(
1545+
"stabilityai/stable-diffusion-xl-base-1.0"
1546+
, torch_dtype = torch.float16
1547+
, use_safetensors = True
1548+
, variant = "fp16"
1549+
, custom_pipeline = "lpw_stable_diffusion_xl",
1550+
)
1551+
1552+
prompt = "photo of a cute (white) cat running on the grass"*20
1553+
prompt2 = "chasing (birds:1.5)"*20
1554+
prompt = f"{prompt},{prompt2}"
1555+
neg_prompt = "blur, low quality, carton, animate"
1556+
1557+
pipe.to("cuda")
1558+
images = pipe(
1559+
prompt = prompt
1560+
, negative_prompt = neg_prompt
1561+
).images[0]
1562+
1563+
pipe.to("cpu")
1564+
torch.cuda.empty_cache()
1565+
images
1566+
```
1567+
1568+
In the above code, the `prompt2` is appended to the `prompt`, which is more than 77 tokens. "birds" are showing up in the result.
1569+
![Stable Diffusion XL Long Weighted Prompt Pipeline sample](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/sdxl_long_weighted_prompt.png)
1570+
15321571
## Example Images Mixing (with CoCa)
15331572
```python
15341573
import requests

0 commit comments

Comments
 (0)