-
Notifications
You must be signed in to change notification settings - Fork 5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP][Docs] Use DiffusionPipeline Instead of Child Classes when Loading Pipeline #2809
[WIP][Docs] Use DiffusionPipeline Instead of Child Classes when Loading Pipeline #2809
Conversation
…g a checkpoint using from_pretrained() instead of a child class (e.g. StableDiffusionPipeline) where possible.
The documentation is not available anymore as the PR was closed or merged. |
Sorry, I just realized that the docs in |
Looked at the docs for
Left unchanged because I think the specific pipelines (e.g.
Since this page discusses the Similarly to the other training docs, since these pages discuss specific pipelines, I think it makes sense to use them rather than the generic
For the first example, using
The examples discuss specific tasks (e.g. inpainting) so I think it makes sense to use the specific pipeline for that task (e.g. |
I realize that instead of documenting specific cases where a change is or is not made, it might make more sense to use a bright line rule about when using (I would also greatly appreciate it if people could give feedback on the verbosity of this PR, so that I can improve in the future.) |
Thanks for the PR @dg845 - I agree with your reasoning. I think we should use I'm in favor of merging this PR as it though, so LGTM. @yiyixuxu could you take a final look? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
…ng Pipeline (huggingface#2809) * Change the docs to use the parent DiffusionPipeline class when loading a checkpoint using from_pretrained() instead of a child class (e.g. StableDiffusionPipeline) where possible. * Run make style to fix style issues. * Change more docs to use DiffusionPipeline rather than a subclass. --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
…ng Pipeline (huggingface#2809) * Change the docs to use the parent DiffusionPipeline class when loading a checkpoint using from_pretrained() instead of a child class (e.g. StableDiffusionPipeline) where possible. * Run make style to fix style issues. * Change more docs to use DiffusionPipeline rather than a subclass. --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
This PR edits the docs to use
DiffusionPipeline
instead of its child classes (such asStableDiffusionPipeline
) where possible when loading a pipeline. The rationale behind this change is thatDiffusionPipeline
should always create a pipeline of the right class when loading a checkpoint and thus should be more user friendly. See #2135 for more discussion; in particular, this PR corresponds to item 4 in this comment.Below I've documented pages with examples that use a child class of
DiffusionPipeline
when loading a checkpoint that I've opted not to change, along with a short explanation of why I think it should stay as is./docs/source/en/using-diffusers/loading.mdx
(link)The examples here specifically discuss loading directly from the appropriate pipeline class (e.g.
StableDiffusionPipeline
)./docs/source/en/using-diffusers/schedulers.mdx
(link)The JAX/Flax example uses
FlaxStableDiffusionPipeline
; since it deals specifically with Flax, I think it makes sense to use the more specific class./docs/source/en/using-diffusers/img2img.mdx
(link)/docs/source/en/using-diffusers/inpaint.mdx
(link)/docs/source/en/using-diffusers/depth2img.mdx
(link)Since these pages talk about specific Stable Diffusion pipelines, I think it's appropriate to reference the specific pipeline class when loading the checkpoints.
/docs/source/en/using-diffusers/reproducibility.mdx
(link)The running example on this page uses
DDIMPipeline
, which I think is relevant to the discussion on reproducibility. I haven't tested the numbers produced if we substituteDiffusionPipeline
forDDIMPipeline
(presumably, they are close if not the same).Edit: in
0.15.0.dev0
the numbers are less important to the exposition, so it might make more sense to replaceDDIMPipeline
withDiffusionPipeline
./docs/source/en/using-diffusers/contribute_pipeline.mdx
(link)I think it makes sense that there is an example that discusses loading a checkpoint explicitly from a custom pipeline (in this case,
UnetSchedulerOneForwardPipeline
)./docs/source/en/using-diffusers/using_safetensors.mdx
(link)I imagine substituting
DiffusionPipeline
forStableDiffusionPipeline
in the loading speed example could make a difference to the final loading numbers (I don't have the required hardware to test this myself)./docs/source/en/optimization/onnx.mdx
(link)I think discussing the difference between
StableDiffusionPipeline
andStableDiffusionOnnxPipeline
is valuable, although perhaps there could be a note about usingDiffusionPipeline
to load the checkpoint as well.Edit: in
0.15.0.dev0
ORTStableDiffusionPipeline
replacesStableDiffusionPipeline
. I think the comment above remains valid, although I'm not sure if 'DiffusionPipelineis an acceptable substitute for
ORTStableDiffusionPipeline`./docs/source/en/optimization/habana.mdx
(link)Using
GaudiStableDiffusionPipeline
here instead of the genericDiffusionPipeline
probably makes more sense. (I'm not familiar with Habana Gaudi.)/docs/source/en/training/text_inversion.mdx
(link)/docs/source/en/training/text2image.mdx
(link)/docs/source/en/training/lora.mdx
(link)These pages specifically discuss fine-tuning a Stable Diffusion model, so it feels weird to replace
StableDiffusionPipeline
withDiffusionPipeline
.(Note that for
/docs/source/en/training/dreambooth.mdx
(link), some of the code snippets useDiffusionPipeline
and some useStableDiffusionPipeline
, so I've changed all checkpoint loading examples to useDiffusionPipeline
.)/docs/source/en/api/outputs.mdx
(link)In the pipeline loading example, we follow up a call to
from_pretrained(...)
with a call to__call__(...)
, so I think it makes sense to use a specific pipeline such asDDIMPipeline
rather thanDiffusionPipeline
(which doesn't implement a__call__
method).(Sorry for making the PR so long!)
Edit: changed the links to point to the PR docs endpoint.