Legacy Inpainting Pipeline for Onnx Models #1237

ctsims · 2022-11-10T00:16:48Z

Issue: #917

Enables parity for onnx platform users to access the legacy pipeline for inpainting, which is required for inpainting with the broader array of models which aren't finetuned for inpainting.

Note: I wasn't sure how to sequence getting the assets for the tests included. They are in a PR into where I assume the best place would be, given the locality of the other test assets.

Example output against the common inpainting example:

HuggingFaceDocBuilderDev · 2022-11-10T00:20:10Z

The documentation is not available anymore as the PR was closed or merged.

HuggingFaceDocBuilderDev · 2022-11-10T16:49:29Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

patrickvonplaten · 2022-11-16T20:41:57Z

@ctsims feel free to let @anton-l and me know once this PR is ready for review :-)

HuggingFaceDocBuilderDev · 2022-11-16T23:32:12Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

anton-l · 2022-11-17T14:52:14Z

@ctsims a quick merge from main should fix the remaining style checks 👍

ctsims · 2022-11-17T16:37:56Z

@anton-l Style and Quality should be set. Sorry for the slog on that, the checks don't seem to be quite consistent for me locally.

Everything else should be good to go. What's the best way to coordinate on the outcome of the slow test? It's consistent for me locally against the onnx sd14 model, but not sure if there are any inconsistencies which would affect the seeding or deterministic outcome between my setup and the build runner.

anton-l · 2022-11-17T16:57:40Z

@ctsims thank you! Checking the slow tests locally...

anton-l

Ok, the slow tests will probably fail on main but no worries, I'll update the reference image in a follow-up PR

tests/pipelines/stable_diffusion/test_onnx_stable_diffusion_inpaint_legacy.py

src/diffusers/pipelines/stable_diffusion/pipeline_onnx_stable_diffusion_inpaint_legacy.py

ctsims · 2022-11-17T21:43:51Z

@anton-l Thanks so much for the quick review. Updated the two points of feedback, let me know if there's anything else to be addressed.

anton-l

Thank you @ctsims! Added one more thing I've missed and then we're good to go if the tests pass for you :)

src/diffusers/pipelines/stable_diffusion/pipeline_onnx_stable_diffusion_inpaint_legacy.py

Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>

anton-l · 2022-11-18T15:33:03Z

Awesome, let's launch it! 🚀

) * add AudioDiffusionPipeline and LatentAudioDiffusionPipeline * add docs to toc * fix tests * fix tests * fix tests * fix tests * fix tests * Update pr_tests.yml Fix tests * parent 499ff34 author teticio <teticio@gmail.com> 1668765652 +0000 committer teticio <teticio@gmail.com> 1669041721 +0000 parent 499ff34 author teticio <teticio@gmail.com> 1668765652 +0000 committer teticio <teticio@gmail.com> 1669041704 +0000 add colab notebook [Flax] Fix loading scheduler from subfolder (#1319) [FLAX] Fix loading scheduler from subfolder Fix/Enable all schedulers for in-painting (#1331) * inpaint fix k lms * onnox as well * up Correct path to schedlure (#1322) * [Examples] Correct path * uP Avoid nested fix-copies (#1332) * Avoid nested `# Copied from` statements during `make fix-copies` * style Fix img2img speed with LMS-Discrete Scheduler (#896) Casting `self.sigmas` into a different dtype (the one of original_samples) is not advisable. In my img2img pipeline this leads to a long running time in the `integrate.quad` call later on- by long I mean more than 10x slower. Co-authored-by: Anton Lozhkov <anton@huggingface.co> Fix the order of casts for onnx inpainting (#1338) Legacy Inpainting Pipeline for Onnx Models (#1237) * Add legacy inpainting pipeline compatibility for onnx * remove commented out line * Add onnx legacy inpainting test * Fix slow decorators * pep8 styling * isort styling * dummy object * ordering consistency * style * docstring styles * Refactor common prompt encoding pattern * Update tests to permanent repository home * support all available schedulers until ONNX IO binding is available Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com> * updated styling from PR suggested feedback Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com> Jax infer support negative prompt (#1337) * support negative prompts in sd jax pipeline * pass batched neg_prompt * only encode when negative prompt is None Co-authored-by: Juan Acevedo <jfacevedo@google.com> Update README.md: Minor change to Imagic code snippet, missing dir error (#1347) Minor change to Imagic Readme Missing dir causes an error when running the example code. make style change the sample model (#1352) * Update alt_diffusion.mdx * Update alt_diffusion.mdx Add bit diffusion [WIP] (#971) * Create bit_diffusion.py Bit diffusion based on the paper, arXiv:2208.04202, Chen2022AnalogBG * adding bit diffusion to new branch ran tests * tests * tests * tests * tests * removed test folders + added to README * Update README.md Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * move Mel to module in pipeline construction, make librosa optional * fix imports * fix copy & paste error in comment * fix style * add missing register_to_config * fix class docstrings * fix class docstrings * tweak docstrings * tweak docstrings * update slow test * put trailing commas back * respect alphabetical order * remove LatentAudioDiffusion, make vqvae optional * move Mel from models back to pipelines :-) * allow loading of pretrained audiodiffusion models * fix tests * fix dummies * remove reference to latent_audio_diffusion in docs * unused import * inherit from SchedulerMixin to make loadable * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

…ce#1334 (huggingface#1426) * add AudioDiffusionPipeline and LatentAudioDiffusionPipeline * add docs to toc * fix tests * fix tests * fix tests * fix tests * fix tests * Update pr_tests.yml Fix tests * parent 499ff34 author teticio <teticio@gmail.com> 1668765652 +0000 committer teticio <teticio@gmail.com> 1669041721 +0000 parent 499ff34 author teticio <teticio@gmail.com> 1668765652 +0000 committer teticio <teticio@gmail.com> 1669041704 +0000 add colab notebook [Flax] Fix loading scheduler from subfolder (huggingface#1319) [FLAX] Fix loading scheduler from subfolder Fix/Enable all schedulers for in-painting (huggingface#1331) * inpaint fix k lms * onnox as well * up Correct path to schedlure (huggingface#1322) * [Examples] Correct path * uP Avoid nested fix-copies (huggingface#1332) * Avoid nested `# Copied from` statements during `make fix-copies` * style Fix img2img speed with LMS-Discrete Scheduler (huggingface#896) Casting `self.sigmas` into a different dtype (the one of original_samples) is not advisable. In my img2img pipeline this leads to a long running time in the `integrate.quad` call later on- by long I mean more than 10x slower. Co-authored-by: Anton Lozhkov <anton@huggingface.co> Fix the order of casts for onnx inpainting (huggingface#1338) Legacy Inpainting Pipeline for Onnx Models (huggingface#1237) * Add legacy inpainting pipeline compatibility for onnx * remove commented out line * Add onnx legacy inpainting test * Fix slow decorators * pep8 styling * isort styling * dummy object * ordering consistency * style * docstring styles * Refactor common prompt encoding pattern * Update tests to permanent repository home * support all available schedulers until ONNX IO binding is available Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com> * updated styling from PR suggested feedback Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com> Jax infer support negative prompt (huggingface#1337) * support negative prompts in sd jax pipeline * pass batched neg_prompt * only encode when negative prompt is None Co-authored-by: Juan Acevedo <jfacevedo@google.com> Update README.md: Minor change to Imagic code snippet, missing dir error (huggingface#1347) Minor change to Imagic Readme Missing dir causes an error when running the example code. make style change the sample model (huggingface#1352) * Update alt_diffusion.mdx * Update alt_diffusion.mdx Add bit diffusion [WIP] (huggingface#971) * Create bit_diffusion.py Bit diffusion based on the paper, arXiv:2208.04202, Chen2022AnalogBG * adding bit diffusion to new branch ran tests * tests * tests * tests * tests * removed test folders + added to README * Update README.md Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * move Mel to module in pipeline construction, make librosa optional * fix imports * fix copy & paste error in comment * fix style * add missing register_to_config * fix class docstrings * fix class docstrings * tweak docstrings * tweak docstrings * update slow test * put trailing commas back * respect alphabetical order * remove LatentAudioDiffusion, make vqvae optional * move Mel from models back to pipelines :-) * allow loading of pretrained audiodiffusion models * fix tests * fix dummies * remove reference to latent_audio_diffusion in docs * unused import * inherit from SchedulerMixin to make loadable * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Add legacy inpainting pipeline compatibility for onnx * remove commented out line * Add onnx legacy inpainting test * Fix slow decorators * pep8 styling * isort styling * dummy object * ordering consistency * style * docstring styles * Refactor common prompt encoding pattern * Update tests to permanent repository home * support all available schedulers until ONNX IO binding is available Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com> * updated styling from PR suggested feedback Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>

…ce#1334 (huggingface#1426) * add AudioDiffusionPipeline and LatentAudioDiffusionPipeline * add docs to toc * fix tests * fix tests * fix tests * fix tests * fix tests * Update pr_tests.yml Fix tests * parent 499ff34 author teticio <teticio@gmail.com> 1668765652 +0000 committer teticio <teticio@gmail.com> 1669041721 +0000 parent 499ff34 author teticio <teticio@gmail.com> 1668765652 +0000 committer teticio <teticio@gmail.com> 1669041704 +0000 add colab notebook [Flax] Fix loading scheduler from subfolder (huggingface#1319) [FLAX] Fix loading scheduler from subfolder Fix/Enable all schedulers for in-painting (huggingface#1331) * inpaint fix k lms * onnox as well * up Correct path to schedlure (huggingface#1322) * [Examples] Correct path * uP Avoid nested fix-copies (huggingface#1332) * Avoid nested `# Copied from` statements during `make fix-copies` * style Fix img2img speed with LMS-Discrete Scheduler (huggingface#896) Casting `self.sigmas` into a different dtype (the one of original_samples) is not advisable. In my img2img pipeline this leads to a long running time in the `integrate.quad` call later on- by long I mean more than 10x slower. Co-authored-by: Anton Lozhkov <anton@huggingface.co> Fix the order of casts for onnx inpainting (huggingface#1338) Legacy Inpainting Pipeline for Onnx Models (huggingface#1237) * Add legacy inpainting pipeline compatibility for onnx * remove commented out line * Add onnx legacy inpainting test * Fix slow decorators * pep8 styling * isort styling * dummy object * ordering consistency * style * docstring styles * Refactor common prompt encoding pattern * Update tests to permanent repository home * support all available schedulers until ONNX IO binding is available Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com> * updated styling from PR suggested feedback Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com> Jax infer support negative prompt (huggingface#1337) * support negative prompts in sd jax pipeline * pass batched neg_prompt * only encode when negative prompt is None Co-authored-by: Juan Acevedo <jfacevedo@google.com> Update README.md: Minor change to Imagic code snippet, missing dir error (huggingface#1347) Minor change to Imagic Readme Missing dir causes an error when running the example code. make style change the sample model (huggingface#1352) * Update alt_diffusion.mdx * Update alt_diffusion.mdx Add bit diffusion [WIP] (huggingface#971) * Create bit_diffusion.py Bit diffusion based on the paper, arXiv:2208.04202, Chen2022AnalogBG * adding bit diffusion to new branch ran tests * tests * tests * tests * tests * removed test folders + added to README * Update README.md Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * move Mel to module in pipeline construction, make librosa optional * fix imports * fix copy & paste error in comment * fix style * add missing register_to_config * fix class docstrings * fix class docstrings * tweak docstrings * tweak docstrings * update slow test * put trailing commas back * respect alphabetical order * remove LatentAudioDiffusion, make vqvae optional * move Mel from models back to pipelines :-) * allow loading of pretrained audiodiffusion models * fix tests * fix dummies * remove reference to latent_audio_diffusion in docs * unused import * inherit from SchedulerMixin to make loadable * Apply suggestions from code review * Apply suggestions from code review Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

ctsims added 3 commits November 9, 2022 13:08

Add legacy inpainting pipeline compatibility for onnx

994908e

remove commented out line

4ff0f2b

Add onnx legacy inpainting test

46677a4

Fix slow decorators

e4e672d

patrickvonplaten assigned anton-l Nov 16, 2022

ctsims added 4 commits November 16, 2022 17:04

pep8 styling

45cf485

isort styling

7884774

dummy object

e610263

ordering consistency

49e7a0d

ctsims added 3 commits November 17, 2022 09:04

Merge branch 'main' into ctsims/legacy_inpaint_onnx

684bf26

style

874a58c

docstring styles

e6c5293

ctsims changed the title ~~[WIP] Legacy Inpainting Pipeline for Onnx Models~~ Legacy Inpainting Pipeline for Onnx Models Nov 17, 2022

anton-l approved these changes Nov 17, 2022

View reviewed changes

tests/pipelines/stable_diffusion/test_onnx_stable_diffusion_inpaint_legacy.py Outdated Show resolved Hide resolved

src/diffusers/pipelines/stable_diffusion/pipeline_onnx_stable_diffusion_inpaint_legacy.py Outdated Show resolved Hide resolved

ctsims added 2 commits November 17, 2022 14:01

Refactor common prompt encoding pattern

da052a3

Update tests to permanent repository home

f0523ba

anton-l approved these changes Nov 18, 2022

View reviewed changes

src/diffusers/pipelines/stable_diffusion/pipeline_onnx_stable_diffusion_inpaint_legacy.py Outdated Show resolved Hide resolved

ctsims and others added 2 commits November 18, 2022 09:50

support all available schedulers until ONNX IO binding is available

b597fb5

Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>

updated styling from PR suggested feedback

6d98ba7

anton-l merged commit 3022090 into huggingface:main Nov 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Legacy Inpainting Pipeline for Onnx Models #1237

Legacy Inpainting Pipeline for Onnx Models #1237

ctsims commented Nov 10, 2022 •

edited

HuggingFaceDocBuilderDev commented Nov 10, 2022 •

edited

HuggingFaceDocBuilderDev commented Nov 10, 2022

patrickvonplaten commented Nov 16, 2022

HuggingFaceDocBuilderDev commented Nov 16, 2022

anton-l commented Nov 17, 2022

ctsims commented Nov 17, 2022

anton-l commented Nov 17, 2022

anton-l left a comment •

edited

ctsims commented Nov 17, 2022

anton-l left a comment

anton-l commented Nov 18, 2022

Legacy Inpainting Pipeline for Onnx Models #1237

Legacy Inpainting Pipeline for Onnx Models #1237

Conversation

ctsims commented Nov 10, 2022 • edited

HuggingFaceDocBuilderDev commented Nov 10, 2022 • edited

HuggingFaceDocBuilderDev commented Nov 10, 2022

patrickvonplaten commented Nov 16, 2022

HuggingFaceDocBuilderDev commented Nov 16, 2022

anton-l commented Nov 17, 2022

ctsims commented Nov 17, 2022

anton-l commented Nov 17, 2022

anton-l left a comment • edited

Choose a reason for hiding this comment

ctsims commented Nov 17, 2022

anton-l left a comment

Choose a reason for hiding this comment

anton-l commented Nov 18, 2022

ctsims commented Nov 10, 2022 •

edited

HuggingFaceDocBuilderDev commented Nov 10, 2022 •

edited

anton-l left a comment •

edited