Initial implementation of ControlNet and MultiControlNet support. #3156

GreggHelt2 · 2023-04-09T12:40:25Z

This replaces PR #3085 (ControlNet backend first draft)

- This commit fixes the update script to work again, as well as fixing the ambiguity between updating to a tag and updating to a branch.

- Compensatory change to the CLI that prevents it from crashing when it tries to import a model. - Bug introduced when the "convert" option removed from the model manager.

GreggHelt2 · 2023-04-09T12:56:23Z

So there are definitely some unresolved architectural questions.
Foremost on my mind, I am very tempted to push the added code in txt2img.py down into base.py (base Generator class). Current implementation only adds ControlNet support to Txt2Img, but moving code down in base.py I think is a clean way to add support to Img2Img, Inpaint, etc. without code duplication. However currently Generator is an abstract base class with no get_make_image() implementation, which is where this code would likely go, so I was hoping to get feedback from others on this idea.

damian0815 · 2023-04-09T18:37:33Z

invokeai/backend/stable_diffusion/diffusers_pipeline.py

        #     i.e. before or after passing it to InvokeAIDiffuserComponent
        latent_model_input = self.scheduler.scale_model_input(latents, timestep)

+        if (self.control_model is not None) and (kwargs.get("control_image") is not None):


i haven't looked closely at how this works, but maybe this chunk of code can go inside self.invokeai_diffuser.do_diffusion_step()?

damian0815 · 2023-04-09T18:38:03Z

invokeai/backend/stable_diffusion/diffusers_pipeline.py

        noise: torch.Tensor,
        callback: Callable[[PipelineIntermediateState], None] = None,
        run_id=None,
+        **kwargs,


rather than propagating kwargs can you put the extra conditioning info into the ExtraConditioningInfo struct?

@artsparkAI also commented in previous PR #3085 that it'd be better to avoid kwargs.
We could possibly could put ControlNet residuals output in ExtraConditioningInfo. But it would add two new vars in the struct, down_block_additional_residuals and mid_block_additional_residuals. These are passed as separate args to the base model diffusers UNet2DConditionalModel.forward() call, and are handled differently from the cross_attention_kwargs arg. Does that still sound better than using kwargs?

yes, definitely better than kwargs. the point of ExtraConditioningInfo is exactly to store this kind of stuff. wrap them in their own struct ControlNetArgs, add an instance of that to ExtraConditioningInfo that defaults to None.

…InvokeAI.

Kyle0654 · 2023-04-10T00:19:59Z

Regarding your comment here - check out the latents nodes. None of them go through base.py or higher layers anymore. It'd be preferable to not go through those (and deprecate them) and generally make it easier to implement new features directly in nodes. We're not all the way there, but that's the ideal direction.

When running this app first time in WSL2 environment, which is notoriously slow when it comes to IO, computing the SHAs of the models takes an eternity. Computing shas for sd2.1 ``` | Calculating sha256 hash of model files | sha256 = 1e4ce085102fe6590d41ec1ab6623a18c07127e2eca3e94a34736b36b57b9c5e (49 files hashed in 510.87s) ``` I increased the chunk size to 16MB reduce the number of round trips for loading the data. New results: ``` | Calculating sha256 hash of model files | sha256 = 1e4ce085102fe6590d41ec1ab6623a18c07127e2eca3e94a34736b36b57b9c5e (49 files hashed in 59.89s) ``` Higher values don't seem to make an impact.

the name of this was changed at some point. nodes still used the old name, so scheduler selection did nothing. simple fix.

…InvokeAI.

…i/InvokeAI into feat/controlnet_backend

- add invocation schema customisation done via fastapi's `Config` class and `schema_extra`. when using `Config`, inherit from `InvocationConfig` to get type hints. where it makes sense - like for all math invocations - define a `MathInvocationConfig` class and have all invocations inherit from it. this customisation can provide any arbitrary additional data to the UI. currently it provides tags and field type hints. this is necessary for `model` type fields, which are actually string fields. without something like this, we can't reliably differentiate `model` fields from normal `string` fields. can also be used for future field types. all invocations now have tags, and all `model` fields have ui type hints. - fix model handling for invocations added a helper to fall back to the default model if an invalid model name is chosen. model names in graphs now work. - fix latents progress callback noticed this wasn't correct while working on everything else.

Co-authored-by: techybrain-dev <techybrain.dev@gmail.com>

Currently translated at 73.0% (368 of 504 strings) Co-authored-by: 唐澤克幸 <4ranci0ne@gmail.com> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/ja/ Translation: InvokeAI/Web UI

Co-authored-by: Bouncyknighter <gebifirm@gmail.com>

Currently translated at 18.2% (92 of 504 strings) translationBot(ui): added translation (Finnish) Co-authored-by: Juuso V <juuso.vantola@gmail.com> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/fi/ Translation: InvokeAI/Web UI

Currently translated at 73.4% (370 of 504 strings) Co-authored-by: Jaulustus <jaulustus@gmail.com> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/de/ Translation: InvokeAI/Web UI

Currently translated at 100.0% (504 of 504 strings) Co-authored-by: Dennis <dennis@vanzoerlandt.nl> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/nl/ Translation: InvokeAI/Web UI

Currently translated at 100.0% (512 of 512 strings) translationBot(ui): update translation (Spanish) Currently translated at 100.0% (511 of 511 strings) translationBot(ui): update translation (Spanish) Currently translated at 100.0% (506 of 506 strings) Co-authored-by: gallegonovato <fran-carro@hotmail.es> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/es/ Translation: InvokeAI/Web UI

Currently translated at 100.0% (512 of 512 strings) translationBot(ui): update translation (Italian) Currently translated at 100.0% (511 of 511 strings) translationBot(ui): update translation (Italian) Currently translated at 100.0% (506 of 506 strings) Co-authored-by: Riccardo Giovanetti <riccardo.giovanetti@gmail.com> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/it/ Translation: InvokeAI/Web UI

Currently translated at 75.8% (384 of 506 strings) translationBot(ui): update translation (Russian) Currently translated at 85.5% (433 of 506 strings) Co-authored-by: mitien <mitien@gmail.com> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/ru/ Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/uk/ Translation: InvokeAI/Web UI

Currently translated at 100.0% (512 of 512 strings) translationBot(ui): update translation (Russian) Currently translated at 100.0% (512 of 512 strings) translationBot(ui): update translation (English) Currently translated at 100.0% (512 of 512 strings) translationBot(ui): update translation (Ukrainian) Currently translated at 100.0% (506 of 506 strings) translationBot(ui): update translation (Russian) Currently translated at 100.0% (506 of 506 strings) translationBot(ui): update translation (Russian) Currently translated at 100.0% (506 of 506 strings) Co-authored-by: System X - Files <vasyasos@gmail.com> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/en/ Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/ru/ Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/uk/ Translation: InvokeAI/Web UI

Currently translated at 81.8% (414 of 506 strings) translationBot(ui): update translation (German) Currently translated at 80.8% (409 of 506 strings) Co-authored-by: Alexander Eichhorn <pfannkuchensack@einfach-doof.de> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/de/ Translation: InvokeAI/Web UI

Currently translated at 34.7% (176 of 506 strings) translationBot(ui): added translation (Swedish) Co-authored-by: figgefigge <qvintuz@gmail.com> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/sv/ Translation: InvokeAI/Web UI

Currently translated at 81.8% (414 of 506 strings) Co-authored-by: Fabian Bahl <fabian98@bahl-netz.de> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/de/ Translation: InvokeAI/Web UI

Currently translated at 75.0% (380 of 506 strings) Co-authored-by: Patrick Tien <ivetien@outlook.com> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/zh_Hans/ Translation: InvokeAI/Web UI

Currently translated at 11.3% (58 of 512 strings) translationBot(ui): added translation (Turkish) Co-authored-by: ismail ihsan bülbül <e-ben@msn.com> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/tr/ Translation: InvokeAI/Web UI

…reducing CPU usage significantly (#3256) I noticed that the current invokeai-new.py was using almost all of a CPU core. After a bit of profileing I noticed that there were many thousands of calls to epoll() which suggested to me that something wasn't sleeping properly in asyncio's loop. A bit of further investigation with Python profiling revealed that the __dispatch_from_queue() method in FastAPIEventService (app/api/events.py:33) was also being called thousands of times. I believe the asyncio.sleep(0.001) in that method is too aggressive (it means that the queue will be polled every 1ms) and that 0.1 (100ms) is still entirely reasonable.

- This is a port of #3184 to the main branch

…3101) - Fix the update script to work again and fixes the ambiguity between when a user wants to update to a tag vs updating to a branch, by making these two operations explicitly separate. - Remove dangling functions and arguments related to legacy checkpoint conversion. These are no longer needed now that all legacy models are either converted at import time, or on-the-fly in RAM.

Prevent legacy CLI crash caused by removal of convert option - Compensatory change to the CLI that prevents it from crashing when it tries to import a model. - Bug introduced when the "convert" option removed from the model manager.

When gallery was empty (and there is therefore no selected image), no progress images were displayed. - fix by correcting the logic in CurrentImageDisplay - also fix app crash introduced by fixing the first bug

…olNet support for InvokeAI.

…tion.

…i/InvokeAI into feat/controlnet_backend

hipsterusername · 2023-05-10T12:36:01Z

@GreggHelt2 - Just checking in here, since Multi-controlnet was seeming close to ready - This almost ready to merge in?

lstein added 7 commits April 2, 2023 11:08

fix invokeai-update script

f682fb8

- This commit fixes the update script to work again, as well as fixing the ambiguity between updating to a tag and updating to a branch.

Merge branch 'main' into bugfix/release-updater

b917ffa

Merge branch 'main' into bugfix/release-updater

28f75d8

prevent legacy CLI crash caused by removal of convert option

32857d8

- Compensatory change to the CLI that prevents it from crashing when it tries to import a model. - Bug introduced when the "convert" option removed from the model manager.

Merge branch 'main' into bugfix/prevent-cli-crash

bc2b950

Merge branch 'main' into bugfix/prevent-cli-crash

8334757

Merge branch 'main' into bugfix/release-updater

cd1b350

GreggHelt2 requested review from JPPhoto, blessedcoolant, damian0815, keturn and lstein as code owners April 9, 2023 12:40

GreggHelt2 mentioned this pull request Apr 9, 2023

ControlNet backend first draft #3085

Closed

Merge branch 'main' into bugfix/prevent-cli-crash

cee159d

damian0815 reviewed Apr 9, 2023

View reviewed changes

Initial implementation of ControlNet and MultiControlNet support for …

5b11bcd

…InvokeAI.

GreggHelt2 force-pushed the feat/controlnet_backend branch from 385c4be to 5b11bcd Compare April 9, 2023 19:42

Increase chunk size when computing SHAs

de189f2

lstein and others added 6 commits April 9, 2023 22:29

fix(nodes): add missing type to ImageField

5bd0bb6

fix(nodes): sampler_name --> scheduler

dad3a7f

the name of this was changed at some point. nodes still used the old name, so scheduler selection did nothing. simple fix.

feat(nodes): fix typo in PasteImageInvocation

427db7c

Initial implementation of ControlNet and MultiControlNet support for …

5ab2164

…InvokeAI.

Merge branch 'feat/controlnet_backend' of https://github.com/invoke-a…

8ac8be4

…i/InvokeAI into feat/controlnet_backend

GreggHelt2 requested a review from Kyle0654 as a code owner April 10, 2023 22:40

psychedelicious added 2 commits April 11, 2023 12:13

fix(nodes): move InvocationConfig to baseinvocation.py

1f2c1e1

techybrain-dev and others added 28 commits April 24, 2023 16:05

translationBot(ui): added translation (Vietnamese)

1e05538

Co-authored-by: techybrain-dev <techybrain.dev@gmail.com>

translationBot(ui): update translation (Japanese)

17d0920

Currently translated at 73.0% (368 of 504 strings) Co-authored-by: 唐澤克幸 <4ranci0ne@gmail.com> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/ja/ Translation: InvokeAI/Web UI

translationBot(ui): added translation (Mongolian)

37096a6

Co-authored-by: Bouncyknighter <gebifirm@gmail.com>

translationBot(ui): update translation (German)

b8dc900

Currently translated at 73.4% (370 of 504 strings) Co-authored-by: Jaulustus <jaulustus@gmail.com> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/de/ Translation: InvokeAI/Web UI

translationBot(ui): update translation (Dutch)

fd00d11

Currently translated at 100.0% (504 of 504 strings) Co-authored-by: Dennis <dennis@vanzoerlandt.nl> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/nl/ Translation: InvokeAI/Web UI

translationBot(ui): update translation (German)

ef605cd

Currently translated at 81.8% (414 of 506 strings) Co-authored-by: Fabian Bahl <fabian98@bahl-netz.de> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/de/ Translation: InvokeAI/Web UI

translationBot(ui): update translation (Chinese (Simplified))

2b58ce4

Currently translated at 75.0% (380 of 506 strings) Co-authored-by: Patrick Tien <ivetien@outlook.com> Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/zh_Hans/ Translation: InvokeAI/Web UI

Merge branch 'main' into reduce-event-polling

a537231

Merge branch 'main' into lstein/enhance/diffusers-0.15

48ce256

Merge branch 'main' into bugfix/prevent-cli-crash

4fa5c96

update to diffusers 0.15 and fix code for name changes (#3201)

fe12938

- This is a port of #3184 to the main branch

Merge branch 'main' into bugfix/prevent-cli-crash

31a904b

[Bugfix] prevent cli crash (#3132)

c2a0e8a

Prevent legacy CLI crash caused by removal of convert option - Compensatory change to the CLI that prevents it from crashing when it tries to import a model. - Bug introduced when the "convert" option removed from the model manager.

fix(ui): fix no progress images when gallery is empty

f03cb8f

fix(ui): fix no progress images when gallery is empty (#3268)

76e5d05

When gallery was empty (and there is therefore no selected image), no progress images were displayed. - fix by correcting the logic in CurrentImageDisplay - also fix app crash introduced by fixing the first bug

Resolving rebase: Initial implementation of ControlNet and MultiContr…

6bd74de

…olNet support for InvokeAI.

Added minimal viable node support for ControlNet in TextToImageInvoca…

cf6941f

…tion.

Merge branch 'feat/controlnet_backend' of https://github.com/invoke-a…

448f6a0

…i/InvokeAI into feat/controlnet_backend

GreggHelt2 closed this May 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Initial implementation of ControlNet and MultiControlNet support. #3156

Initial implementation of ControlNet and MultiControlNet support. #3156

Uh oh!

GreggHelt2 commented Apr 9, 2023 •

edited

Loading

Uh oh!

GreggHelt2 commented Apr 9, 2023

Uh oh!

damian0815 Apr 9, 2023

Uh oh!

damian0815 Apr 9, 2023

Uh oh!

GreggHelt2 Apr 9, 2023

Uh oh!

damian0815 Apr 9, 2023

Uh oh!

Kyle0654 commented Apr 10, 2023

Uh oh!

hipsterusername commented May 10, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Initial implementation of ControlNet and MultiControlNet support. #3156

Initial implementation of ControlNet and MultiControlNet support. #3156

Uh oh!

Conversation

GreggHelt2 commented Apr 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GreggHelt2 commented Apr 9, 2023

Uh oh!

damian0815 Apr 9, 2023

Choose a reason for hiding this comment

Uh oh!

damian0815 Apr 9, 2023

Choose a reason for hiding this comment

Uh oh!

GreggHelt2 Apr 9, 2023

Choose a reason for hiding this comment

Uh oh!

damian0815 Apr 9, 2023

Choose a reason for hiding this comment

Uh oh!

Kyle0654 commented Apr 10, 2023

Uh oh!

hipsterusername commented May 10, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

GreggHelt2 commented Apr 9, 2023 •

edited

Loading