feat(nodes): better controlnet processors #6021
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some processors, like Canny, didn't use `detect_resolution`. The resultant control images were then resized by the processors from 512x512 to the desired dimensions. The result is that the control images are the right size, but very low quality. Using detect_resolution fixes this.
These all support controlnet processors. - `pil_to_cv2` - `cv2_to_pil` - `pil_to_np` - `np_to_pil` - `normalize_image_channel_count` (a readable version of `HWC3` from the controlnet repo) - `fit_image_to_resolution` (a readable version of `resize_image` from the controlnet repo) - `non_maximum_suppression` (a readable version of `nms` from the controlnet repo) - `safe_step` (a readable version of `safe_step` from the controlnet repo)
Adapted from controlnet processors package fix: do final resize in canny processor canny
Adapted from controlnet repo
psychedelicious
requested review from
blessedcoolant,
maryhipp,
hipsterusername,
lstein,
GreggHelt2,
brandonrising and
RyanJDick
as code owners
github-actions
bot
added
python
hipsterusername
approved these changes
Mar 21, 2024
|
Nailed it! |
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
I also added
detect_resolutionto:I didn't fully understand what
detect_resolutionandimage_resolutiondid until I read through the code. These call the same function to resize the image to a given "resolution", maintaining the aspect ratio.detect_resolution: This takes the input image and resizes it to the given resolution before processing.image_resolution: This does the same thing, but after the image was processed.Both
detect_resolutionandimage_resolutionreally need to be correct for the control image to work as it should. Ideally, these numbers match exactly the control image and output image, respectively, but the logic helpfully allows for some fudging.Examples
Here's my input control image:
And my settings (I will be changing only the two resolution settings) - I did a bit of tweaking to get the edges to not be too busy:
detect_resolution512,image_resolution512detect_resolution512,image_resolution1024Note that the control image is still very low quality - that's because it starts out @ 512 and is then resized to 1024.
detect_resolution1024,image_resolution1024Note that the control image is now very sharp.
Removing dependency on
controlnet_auxThis package provides the controlnet processors, but it bundles up model loading in a rather inconvenient, inaccessible way. We will want to manage the loading of the controlnet processor models ourselves at some point, which may be tricky.
The package is also very messy, being a copy-paste from the OG controlnet repo (unmaintained).
controlnet_auxhas seen some activity from non-HF contributors, but with Patrick no longer at HF, it's not clear how much support it will get.As I was working on adding
detect_resolution, I looked atcontrolnet_aux's source and recoiled in horror. I was compelled to clean it up a bit.So far, I've adapted these things:
The code is cleaned up, with type annotations as best as I could figure. I did back-to-back tests of the processors before and after my adaptations and confirmed the output images were perfect matches, so I'm confident I didn't break anything.
Maybe we should contribute this back to controlnet_aux at some point? Or make a separate package?
Related Issues / Discussions
It's possible that the lack of image and detect resolution contributed to the issues noted in #5943.
QA Instructions
Try out the new settings on Canny. You should get markedly better results for controlnet and T2I adapter if you have the resolutions correct.
Merge Plan
N/A
Checklist