[Feature Request]: Can we get native Pix2Pix model support?

[bbecausereasonss]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Trained on 1.5 (allows GPT3) natural language requests. VERY cool!

https://github.com/timothybrooks/instruct-pix2pix

https://huggingface.co/timbrooks/instruct-pix2pix

Proposed workflow

Obvious.

Additional information

No response

[HWiese1980]

Yes, please!

[Hellisotherpeople]

@AUTOMATIC1111 have you seen this yet?

[HWiese1980]

Considering how many open issues this repository has, I could imagine AUTOMATIC1111 is a little overwhelmed.

[Unstackd]

For anyone interested, a user posted on reddit that he was working on a fork of Automatic1111 with this feature. I tried it out and it seems to work.

On a related note, it looks like you can use the weighted difference trick to convert other models into InstructPix2Pix models in much the same way you can convert any model into an inpainting model.

So that means you can use this technique with other models like AnythingV3 or whatever.

You just have to make this little hacky change to extras.py so that line 185-194 becomes:
`
.

        # this enables merging an inpainting model (A) with another one (B);

        # where normal model would have 4 channels, for latenst space, inpainting model would

        # have another 4 channels for unmasked picture's latent space, plus one channel for mask, for a total of 9

 if a.shape != b.shape and a.shape[0:1] + a.shape[2:] == b.shape[0:1] + b.shape[2:]:

            if a.shape[1] == 4 and b.shape[1] == 9:

                raise RuntimeError("When merging inpainting model with a normal one, A must be the inpainting model.")

            if a.shape[1] == 8 and b.shape[1] == 4:#If we have a instructpix2pix model...

                theta_0[key][:, 0:4, :, :] = theta_func2(a[:, 0:4, :, :], b, multiplier)

            else:

                assert a.shape[1] == 9 and b.shape[1] == 4, f"Bad dimensions for merged layer {key}: A={a.shape}, B={b.shape}"

                theta_0[key][:, 0:4, :, :] = theta_func2(a[:, 0:4, :, :], b, multiplier)

                result_is_inpainting_model = True

        else:

            theta_0[key] = theta_func2(a, b, multiplier)

`
Then go to the merge tab and do a weighted difference merge using
A = instruct-pix2pix-00-22000.safetensors,
B = Whatever model you want to convert
C = v1-5-pruned-emaonly.ckpt

When you merge, the resulting model it saves should be the InstructPix2Pix version of whatever you chose for B. You can then use that in the fork I linked above.

[HWiese1980]

Now this is really cool! Thank you very much for that. I'll give it a shot.

[Klace]

#7195 has the hijack necessary to load the pix2pix model, if approved by auto I can publish an extension version of my changes

If it's not approved for whatever reason with no compromise then I will edit the extension to side load the model instead and publish

[Jonseed]

Looks like it was approved!

ee0a0da

[Jonseed]

@Unstackd can you submit a PR for your extras.py edit to allow merging instructpix2pix models? I'm sure many people will want to use their own models in the instructpix2pix method.

[Jonseed]

@Unstackd I can confirm that the merging method you outlined works to turn other models into instruct-pix2pix models!

[Gargantuanman]

Would be amazing to have as native option for conversion

[toyxyz]

I merged Pix2Pix with OrangeMix2. 1:Original / 2:Pix2Pix+OrangeMix2 / 3: Normal Img2Img of 2
https://huggingface.co/WarriorMama777/OrangeMixs

[Unstackd]

@Unstackd can you submit a PR for your extras.py edit to allow merging instructpix2pix models? I'm sure many people will want to use their own models in the instructpix2pix method.

Just submitted it! I'm still new to github PRs so here's hoping I did it right.

[Keavon]

@Klace do you foresee your extension being integrated into the main project by default? In particular, I'd be interested in using it through the API, but I don't know if that works with extensions.

[Klace]

Yes, I'm working on doing a clean integration with img2img but won't do a PR until it's perfect. In the meantime I'll continue fixing the extension too.

…

On Sat, Jan 28, 2023, 2:25 PM Keavon Chambers ***@***.***> wrote: @Klace <https://github.com/Klace> do you foresee your extension being integrated into the main project by default? In particular, I'd be interested in using it through the API, but I don't know if that works with extensions. — Reply to this email directly, view it on GitHub <#7010 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGGO6IZIVDZXOLTICIUCDALWUVXCVANCNFSM6AAAAAAUCPNO4M> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Keavon]

Fantastic to hear that, thank you!

[HWiese1980]

Really awesome! That includes sending images from and to img2img respectively inpainting, right?

[Klace]

Yep! The extension should have sends but can't receive at the moment.

…

On Sat, Jan 28, 2023, 2:52 PM Hendrik Wiese ***@***.***> wrote: Really awesome! That includes sending images from and to img2img respectively inpainting, right? — Reply to this email directly, view it on GitHub <#7010 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGGO6IYG27FS6XBWT3TDWHTWUV2HVANCNFSM6AAAAAAUCPNO4M> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[HWiese1980]

Really cool! Looking forward to get my hands on it!

[screan]

i get an error using batch input images on pix2pix.. anyone else?

RuntimeError: Given groups=1, weight of size [128, 3, 3, 3], expected input[1, 4, 1088, 640] to have 3 channels, but got 4 channels instead

[HWiese1980]

afaik batch is not supported yet

[Sanster]

Playgroundai released a new 'InstructPix2Pix' like feature, with good mask support. I have tried directly get the masked area from the output of InstructPix2Pix model result, but it's not good. Curious how they do it.

playgroundai_pix2pix.mp4

The video is from https://twitter.com/Suhail/status/1620183881273196544

Other guys thoughts, snapshots from twitter reply: