Image2image - swift

[littleowl]

Adds image2image functionality.

This is only the swift portion. A separate library that generates the model for the VAE Encoder exists here: #115

The original PR with both Swift and Python libs can be found here: #73

Sadly, I could not complete the request for the python simplification. I believe it is better not to do so. Consequently, sorry for the delay splitting up the PR. There are not many changes from the previous PR apart from removing the python and rebasing the project.

In Swift, an Encoder class is created. Various changes to the scheduler, pipeline, and CLI to support input image and strength. CGImage creation from MLShapedArray is moved into its own file along with the new function to create a MLShapedArray from a CGImage. Image loading and preparation is currently handled / optimized with vImage.

This should work with both Schedulers that we have so far. (Thanks @pcuenca) for the fix to work with DPMSolverMultistepScheduler.

Do not erase the below when submitting your pull request:
#########

• I agree to the terms outlined in CONTRIBUTING.md

[atiorh]

Thank you @littleowl ! I reviewed #115 and leaving the noise situation as is should be fine, please see my comment there :) I believe the only implication of my review on #115 on this PR is that the model I/O names should be changed from camelCase to underscore and that is it. Thank you again for your amazing contributions!

[alejandro-isaza]

Looks good, just a couple of small things.

[littleowl]

@atiorh, @alejandro-isaza - Thanks for your reviews! I believe that I resolved all the comments. Please let me know if there is anything more that I can do.

[alejandro-isaza]

The header comment needs to be fixed. Other than that just a couple more nits.

[littleowl]

@alejandro-isaza, thanks again for the review. The additional comments have been resolved.

[alejandro-isaza]

Sorry, one more thing. Let's rename this file StableDiffusionPipeline.Configuration.swift

[atiorh]

@littleowl In addition to the file name change that @alejandro-isaza asked for, if you could just add a quick note in the Example CLI Usage section to indicate that an image argument is now supported, that would be great!

[atiorh]

I just tested the CLI with different images and it seems that we need to guard against or fix a few things:
1-) When the --image file's resolution does not match the VAEEncoder input resolution, we get zsh: trace trap
2-) Loading a PNG image with the correct resolution works but trying to load the same image encoded as JPEG fails without a sufficient explanation

[atiorh]

@littleowl I just realized that we left scattered comments after the initial review, apologies! Here are the two remaining items before we can merge:

• swift/StableDiffusion/pipeline/StableDiffusionPipeline+SampleInput.swift renamed to swift/StableDiffusion/pipeline/StableDiffusionPipeline.Configuration.swift
• specific error message when input image file format and/or resolution do not match expectations

[pj4533]

I know you all have this awesome PR under control, but just wanted to say that I successfully converted the v2 base model VAEEncoder using the main (w/ the python image2image PR), and then converted the rest of v2 base using this PR, and am currently generating image2image using my own swift code. So from an end-user perspective: well done!

[saiedg]

@pj4533 well done! I'm so jealous! how can I try??

[littleowl]

Again, sorry for the delay @atiorh.
I added some error detection. It needs to happen right before prediction due to the dynamic model.
I fixed a hard coded instance of 512.
Renamed the file.
Fixed Jpeg Support by utilizing Cocoa.

[atiorh]

Thanks for your hard work @littleowl, this is going to benefit many developers! I just tested your changes and they are working like a charm. Merging now.

[saiedg]

@littleowl congratulations on merging! I'm sooo excited to use it but don't know how. Can you please add instructions in the readme or make a YouTube video?

[ynagatomo]

here is a sample;

[martinlexow]

Thanks for your exceptional work @littleowl and thank you for sharing the example @ynagatomo!

Can you please explain what the strength parameter does in this context? It seems to just decrease the steps in my observation (like: steps × strength) and it also doesn’t seem obvious to me why strength has to be less than 1.0 to trigger the imageToImage mode.

[ynagatomo]

an experiment on strength;

(code: https://github.com/ynagatomo/ImgGenSD2)