fix(nodes): blend latents with weight=0 with DPMSolverSDEScheduler

[psychedelicious]

Summary

• Pass the seed from latents_a to the output latents. Fixed an issue where using BlendLatentsInvocation could result in different outputs during denoising even when the alpha or slerp weight was 0.

Explanation

LatentsField has an optional seed field. During denoising, if this seed field is not present, we fall back to 0 for the seed. The seed is used during denoising in a few ways:

Initializing the scheduler.

The seed is used in two places in invokeai/app/invocations/latent.py.

The get_scheduler() utility function has special handling for DPMSolverSDEScheduler, which appears to need a seed for deterministic outputs.

DenoiseLatentsInvocation.init_scheduler() has special handling for schedulers that accept a generator - the generator needs to be seeded in a particular way. At the time of this commit, these are the Invoke-supported schedulers that need this seed:

• DDIMScheduler
• DDPMScheduler
• DPMSolverMultistepScheduler
• EulerAncestralDiscreteScheduler
• EulerDiscreteScheduler
• KDPM2AncestralDiscreteScheduler
• LCMScheduler
• TCDScheduler

Adding noise during inpainting.

If a mask is used for denoising, and we are not using an inpainting model, we add noise to the unmasked area. If, for some reason, we have a mask but no noise, the seed is used to add noise.

I wonder if we should instead assert that if a mask is provided, we also have noise.

This is done in invokeai/backend/stable_diffusion/diffusers_pipeline.py in StableDiffusionGeneratorPipeline.latents_from_embeddings().

The problem

When we create noise to be used in denoising, we are expected to set LatentsField.seed to the seed used to create the noise. This introduces some awkwardness when we manipulate any "latents" that will be used for denoising. We have to pass the seed along for every operation.

If the wrong seed or no seed is passed along, we can get unexpected outputs during denoising. One notable case relates to blending latents (slerping tensors).

If we slerp two noise tensors (LatentsFields) without passing along the seed from the source latents, when we denoise with a seed-dependent scheduler*, the schedulers use the fallback seed of 0 and we get the wrong output. This is most obvious when slerping with a weight of 0, in which case we expect the exact same output after denoising.

*It looks like only the DPMSolver- schedulers are affected, but I haven't tested all of them.

Passing the seed along in the output fixes this issue.

Related Issues / Discussions

• https://discord.com/channels/1020123559063990373/1149510134058471514/1247236122191331338
• Related: [enhancement]: Add Seed Variations to the Linear UI #6476

QA Instructions

Repro steps:

• Set scheduler to DPM++ SDE Karras
• Generate using linear UI
• Load workflow
• Slot in another noise and blend latents w/ alpha of 0:
  
• Generate and compare the first and second images
• Increase the alpha to 0.1 and generate
• Compare the second and third images, there should be small changes

On main, we expect:

• The first and second images have substantial differences.
• The first and third images have substantial differences.
• The second and third images have minor differences.

On this PR, we expect:

• The first and second images are identical.
• The first and third images have minor differences.
• The second and third images have minor differences.

Merge Plan

n/a

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• Documentation added / updated (if applicable)

[psychedelicious]

@RyanJDick @lstein I've just fixed the specific issue here, but I wonder if we should work towards cleaning this up. Maybe it's best saved for a future refactor/split of denoising into separate nodes.

[RyanJDick]

@RyanJDick @lstein I've just fixed the specific issue here, but I wonder if we should work towards cleaning this up. Maybe it's best saved for a future refactor/split of denoising into separate nodes.

I don't think I've ever noticed that the seed is attached to the LatentsField. That seems like a weird coupling. I can't think of a good reason for it. I agree it might make sense to bundle a fix for it with a future refactor. We could probably find a way to move away from that without breaking reproducibility of existing workflows, but it would be cleaner if we did it at a time when we were ok with breaking reproducibility.