Is it possible to change the prompt on the fly?

[nickkolok]

E.g. if 20 inference steps are to be performed, is it possible to run the first 10 steps with one prompt and the second 10 steps with another? lpw_stable_diffusion seems to change the effective prompt every inference step, doesn't it?

This question is motivated by the observation that some words in the prompt affect not only the points they are suggested to. For example, adding bad anatomy and extra limb to the negative prompt sometimes vanished the effect of flat_colors tag in positive prompt.

For example, I daresay that trying to draw perfect eyes is totally useless for the first 5-10 steps or inference; and vice versa, on the 18th step of 20 it is too late to correct bad anatomy, amputee, floating limbs and even to try to make cowboy_shot. I may be wrong (and I'll be glad to be corrected if neural networks do not work this way ), but I have to admit that I haven't managed to check this on my own.

Maybe there is a native way to run the pipeline twice consequently on the same image , something like

result = pipe(prompt = prompt, guidance_scale=7.5).pipe(prompt = another_prompt, guidance_scale=7.5)

?

[nickkolok]

Just compare.

First image:

asuka langley in bikini, cowboy_shot, 1990s_\(style\), masterpiece, frontal shot
Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 3495230995, Size: 512x512, Model hash: 6569e224

Second image:

asuka langley in bikini, cowboy_shot, 1990s_\(style\), masterpiece, frontal shot
Negative prompt: (bad anatomy), (extra limb)
Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 3495230995, Size: 512x512, Model hash: 6569e224

The styles differ, don't they?

[nickkolok]

A very draft version, even not tested (my GPU is too old):
https://github.com/nickkolok/diffusers/blob/lpw_halfs_draft1/examples/community/lpw_stable_diffusion_halfs.py
I'm going to test it in the nearest future; if not, may be a good starting point for someone

[patrickvonplaten]

We currently don't have that functionality in diffusers. You could however try to use the cross attention processors to create such a pipeline: #1639

[nickkolok]

I've created a proof-of-concept demo colab for my implementation:
https://colab.research.google.com/drive/1dqS3J3mKG5kJu7tkU5jiqCZtdf-PMMEe?usp=sharing
However, Anonymous on 2ch told me that AUTO1111 has the same feature implemented in a much more flexible way.

[williamberman]

Hey @nickkolok a pipeline with multiple prompts sounds cool and would be a good fit for a community pipeline if you would like to open a PR?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[nickkolok]

I've created a PR, but it looks like I probably need help with it!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.