Not sure if I found a stablediffusion (main repo) issue or not with DDIM... #1091

ilikenwf · 2023-05-20T07:07:56Z

ilikenwf
May 20, 2023

When upscaling, including using tiled vae, I'm hitting an issue here in ldm/modules/diffusionmodules/util.py - I think that for whatever reason, the ddim_timesteps is one over the max number of indicies in alphacums:

IndexError: index 1000 is out of bounds for dimension 0 with size 1000

I made the following adjustment and it tries to proceed:

 def make_ddim_sampling_parameters(alphacums, ddim_timesteps, eta, verbose=True):
     # select alphas for computing the variance schedule
-    alphas = alphacums[ddim_timesteps]
+    alphas = alphacums[ddim_timesteps-1]

This is when sampling 150 (max) steps...DDIM says it's sampling 167 in the console, though...

Either way, doing fewer than 150 steps, or with the above code adjustment, the below error results so it may be a bug in this repo after all?


╭────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────╮
│ stable-diffusion/stable-diffusion-webui/modules/call_queue.py:33 in f                                                               │
│                                                                                                                                                │
│    32 │   │   │   try:                                                                                                                         │
│ ❱  33 │   │   │   │   res = func(*args, **kwargs)                                                                                              │
│    34 │   │   │   │   progress.record_results(id_task, res)                                                                                    │
│                                                                                                                                                │
│ stable-diffusion/stable-diffusion-webui/modules/txt2img.py:57 in txt2img                                                            │
│                                                                                                                                                │
│   56 │   if processed is None:                                                                                                                 │
│ ❱ 57 │   │   processed = process_images(p)                                                                                                     │
│   58 │   p.close()                                                                                                                             │
│                                                                                                                                                │
│                                                            ... 4 frames hidden ...                                                             │
│                                                                                                                                                │
│ stable-diffusion/stable-diffusion-webui/modules/sd_samplers_compvis.py:171 in sample_img2img                                        │
│                                                                                                                                                │
│   170 │   │   steps, t_enc = sd_samplers_common.setup_img2img_steps(p, steps)                                                                  │
│ ❱ 171 │   │   steps = self.adjust_steps_if_invalid(p, steps)                                                                                   │
│   172 │   │   self.initialize(p)                                                                                                               │
│                                                                                                                                                │
│ stable-diffusion/stable-diffusion-webui/modules/sd_samplers_compvis.py:163 in adjust_steps_if_invalid                               │
│                                                                                                                                                │
│   162 │   │   │   │   num_steps = shared.opts.uni_pc_order                                                                                     │
│ ❱ 163 │   │   │   valid_step = 999 / (1000 // num_steps)                                                                                       │
│   164 │   │   │   if valid_step == math.floor(valid_step):

vladmandic · 2023-05-20T13:05:21Z

vladmandic
May 20, 2023
Maintainer

this code is 100% not mine and goes waaay back.
how ddim works is that it creates interpolated array going from 0..1000 and using steps as "guidance" on size of that array. but actual number of steps will be size of created array (each array member becomes a step).
there is option to force steps count to be as-is, but as far as i can tell, that was never used and i don't want to experiment with that much legacy code.
for example, for steps=60, it creates: [993 977 961 945 929 913 897 881 865 849 833 817 801 785 769 753 737 721 705 689 673 657 641 625 609 593 577 561 545 529 513 497 481 465 449 433 417 401 385 369 353 337 321 305 289 273 257 241 225 209 193 177 161 145 129 113 97 81 65 49 33 17 1] which results in actual timesteps=63 and thats what you see in progress bar.

but this code sounded fishy:

            if valid_step == math.floor(valid_step):
                return int(valid_step) + 1

i've changed it to:

            if valid_step == math.floor(valid_step):
                return min(int(valid_step) + 1, num_steps)

2 replies

ilikenwf May 20, 2023
Author

this code is 100% not mine and goes waaay back. how ddim works is that it creates interpolated array going from 0..1000 and using steps as "guidance" on size of that array. but actual number of steps will be size of created array (each array member becomes a step). there is option to force steps count to be as-is, but as far as i can tell, that was never used and i don't want to experiment with that much legacy code. for example, for steps=60, it creates: [993 977 961 945 929 913 897 881 865 849 833 817 801 785 769 753 737 721 705 689 673 657 641 625 609 593 577 561 545 529 513 497 481 465 449 433 417 401 385 369 353 337 321 305 289 273 257 241 225 209 193 177 161 145 129 113 97 81 65 49 33 17 1] which results in actual timesteps=63 and thats what you see in progress bar.

but this code sounded fishy:
            if valid_step == math.floor(valid_step):
                return int(valid_step) + 1
i've changed it to:
            if valid_step == math.floor(valid_step):
                return min(int(valid_step) + 1, num_steps)

Dev branch?

ilikenwf May 20, 2023
Author

Ah, I hadn't pulled.

ilikenwf · 2023-05-20T22:06:57Z

ilikenwf
May 20, 2023
Author

And to be honest, we can thank the Tiled VAE dev, as these were more or less obscured issues (this and the empty tensor PR you merged from me) - they didn't happen when not upscaling, and I like to use a high sample rate because it usually results in crazy good images.

0 replies

ilikenwf · 2023-05-21T01:51:43Z

ilikenwf
May 21, 2023
Author

@vladmandic now I get this...fun stuff!


│ stable-diffusion/stable-diffusion-webui/modules/sd_samplers_compvis.py:171 in sample_img2img                                        │
│                                                                                                                                                │
│   170 │   │   steps, t_enc = sd_samplers_common.setup_img2img_steps(p, steps)                                                                  │
│ ❱ 171 │   │   steps = self.adjust_steps_if_invalid(p, steps)                                                                                   │
│   172 │   │   self.initialize(p)                                                                                                               │
│                                                                                                                                                │
│ stable-diffusion/stable-diffusion-webui/modules/sd_samplers_compvis.py:163 in adjust_steps_if_invalid                               │
│                                                                                                                                                │
│   162 │   │   │   │   num_steps = shared.opts.uni_pc_order                                                                                     │
│ ❱ 163 │   │   │   valid_step = 999 / (1000 // num_steps)                                                                                       │
│   164 │   │   │   if valid_step == math.floor(valid_step):

2 replies

vladmandic May 21, 2023
Maintainer

ok, that's the traceback just before error - but what's the actual error line and error?

ilikenwf May 21, 2023
Author

My prompt is a preset so I'm just sing some negs "blurry, smudge, smudged"

With DDIM:

1:43:00-239362 ERROR    Exception: integer division or modulo by zero                                                                            
21:43:00-240368 ERROR    Arguments: args=('task(i412thfam05xx3n)', '', 'blurry, smudge, smudged', ['NegOnly'], 150, 17,      
                         False, False, 20, 1, 7, 1, -1.0, -1.0, 0, 0, 0, False, 1024, 512, True, 0, 3.5, '4xESRGAN', 0, 1592, 0, [], 0, False,    
                         'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 1, 'None', 2, False, 10, 1, 1, 64, False, False, False, False,    
                         False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0,  
                         False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0,  
                         False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0,  
                         False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0,  
                         True, 256, 48, False, True, True, True, False, 7, 100, 'Constant', 0, 'Constant', 0, 4, False, 'x264', 'blend', 10, 0, 0,
                         False, True, True, True, 'intermediate', 'animation', <controlnet.py.UiControlNetUnit object at 0x7f551091ab50>, False,  
                         '', '', '', '', '', '', '', '', '', '', '', '', '', False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0,
                         '', [], 0, '', [], True, False, False, False, 0, False, None, False, 50, False, 4.0, '', 10.0, 'Linear', 3, False, 30.0, 
                         True, False, False, 0, 0.0, 'Lanczos', 1, True, 0, 0, 0.001, 75, 0.0, False, True, 10.0, 30.0, True, 0.0, 'Lanczos', 1,  
                         0, 0, 75, 0.0001, 0.0, False, True, False) kwargs={}                                                                     
21:43:00-245054 ERROR    gradio call: ZeroDivisionError                                                                                           
╭────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────╮
│ stable-diffusion/stable-diffusion-webui/modules/call_queue.py:33 in f                                                               │
│                                                                                                                                                │
│    32 │   │   │   try:                                                                                                                         │
│ ❱  33 │   │   │   │   res = func(*args, **kwargs)                                                                                              │
│    34 │   │   │   │   progress.record_results(id_task, res)                                                                                    │
│                                                                                                                                                │
│ stable-diffusion/stable-diffusion-webui/modules/txt2img.py:57 in txt2img                                                            │
│                                                                                                                                                │
│   56 │   if processed is None:                                                                                                                 │
│ ❱ 57 │   │   processed = process_images(p)                                                                                                     │
│   58 │   p.close()                                                                                                                             │
│                                                                                                                                                │
│                                                            ... 5 frames hidden ...                                                             │
│                                                                                                                                                │
│ stable-diffusion/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/models/diffusion/ddim.py:25 in make_schedule │
│                                                                                                                                                │
│    24 │   def make_schedule(self, ddim_num_steps, ddim_discretize="uniform", ddim_eta=0., verb                                                 │
│ ❱  25 │   │   self.ddim_timesteps = make_ddim_timesteps(ddim_discr_method=ddim_discretize, num                                                 │
│    26 │   │   │   │   │   │   │   │   │   │   │   │     num_ddpm_timesteps=self.ddpm_num_times                                                 │
│                                                                                                                                                │
│ stable-diffusion/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/modules/diffusionmodules/util.py:55 in       │
│ make_ddim_timesteps                                                                                                                            │
│                                                                                                                                                │
│    54 │   if ddim_discr_method == 'uniform':                                                                                                   │
│ ❱  55 │   │   c = num_ddpm_timesteps // num_ddim_timesteps                                                                                     │
│    56 │   │   ddim_timesteps = np.asarray(list(range(0, num_ddpm_timesteps, c)))                                                               │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ZeroDivisionError: integer division or modulo by zero

Well, with DPM Adaptive and Tiled Diffusion, I get this:


21:38:22-377519 ERROR    Exception: index -2 is out of bounds for dimension 0 with size 1                                                         
21:38:22-378436 ERROR    Arguments: args=('task(252qeluppzk5hr3)', '', 'blurry, smudge, smudged', ['NegOnly'], 150, 10,      
                         False, False, 20, 1, 7, 1, -1.0, -1.0, 0, 0, 0, False, 1024, 512, True, 0, 3.5, '4xESRGAN', 0, 1592, 0, [], 0, False,    
                         'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 1, 'None', 2, False, 10, 1, 1, 64, False, False, False, False,    
                         False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0,  
                         False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0,  
                         False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0,  
                         False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0,  
                         True, 256, 48, False, True, True, True, False, 7, 100, 'Constant', 0, 'Constant', 0, 4, False, 'x264', 'blend', 10, 0, 0,
                         False, True, True, True, 'intermediate', 'animation', <controlnet.py.UiControlNetUnit object at 0x7f5516875c10>, False,  
                         '', '', '', '', '', '', '', '', '', '', '', '', '', False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0,
                         '', [], 0, '', [], True, False, False, False, 0, False, None, False, 50, False, 4.0, '', 10.0, 'Linear', 3, False, 30.0, 
                         True, False, False, 0, 0.0, 'Lanczos', 1, True, 0, 0, 0.001, 75, 0.0, False, True, 10.0, 30.0, True, 0.0, 'Lanczos', 1,  
                         0, 0, 75, 0.0001, 0.0, False, True, False) kwargs={}                                                                     
21:38:22-382737 ERROR    gradio call: IndexError                                                                                                  
╭────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────╮
│ stable-diffusion/stable-diffusion-webui/modules/call_queue.py:33 in f                                                               │
│                                                                                                                                                │
│    32 │   │   │   try:                                                                                                                         │
│ ❱  33 │   │   │   │   res = func(*args, **kwargs)                                                                                              │
│    34 │   │   │   │   progress.record_results(id_task, res)                                                                                    │
│                                                                                                                                                │
│ stable-diffusion/stable-diffusion-webui/modules/txt2img.py:57 in txt2img                                                            │
│                                                                                                                                                │
│   56 │   if processed is None:                                                                                                                 │
│ ❱ 57 │   │   processed = process_images(p)                                                                                                     │
│   58 │   p.close()                                                                                                                             │
│                                                                                                                                                │
│                                                            ... 3 frames hidden ...                                                             │
│                                                                                                                                                │
│ stable-diffusion/stable-diffusion-webui/modules/processing.py:879 in sample                                                         │
│                                                                                                                                                │
│    878 │   │   │   log.debug('Applied token merging for high-res pass')                                                                        │
│ ❱  879 │   │   samples = self.sampler.sample_img2img(self, samples, noise, conditioning, uncond                                                │
│    880 │   │   self.is_hr_pass = False                                                                                                         │
│                                                                                                                                                │
│ stable-diffusion/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py:336 in sample_img2img                                     │
│                                                                                                                                                │
│   335 │   │   │   ## last sigma is zero which isn't allowed by DPM Fast & Adaptive so taking v                                                 │
│ ❱ 336 │   │   │   extra_params_kwargs['sigma_min'] = sigma_sched[-2]                                                                           │
│   337 │   │   if 'sigma_max' in parameters:

ilikenwf · 2023-05-21T01:55:39Z

ilikenwf
May 21, 2023
Author

Would something like this fix it?

    def adjust_steps_if_invalid(self, p, num_steps):
        if num_steps > 0 and (((self.config.name == 'DDIM') and p.ddim_discretize == 'uniform') or (self.config.name == 'PLMS') or (self.config.name == 'UniPC')):
            if self.config.name == 'UniPC' and num_steps < shared.opts.uni_pc_order:
                num_steps = shared.opts.uni_pc_order

            valid_step = 999 / (1000 // num_steps)

            if valid_step == math.floor(valid_step):
                num_steps = min(int(valid_step) + 1, num_steps)

        return num_steps

1 reply

ilikenwf May 21, 2023
Author

This doesn't actually fix the issue, just forwards it along to diffusers...

vladmandic · 2023-05-21T12:52:54Z

vladmandic
May 21, 2023
Maintainer

digging a bit deeper in this code - it all comes down from fact that steps are only used as guidance to determine actual timesteps which is a linear interpolation of steps in a scheduler range and can differ from actual steps (for example, you've already noticed that 150 steps equals 167 timesteps for ddim)

and higher number of steps, bigger the likelihood of timesteps becoming silly values - for example, with steps above 100 there is a non-trivial chance that some timestep will be interpolated as zero, so it can lead to division by zero (and that's just an example).

so what can we do?

trace down every possible interpolation error and fix it as it occurs
omg, i really don't want to do that.
enable alternative mode where interpolation does not occur and steps are used as-is instead of timesteps
i could do that, but what is the use case? and it could never be default as its not a desirable behavior for majority of users
simply reduce maximum allowed steps to be 99
is there a valid use case of running more steps than that ever?
note that if a scheduler does not converge by then and there are still a lot of artifacts,
its probably better to use a different scheduler than trying to force bad one.

i'm leaning towards option (3). thoughts/opinions?

1 reply

ilikenwf May 22, 2023
Author

3 Sounds good to me. I randomly use different samplers, is the only reason I even noticed this.

vladmandic · 2023-05-22T03:59:12Z

vladmandic
May 22, 2023
Maintainer

steps = gr.Slider(minimum=1, maximum=99, step=1, elem_id=f"{tabname}_steps", label="Sampling steps", value=20)

done. although its applied only on fresh configs, so if you already have ui-config.json, you'll still see 150 as limit.

0 replies

TheOnlyHolyMoly · 2023-05-29T08:04:50Z

TheOnlyHolyMoly
May 29, 2023

is this the origin of the issue? huggingface/diffusers#2585

1 reply

vladmandic May 29, 2023
Maintainer

i believe so - nice job on digging it out :)

TheOnlyHolyMoly · 2023-05-29T17:08:25Z

TheOnlyHolyMoly
May 29, 2023

Thanks, the funny thing is that I had already read the abstract of samplers being flawed when the paper was released and was aware of that "medium brightness" issue, I just had lost track on that topic until I saw their paper again. Have you seen their comparative report before/after their proposed fixes?

1 reply

vladmandic May 29, 2023
Maintainer

i have and its impressive difference (if that's the reality).
the problem with messing with samplers is:

it would need to be done as a switch to preserve backward compatibility
it would need to be applicable to all samplers that follow timesteps logic, not just ddim (majority don't use ddim anymore anyhow)
how to get the code in? both compvis and k-diffusion sampler families are defined in upstream repos.

this can probably be done with monkey-patching (redefining set_timesteps method of the parent class).
and entire implementation can be done in sd_samplers_compvis.py (for compvis samplers).

i'd be very curious if someone attempted it :)

vladmandic · 2023-05-29T18:15:31Z

vladmandic
May 29, 2023
Maintainer

@TheOnlyHolyMoly do you mind creating a feature request for this so it doesn't fall off the radar?

2 replies

TheOnlyHolyMoly May 29, 2023

sure, I'll make on tonight, I can also link the associated cross repo issues in the request, so we'll have all info in one spot.

TheOnlyHolyMoly Jun 2, 2023

@vladmandic Done. Discussion can be closed here if you wish.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Not sure if I found a stablediffusion (main repo) issue or not with DDIM... #1091

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 9 comments 10 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Not sure if I found a stablediffusion (main repo) issue or not with DDIM... #1091

ilikenwf May 20, 2023

Replies: 9 comments · 10 replies

vladmandic May 20, 2023 Maintainer

ilikenwf May 20, 2023 Author

ilikenwf May 20, 2023 Author

ilikenwf May 20, 2023 Author

ilikenwf May 21, 2023 Author

vladmandic May 21, 2023 Maintainer

ilikenwf May 21, 2023 Author

ilikenwf May 21, 2023 Author

ilikenwf May 21, 2023 Author

vladmandic May 21, 2023 Maintainer

ilikenwf May 22, 2023 Author

vladmandic May 22, 2023 Maintainer

TheOnlyHolyMoly May 29, 2023

vladmandic May 29, 2023 Maintainer

TheOnlyHolyMoly May 29, 2023

vladmandic May 29, 2023 Maintainer

vladmandic May 29, 2023 Maintainer

TheOnlyHolyMoly May 29, 2023

TheOnlyHolyMoly Jun 2, 2023

ilikenwf
May 20, 2023

Replies: 9 comments 10 replies

vladmandic
May 20, 2023
Maintainer

ilikenwf May 20, 2023
Author

ilikenwf May 20, 2023
Author

ilikenwf
May 20, 2023
Author

ilikenwf
May 21, 2023
Author

vladmandic May 21, 2023
Maintainer

ilikenwf May 21, 2023
Author

ilikenwf
May 21, 2023
Author

ilikenwf May 21, 2023
Author

vladmandic
May 21, 2023
Maintainer

ilikenwf May 22, 2023
Author

vladmandic
May 22, 2023
Maintainer

TheOnlyHolyMoly
May 29, 2023

vladmandic May 29, 2023
Maintainer

TheOnlyHolyMoly
May 29, 2023

vladmandic May 29, 2023
Maintainer

vladmandic
May 29, 2023
Maintainer