Live preview

[sebaxakerhtc]

Hi!
As I remember, about a year ago we talked about adding live preview, but you said you will create it by yourself another way.
There's still no live preview as I see, that's why I create this PR.
In the future, when you'll do your own code for it - you can just revert this PR ;)

[JustGuyThatGuy]

Cool feature, I noticed core/stablecog.py lines 404-407 contains non-English text and sure it gets updated when the gen starts, but it would probably be worth updating to something like:

f'**Prompt**: `{queue_object.prompt}`\n**Progress**: Initialising...'
f'\n0/{queue_object.steps} iterations, 0.00 it/s'
f'\n**ETA**: Initialising...'))

[JustGuyThatGuy]

After playing about with this a bit more, I've also noticed it's causing a bunch of Something goes wrong... 404 Not Found (error code: 10008): Unknown Message... not sure if it's somehow trying to interact with messages after they've been deleted perhaps?

[sebaxakerhtc]

I noticed core/stablecog.py lines 404-407 contains non-English text

Thank you. Edited

[sebaxakerhtc]

I've also noticed it's causing a bunch of Something goes wrong... 404 Not Found (error code: 10008): Unknown Message...

Can you provide the log (if you see it in console) ?
Because I use it every day last year and didn't see it.

[JustGuyThatGuy]

Unfortunately that's all that's shown in the console:

Someone else was playing about with it and mentioned that Discord was also saying interaction failed when trying to click buttons sometimes after applying this change, not sure if that'll help narrow anything down?

[sebaxakerhtc]

Unfortunately that's all that's shown in the console

What hardware do you have?
As you can see on a screenshot above - there's no errors.
I'm always using latest version of auto1111 (every start with git pull)
RTX3090

[JustGuyThatGuy]

I've replicated the same issue on two different systems:
4060 Ti 16GB (Windows 11 via CUDA)
RX 6800 XT (Ubuntu Desktop 22.04.4 via ROCm 6.0.2)

This was done on the latest master of Automatic1111 with just your change applied both times

[sebaxakerhtc]

I've replicated the same issue on two different systems

Did you edited files manually, or did you applied PR to the main branch?

[JustGuyThatGuy]

At first I'd manually applied the changes directly to master, but I also tried just cloning your fork and checking out the Live-preview branch, both did the same

[sebaxakerhtc]

At first I'd manually applied the changes directly to master, but I also tried just cloning your fork and checking out the Live-preview branch, both did the same

Dude... I can't reproduce that. There's no errors.
Full log of installing and work of my branch "Live-preview"
log.txt

[JustGuyThatGuy]

Not sure what to tell you, checked out your latest branch again and it's still doing it on both machines for me

[sebaxakerhtc]

After playing about with this a bit more, I've also noticed it's causing a bunch of Something goes wrong... 404 Not Found (error code: 10008): Unknown Message... not sure if it's somehow trying to interact with messages after they've been deleted perhaps?

I have to reproduce it on my son's PC with RTX3060 8GB VRAM (SDXL), because on SD1.5 no problems.
Looks like it not responds few first seconds. So...
For a slower cards the solution is to increase seconds for tries between requesting to access the progress here (line 28):

        if progress_data["current_image"] is None and tries <= 10:
            time.sleep(1)

For example like this:

        if progress_data["current_image"] is None and tries <= 10:
            time.sleep(3) # or even (5)

[Kilvoctu]

Heya, is this good to merge? I see some discussions back and forth

[JustGuyThatGuy]

Just checked out sebaxakerhtc's latest change with sleep(3) being used instead and haven't been able to replicate the issue I was having, so I think it should be good to merge now

[sebaxakerhtc]

Heya, is this good to merge? I see some discussions back and forth

Hi, @Kilvoctu!
I think it depends on hardware, checkpoint, resolution, batch size, etc. Start time of Live preview may be variable.
Errors are just output that there's no progress (IMHO) because with SDXL it starts later than on SD1.5 based checkpoints.
That's why i just edited stablecog.py to increase pause from 1 to 3 seconds between tries of access to progress API.
I havent these errors on my RTX 3090 as mentioned here

BTW, I found an issue at discord github page about this 10008 API error

But, yeah - it's awesome! To view progress and stop/skip it before it finishes the generation process and you will see unwanted result!

test.mp4

[FoxxMD]

I think this may also be an issue. What I'm experiencing:

• start /draw
• live preview updates begin
• generation finishes (I can see in SD logs it is done and breakpoint hit in aiyabot in dream)
• live preview hangs for some time
• eventually, live preview deletes and bot posts finished image

I believe I am seeing this because my generation times are under ~8 seconds (using 3070 and SDXL Lightning) but the update_progress task waits at least 10 tries before finishing, regardless of whether the generation is done yet or not.

Based on my breakpoints what I see happening:

• start /draw
• live preview updates begin and queues to event_loop
• generation finishes but cannot post_dream because queue is still full of update_progress tasks that keep being created
• eventually update_progress hits 10 tries and exits, queue is cleared, and eventually gets to post_dream where it queues up finished image message

I see that the progress view is also deleted once post_dream starts so that seems like a good order to do things in but waiting for update_progress to finish takes too long.

This isn't perfect but its doing better:

async def update_progress(event_loop, status_message_task, s, queue_object, tries, had_image):
    status_message = status_message_task.result()
    has_image = False
    try:
        progress_data = s.get(url=f'{settings.global_var.url}/sdapi/v1/progress').json()

        if not had_image and progress_data["current_image"] is None and tries <= 10:
            time.sleep(1)
            event_loop.create_task(update_progress(event_loop, status_message_task, s, queue_object, tries + 1, had_image))
            return

        if progress_data["current_image"] is None and tries > 10:
            return

        if had_image and not has_image:
            return

        has_image = True
        had_image = True

        image = Image.open(io.BytesIO(base64.b64decode(progress_data["current_image"])))

        with contextlib.ExitStack() as stack:
            buffer = stack.enter_context(io.BytesIO())
            image.save(buffer, 'PNG')
            buffer.seek(0)
            file = discord.File(fp=buffer, filename=f'{queue_object.seed}.png')
        ips = '?'
        if progress_data["eta_relative"] != 0:
            ips = round((int(queue_object.steps) - progress_data["state"]["sampling_step"]) / progress_data["eta_relative"], 2)

        view = viewhandler.ProgressView()

        await status_message.edit(
            content=f'**Author**: {queue_object.ctx.author.id} ({queue_object.ctx.author.name})\n'
                    f'**Prompt**: `{queue_object.prompt}`\n**Progress**: {round(progress_data.get("progress") * 100, 2)}% '
                    f'\n{progress_data.get("state").get("sampling_step")}/{queue_object.steps} iterations, '
                    f'~{ips} it/s'
                    f'\n**ETA**: {round(progress_data.get("eta_relative"), 2)} seconds',
            files=[file], view=view)
    except Exception as e:
        print('Something goes wrong...', str(e))

    time.sleep(1)

    event_loop.create_task(update_progress(event_loop, status_message_task, s, queue_object, tries, had_image))

The main difference is that it returns from update_progress if the previous task got an image and the current task did not.

EDIT: I think it may be better to do the "has" check but check for empty string in progress_data["state"]["job"] instead. With a small 1-2 try buffer.

[FoxxMD]

Version using jobs...this seems to work better

async def update_progress(event_loop, status_message_task, s, queue_object, tries, any_job, tries_since_no_job):
    status_message = status_message_task.result()
    try:
        progress_data = s.get(url=f'{settings.global_var.url}/sdapi/v1/progress').json()
        job_name = progress_data.get('state').get('job')
        if job_name != '':
            any_job = True

        if progress_data["current_image"] is None:
            if job_name == '':
                if any_job:
                    if tries_since_no_job >= 2:
                        return
                    time.sleep(1)
                    event_loop.create_task(
                        update_progress(event_loop, status_message_task, s, queue_object, tries + 1, any_job, tries_since_no_job + 1))
                    return
                else:
                    # escape hatch
                    if tries > 10:
                        return
                    time.sleep(1)
                    event_loop.create_task(
                        update_progress(event_loop, status_message_task, s, queue_object, tries + 1, any_job, tries_since_no_job))
                    return
            else:
                time.sleep(1)
                event_loop.create_task(
                    update_progress(event_loop, status_message_task, s, queue_object, tries + 1, any_job, 0))
                return

        image = Image.open(io.BytesIO(base64.b64decode(progress_data["current_image"])))

        with contextlib.ExitStack() as stack:
            buffer = stack.enter_context(io.BytesIO())
            image.save(buffer, 'PNG')
            buffer.seek(0)
            file = discord.File(fp=buffer, filename=f'{queue_object.seed}.png')
        ips = '?'
        if progress_data["eta_relative"] != 0:
            ips = round(
                (int(queue_object.steps) - progress_data["state"]["sampling_step"]) / progress_data["eta_relative"], 2)

        view = viewhandler.ProgressView()

        await status_message.edit(
            content=f'**Author**: {queue_object.ctx.author.id} ({queue_object.ctx.author.name})\n'
                    f'**Prompt**: `{queue_object.prompt}`\n**Progress**: {round(progress_data.get("progress") * 100, 2)}% '
                    f'\n{progress_data.get("state").get("sampling_step")}/{queue_object.steps} iterations, '
                    f'~{ips} it/s'
                    f'\n**ETA**: {round(progress_data.get("eta_relative"), 2)} seconds',
            files=[file], view=view)
    except Exception as e:
        print('Something goes wrong...', str(e))

    time.sleep(1)

    event_loop.create_task(
        update_progress(event_loop, status_message_task, s, queue_object, tries + 1, any_job, 0))

[sebaxakerhtc]

I believe I am seeing this because my generation times are under ~8 seconds (using 3070 and SDXL Lightning) but the update_progress task waits at least 10 tries before finishing, regardless of whether the generation is done yet or not.

When there was a time.sleep(1) no errors but messages 404
That's why here I replaced 1 with 3
Bad choice.

[FoxxMD]

Still need to address sebaxakerhtc#11 (comment)

[sebaxakerhtc]

Still need to address sebaxakerhtc#11 (comment)

I think these errors was because of local changes
Deleted and re-cloned Live-preview branch - there's no errors anymore even with queues
Only this still happens in console:

Something goes wrong... 404 Not Found (error code: 10008): Unknown Message

[sebaxakerhtc]

Better implementation by the @FoxxMD in #253