Prompt role blocks debug output + new prompt implementation example

[cpcdoy]

Description

1. Adds prompt debug output for the HTML display to help with implementing a new model prompt
2. Also adds an example notebook that shows how to use this to implement a new chat model prompt

This PR comes after I wanted to implement a new model prompt and found it could be simpler (discussed in #571)

Implementation

The implementation remains as simple as possible and uses the kwargs from each role blocks to pass a debug variable that then will either wrap each blocks with a NODISP block to hide the prompt in the HTML output or with a span block to display the prompt debug in an orange color.

Usage

Let's say we implement a new chat model with its own prompt:

from guidance import gen, system, user, assistant

# Custom prompt implementation
from guidance.models.transformers._transformers import Transformers, TransformersChat

class Orca(Transformers):
    pass

class OrcaChat(TransformersChat, Orca):
    def get_role_start(self, role_name, **kwargs):
        if role_name == "system":
            return "### System:\n"
        elif role_name == "user":
            if str(self).endswith("\n\n### User:\n"):
                return "" # we don't need to start anything if we are starting with a top level unnested system tag
            else:
                return "### System:\n"
        else:
            return " "

    def get_role_end(self, role_name=None):
        if role_name == "system":
            return "\n\n### User:\n"
        elif role_name == "user":
            return "\n\n### Response:\n"
        else:
            return " "

Then we can try the new prompt this way:

# Load the model

import torch

orca = OrcaChat('pankajmathur/orca_mini_3b', torch_dtype=torch.float16, device_map='auto')

with system(debug=True):
    lm = orca + "You are a cat expert."

with user(debug=True):
    lm += "What are the smallest cats?"

with assistant(debug=True):
    lm += gen("answer", stop=".", max_tokens=20)

It is also possible to only show some parts of the prompt (only user or both system and assistant blocks for example) to help understand how Guidance is generating the prompt. For example only showing the system block:

with system(debug=True):
    lm = orca + "You are a cat expert."

with user():
    lm += "What are the smallest cats?"

with assistant():
    lm += gen("answer", stop=".", max_tokens=20)

In practice, this works for all the other blocks too.

Compatibility

There are no compatibility issues since the default usage remains the same as before.

Full Example Usage

I also provide a notebook in the PR that shows how to easily use this to debug a new prompt Chat model implementation. The example implements Orca mini 3b from HF Hub.

[slundberg]

Thanks @cpcdoy ! This is great idea. I'll review more closely when I am not on mobile, but one quick thought: could we somehow show the white space?

Further thought: I wonder if we should have a "debug()" context that allows everything to act differently in debug mode if it wants, so we could show all the white spaces, perhaps not show the fancy role rows etc.

[cpcdoy]

Thanks @slundberg! It actually already shows white space (as can be seen in the first screenshot on the assistant line), it just doesn't show \n newlines. Since it was also the default behavior in the default output, which just goes to newline without printing the character, I kept it this way.

For your second point, I actually already thought of the possibility of putting a with debug(): context on top of everything, but didn't go with it because, at least for this prompt debugging case, I wanted the ease of activating/deactivating each parts of the prompt debug output easily (to isolate system or user or assistant or a combination of them as shown in the 2nd screenshot) without having to put a lot of with debug(): everywhere if that makes sense.

Lmk what you think and I can make adjustments if needed when I have time this week!

[slundberg]

Alright I dug into a bit today and decided that roles make more sense long term I think. This is because you may want to be able to print out the raw format of guidance functions you didn't write without having to go inside them and find all their role tags. I implemented this with a new indent_roles() context that I think can still allow for the easy switching you are talking about:

with indent_roles(False), system():
    lm = orca + "You are a cat expert."

with user():
    lm += "What are the smallest cats?"

with assistant():
    lm += gen("answer", stop=".", max_tokens=20)

It also allows for easy control of more complex functions:

with indent_roles(False):
    lm += my_complex_function()

In the process I flushed out a few ordering bugs about nested contexts. Let me know if this looks good to you. Thanks!

[slundberg]

@cpcdoy I am going to merge this for now so I can use the fixes to context block ordering, but if you want to tweak something later just let me know. thanks again!

[cpcdoy]

@slundberg Sorry for the late reply, I had quite a busy week and I'm also traveling.

I think your approach of using blocks sounds good to me! I'll try it next time I'm debugging a model and will see if I have any feedback! Thanks for the help on this PR :)

[cpcdoy]

I see you implemented this approach to set the same values as in the set_attribute method you did for blocks, but I don't see it used anywhere for now. Is this something you're planning to use later on?

[cpcdoy]

Ah, I just checked your commit history and you were using variables instead of attributes for the blocks before, so maybe a forgotten file, unless you think it's still useful for other purposes?