Prompt templates

[adieyal]

I'm wondering about how we should implement prompt templates. Here is my use case:

https://publicprompts.art/cute-stuffed-animals/

The prompt is

Cute kawaii Squishy elephant plush toy, realistic texture, visible stitch line, soft smooth lighting, vibrant studio lighting, modular constructivism, physically based rendering, square image

say I want to save to a wildcard, e.g. __publicprompts/plush_toy__

Here is my yaml file

- > 
  Cute kawaii Squishy {
    elephant
    | giraffe
    | tiger
    | lion
    | darth vadar
    | hulk hogan
    | dinosaur
    | pelican
  } plush toy, realistic texture, visible stitch line, soft smooth lighting, vibrant studio lighting, modular constructivism, physically based rendering, square image 

It would be nice to be able parameterise this template by passing in the animal/character rather than baking it into the prompt.

Some ideas

1. Have a companion wildcard collection that this prompt uses, e.g.

- > 
  Cute kawaii Squishy {
    __plushtoy__
  } plush toy, realistic texture, visible stitch line, soft smooth lighting, vibrant studio lighting, modular constructivism, physically based rendering, square image 

That's gross and not a very friendly way to do it.
2. We could use jinja templates with variables, e.g.

{% with plushtoy = "elephant" %}
    {% prompt %}
        prompt_template("__publicprompts/plush_toy__", "plushtoy")
    {% endprompt %}
{% endwith %}

I know some people use jinja templates, but I've never felt the need, they feel less elegant than the simple DP language. They're also not accessible to non-coders

3. We could introduce a new argument syntax, e.g.
   __publicprompts/plush_toy(elephant)__
   I'm not sure where I stand on that. We certainly need to be careful not to create a Turing complete language, especially since jinja templates are meant to handle this sort of complexity. The option doesn't feel very intrusive though.

@akx Do you have any opinions/other suggestions?

[akx]

I like the argument syntax idea, but what about making it an assignment syntax, something like

• $(name=expr) to assign to a variable (which, dynamicprompts-wise, would evaluate expr (i.e. yes, it could contain non-literal commands) and assign the result into the SamplingContext this command appears in; we'd probably need to "harvest" these assignments out of the sequence first or something)
• $(name) (and maybe $(name:default)) to refer to a variable?

IOW, your prompt could be $(animal=foo) __publicprompts/plush_toy__, and your plush_toy wildcard would be Cute kawaii Squishy $(animal) plush toy etc. Or is that verging on "too complex" territory?

EDIT: That said, __publicprompts/plush_toy(animal=elephant)__ would definitely be an useful shorthand...

[adieyal]

There have been a number of requests for assignment so it may be a good idea regardless. Re syntax, in order to avoid stepping on too many toes, we might want to stick with braces. How about this {=animal$$expr}? It's not necessarily the most natural way of expressing an assignment, but it benefits from being consistent with the rest of our syntax.

assign the result into the SamplingContext this command appears in

Interesting, so does that mean that adding scope would be quite natural to implement if we ever decided we wanted it?

we'd probably need to "harvest" these assignments out of the sequence first or something)

it could be implemented as variable and variable assignment command types. Then as a variable assignment command is processed, its value is added to the sampling context which is that accessed by downstream variable commands.

[akx]

I'd like the syntax for assignment and access to feel the same. How about ${name=expr} and ${name}, as a compromise?

Also, there's an option for deferred evaluation here – i.e. that expr when assigned to name is only evaluated when name is used, with the sampler etc. from the context it's being used in, but that's hard to wrap one's head around and should be an option if ever implemented at all; I think by default we should just eagerly sample one (just one? could do more, so the variable ends up containing a VariantCommand...) value from expr and store that into the variable.

Interesting, so does that mean that adding scope would be quite natural to implement if we ever decided we wanted it?

Well yeah, I think scope is actually harder to not implement, thanks to how the SamplingContext cascade works. 😁

it could be implemented as variable and variable assignment command types. Then as a variable assignment command is processed, its value is added to the sampling context which is that accessed by downstream variable commands.

Sure, point just being that assignment commands (which can only exist in SequenceCommands, I believe) need to be evaluated exactly once and plucked out of the sequence before evaluating the rest of it.

[adieyal]

All sounds good to me. Once we have a basic implementation, we can play around with it to see how well it works and whether it needs tweaks.

[akx]

I can take a look at an initial implementation 👍