[Feature] Negative Prompts using Examples trainings set

[Solido]

What feature would you like to see?

Using training sets to guide the LLM we generate a prompt in the form of

User message:

[[ ## query ## ]]
Nitroglycerin


Assistant message:

[[ ## range_value ## ]]
glyceryl trinitrate

I need to send some negative prompts but I would like to conserve the same data structure approach with a collection of dspy.Example.

How can I express that?

should I add another field in Example but then they are mixed with good process?

should I hack something in module description but then force an explicit format like list bullets?

Should there be a Negative property in compile like negativeTrainset or a NegativeExample?

Thank you!

[okhat]

Hey @Solido !! Welcome.

What are negative prompts? Are they Examples of bad behavior that you want the LLM to avoid?

Maybe the better question I should ask is: what are you trying to achieve? What's the problem being fixed?

[Solido]

Hey @okhat

Negative prompt is a common term first found in image generation where you want to avoid specific item on the generation but it's also used in basic code generation where you clarify too a kind of grammar you want to avoid to respect current repo style.

[torchss]

@Solido is there a MRE you can propose where we can see the need for a negative prompt?

I linked to a topic I'm tracking myself and there's some related research but I am yet to come across real life examples of negative prompts

[Solido]

Hi @torchss!

Here are some cases we could start to discuss.
This are recognized best practices that I would like to port to DSPy but beyond building the full signature.

It can be Negative prompts only or a mix apply/avoid to help LLM heuristics

Addressing a coding agent with apply/avoid with explicit rule (XML is used to help parse complex code format and support any language grammar) :

Avoid using foreach and collection iteration always use `for` iteration
<good-code>
   for(final event in events) eventMethod(e))
</good-code>

<bad-code>
   events.forEach((e) => eventMethod(e))
</bad-code>

Only Negative Example

<bad-code>
   a ? c : d
</bad-code>
<forbidden-code>
   eval('dangerous_hidden_value')
</forbidden-code>

Generate a image

supermarket drinks alley image --no alcohol --no coca""

Google imagen

supermarket drinks alley -alcohol -coca""

How it could be expressed from a easy code entry:

Explicit (meh)

avoidTrainset = [
    dspy.Example(query='no alcohol', append="-alcohol"),
    dspy.Example(query='remove sugar drinks', append="-no sugar drinks"),
]

Negative handle the format

avoidTrainset = [
    dspy.Negative(query='alcohol'),
    dspy.Negative(query='sugar drinks'),
]

Problems appears how Example are added to the final prompt as explained in the ticket description.

Happy to ear how you would approach this!

[okhat]

Could this be part of the instructions in the Signature?

[Solido]

@okhat that my first concern. If I have to use a string template to do that then it defeat the whole DSPy programming concept.

I'm quite surprised that there not much traction on a somewhat common practice. You can find it in every coding LLM (XML portions)

I think previously there was some instructions for that kind of hint but it was removed with the last refactor with no alternatives.

Right now I'm thinking it this has the same for as Example so it should be an iterable of programmable objects passed to the signature as many will benefits from it but...
If many signatures have the same function, they can have a very different goal but you don't want to repeat the negative prompts, could be too much of them, so you want to share them at the module or app level (coding styles is the best example)