Backtrack retry handler supports ChainOfThought but not Predict

[bjsi]

Here's my simple example program.

import dspy
from dspy.predict.retry import Retry
from dspy.primitives.assertions import (
    assert_transform_module,
    suggest_backtrack_handler,
)

turbo = dspy.OpenAI(model="gpt-3.5-turbo", max_tokens=1000)
dspy.settings.configure(lm=turbo, trace=[])


class GenerateQuestions(dspy.Signature):
    concept = dspy.InputField(desc="a concept to generate questions for")
    questions_and_answers = dspy.OutputField(
        desc="five questions and answers about the concept in the form 'question = answer'",
        format=lambda x: x,
    )


class Program(dspy.Module):
    def __init__(self):
        super().__init__()
        self.generate_questions = dspy.Predict(GenerateQuestions)

    def forward(self, concept):
        qas = self.generate_questions(
            concept=concept
        ).questions_and_answers.splitlines()
        dspy.Suggest(
            len(qas) == 5,
            f"Generate five questions about that concept. Found {len(qas)} questions.",
        )
        dspy.Suggest(
            all("?" not in qa for qa in qas),
            f"Questions should not end with a question mark.",
        )
        dspy.Suggest(
            all(len(qa.split(" ")) <= "3" for qa in qas),
            f"Questions should be heavily abbreviated and less than or equal to 3 words long.",
        )
        return dspy.Prediction(questions_and_answers=qas)


program = assert_transform_module(
    Program().map_named_predictors(Retry), suggest_backtrack_handler
)
program(concept="the moon")
turbo.inspect_history(n=1)

When the program backtracks when one of the assertions fails, calling self.generate_questions in Program's forward method fails with the following error message:

Exception has occurred: TypeError
'dict' object is not callable
  File "/Users/james/Projects/Python/dspy-experiments/test.py", line 26, in forward
    qas = self.generate_questions(
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/james/Projects/Python/dspy-experiments/test.py", line 47, in <module>
    program(concept="the moon")
TypeError: 'dict' object is not callable

[arnavsinghvi11]

Hi @bjsi ,

Thanks for bringing up this issue!

This error actually arises when using Assert and Suggestwith dspy.Predict. When replacing dspy.Predict with dspy.ChainOfThought, the backtracking is handled correctly.

Thanks for pointing this out! We will keep a note of accounting for Assertions usages for dspy.Predict modules.

Assertions are still in a fairly experimental phase and are currently compatible with dspy.ChainOfThought modules, which usually provides better quality responses than dspy.Predict so feel free to use dspy.ChainOfThought for your use case instead!

[bjsi]

Okay thanks for letting me know. I'll just switch to dspy.ChainOfThought instead 👍

[thomasnormal]

Are we still supposed to do

    program_with_assertions = assert_transform_module(
        program.map_named_predictors(Retry), suggest_backtrack_handler
    )

I can't find suggest_backtrack_handler anymore in dspy/primitives, but there is a similar sounding backtrack_handler function. Should I use that instead?

Looks like it was changed in 342f026

[Shangyint]

Hi. This is correct! We renamed the function to backtrack_handler.