Clarification: Is DSP only meant for short factual answers?

[karrtikiyer]

Might be a silly questions, however all the examples I have seen has to do with short factoid answer, often between 1 and 5 words, wanted to know if DSP is only meant for short factual answers? Or can it be also used for cases where we want paraphrasing or summarising over multiple chunks of text that might contain the answers of a question asked by the user? If yes, it would be great to see an example of the same.

Thanks for all the great work!,
Regards,
Karrtik

[okhat]

You can definitely use it for what you described! Just change the training examples, the Template, and the program’s generate steps!

[karrtikiyer]

Thanks for the quick answer, I will give it a try and reach out for any help. Also in your videos on youtube, I see you mentioning that do not throw away unlabelled questions from the demo, instead feed them as unlabelled examples. Is there some documentation that explains how this works? How does the model learn from unlabelled samples?
Regards,
Karrtik

[ksgr5566]

Hey @okhat, if we are looking at answers which are long passages, how to do we demonstrate train samples? The passage_match and answer_match always returns none for them right?

[okhat]

@ksgr5566 Use a dsp.generate call to do the approximate passage_match and anwer_match!

[ksgr5566]

@ksgr5566 Use a dsp.generate call to do the approximate passage_match and anwer_match!

@okhat I don't understand on how that would work. Let me be more clear about my question:

@dsp.transformation
def multihop_attempt(d: dsp.Example) -> dsp.Example:
    # Prepare unaugmented demonstrations for the example.
    x = dsp.Example(question=d.question, demos=dsp.all_but(train, d))

    # Search. And skip examples where search fails.
    # Annotate demonstrations for multihop_search_v2 with the simpler multihop_search_v1 pipeline.
    x = multihop_search_v1(x)
    if not dsp.passage_match(x.context, d.answer): return None

    # Predict. And skip examples where predict fails.
    x = QA_predict(x, sc=False)
    if not dsp.answer_match(x.answer, d.answer): return None

    return d.copy(**x)

@dsp.transformation
def multihop_demonstrate(x: dsp.Example) -> dsp.Example:
    x.demos = dsp.annotate(multihop_attempt)(demos, k=4, return_all=True)
    return x

My answers (d.answer) here are not factoids, instead a passage which is two to three sentences long. I observed that dsp.passage_match and dsp.answer_match are made for factoid answers from here.

I want to be able to do multihop_demonstrate when demos consists of answers which are 2-3 sentences long. Could you be more clear on how to do this?

[okhat]

Write a DSP program whose job is to implement passage match and answer match using dsp.generate (using the LLM)

[ksgr5566]

ohkk.. I made another answer_match like this:

True_Answer = dsp.Type(prefix="The correct answer: ", desc="${The true answer}")
Predicted_Answer = dsp.Type(prefix="Another answer: ", desc="${The answer to compare to}")
Resp = dsp.Type(prefix="Response: ", desc="${True (or) False}")

answer_match_template = dsp.Template(
    instructions="Return True if the essence of both answers is same else return False. Respond with only 'True' and 'False'.",
    true_answer=True_Answer(), predicted_answer=Predicted_Answer(), response=Resp()
)

def answer_match(true_ans, pred_ans):
  match_example = dsp.Example(true_answer=true_ans, predicted_answer=pred_ans, demos=[])
  _, completions = dsp.generate(answer_match_template)(match_example, stage='answer_match')
  return completions[0].response

This works for answer_match.

One other doubt, why do we need passage_match in between, because finally answer_match does the work right?

[okhat]

Excellent