Custom Prompts

[Hakimovich99]

Hey!

Sorry for not following the issues template. I saw a few issues about the ability to have custom prompts (#245 #334 ). I wanted to know if we can expect it to be released in ragas or if there was any reason for not providing this ability.

I would like to compare different evaluations using different LLMs in different languages. It would be nice to be able to modify the evaluation prompt when computing metrics like context_relevancy and context_precision for example.

A feature similar to that would be great (lines 20-28 and 109)!:
https://github.com/run-llama/llama_index/blob/218392bc3006c344dc2a3407feaf10a61b8193b8/llama_index/evaluation/dataset_generation.py#L109

Thanks and looking forward to reading you!

[shahules786]

Hey @Hakimovich99 , Sure. Can you share why would you want to modify prompts in metrics?

[Hakimovich99]

Hey @shahules786, thanks for the reply and sorry for answering this late!

I will take the example of your context_relevancy metric. The prompt you're using is this one:

"""
Please extract relevant sentences from the provided context that is absolutely required answer the following question. If no relevant sentences are found, or if you believe the question cannot be answered from the given context, return the phrase "Insufficient Information". While extracting candidate sentences you're not allowed to make any changes to sentences from given context.

question:{question}
context:\n{context}
candidate sentences:\n"""

I have a Dutch dataset and would like to compute this metric, but I don't think the metric will be correctly computed with this prompt in English, which is why I would like to be able to modify the prompt and put it in Dutch.
Furthermore, this implies another need which is the ability to see the LLM's answer to this prompt (based on which the context_relevancy is computed), ie, a way to make sure that the LLM correctly understood the task (in this example that the answer is composed of sentences from, and only from, the given context).

To reformulate, it would be nice to be able to see the LLM answer to the query used to compute a metric, and modify the prompt accordingly if the output is not the one desired.

I don't have a code example for this but I can provide one if needed.

[shahules786]

@Hakimovich99 , thanks for your reply. I got your point. For modifying prompts, we are considering

1. A general abstraction to allow users to specify a prompt
2. An automatic method by which prompts are adjusted to the user's data domain.

How does that sound to you?

[Hakimovich99]

Hey @shahules786.

Sounds great, and pretty curious about how the second option will look like!

In my opinion, it would be also great to see the LLM answer in a case where the users specify the prompt. Just to make sure the specified prompt did not break/deteriorate the metric computation.

Thanks for your openness!

[shahules786]

Thank you @Hakimovich99 , I would love to have a chat and understand your requirements more and offer quick help if possible. If you're free sometime next week let me know calendly

[Hakimovich99]

Hey, I had a meeting planned with @shahules786 last Tuesday, but he unfortunately could not make it.
Is there anyone else with whom we can plan a meeting? We would be very interested in having a conversation with one of you on that topic!
Thanks!

[jjmachan]

hey @Hakimovich99 sorry about missing out on that meeting, he got sick and had to be hospitalised. but now he is all good so we will be rescheduling this shortly.

really looking forward to chatting with you about this :)

[weissenbacherpwc]

@Hakimovich99 is there an update yet/ did you find a workaround?

[AgenP]

+1 on this custom prompt feature.

I’d love to be able to test things out to really minimise hallucinations & maximise reliability, regardless of model choice. Things like:

• 5-shot vs 10-shot vs 20 shot prompts —> How much better would the chosen LLM be at scoring w/o hallucinations?
• Domain specific prompt examples, for improving the binary classifications (e.g. w/ context recall and context precision prompts)
• Fine-tuned LLM performance with different prompts or no prompt —> Being able to test if the prompt will boost or conflict w/ eval scoring capability
• & more

Great work with the framework! It has been super useful. I felt I should give some feedback since Ragas has helped me build and also as you can be/are the foundation for production-grade apps and I would love to improve the impact we devs have on the world 💪

Thanks

[shahules786]

Thanks a lot @AgenP . We are merging some non-trivial Prompt related features in ragas including custom prompts and automatic prompt adaptation for different languages #407

[shahules786]

@AgenP @Hakimovich99 @weissenbacherpwc Just merged the docs for adding custom prompts and language adaptation with ragas. Check it out and let us know your review
https://docs.ragas.io/en/latest/howtos/applications/index.html

[jjmachan]

closing now - feel free to reopen if there is anything more 🙂