Define a strategy for prompt testing

[mspronesti]

Hi,
we've been changing / adjusting the code generation prompt a couple of times in the past days. I highly suggest defining a strategy for prompt evaluation, to measure the improvement and make sure it is not patching one failing scenario while breaking several others. Do we have one at the moment ?

[gventuri]

Totally. This is actually a very good point, also considering that some of the recent changes have introduced bugs.
Here's the strategy as of now: we try not to edit the prompt unless it's strictly necessary.

However, it might happen we will need to change it at some point. In that case, I recommend to write down a list of "questions/prompts" we can ask PandasAI to benchmark whether there are regressions.

I suggest we write them down in this conversation and once we have, say 50 different use cases, we add to the documentation.

In the long run, it would also be cool to run them in the CI when something is changed within a prompt (a little bit expensive, but still shouldn't happen too often).

What do you think?

[mspronesti]

I believe we need a meaningful dataset of "must work" queries. Adding them to the CI makes sense, as long we test the actual service without mocking it :-)

Another point I'd like to raise is that I'm not sure having a unique prompt for all the supported model is a good idea and I recommend further investigating this aspect.

[gventuri]

@mspronesti totally! To to that, tho, we need to refactor a little bit the architecture of prompts in general.
I was considering having a Prompt class which contains the prompt + other info. And we could assign a prompt for each use case to each LLM (or worse case fallback to the "default" prompt for each use case.
Then I think it will make sense, at some point, to also have a different prompt when we want to generate a chart to make it more effective. This might require some kind of "orchestration", but still make sense.