[patch] Reduce NaN Occurrences by Simple Prompt Modification for JSON Output for context_precision

[i-w-a]

Overview

During the calculation of context_precision, an issue was observed where increasing the context amount led to a surge in NaN occurrences. Comparatively, context_recall does not exhibit this problem. An investigation into the causes of the difference uncovered that the issue stems from whether the prompts specify outputting in JSON format.

Discovery

It was found that simply specifying JSON output for context_precision, similar to what is done for context_recall, significantly reduces the incidence of NaN. Utilizing JSON mode appears to be crucial, as noted in the OpenAI reference for text generation in JSON mode:

"If you don't include an explicit instruction to generate JSON, the model may generate an unending stream of whitespace and the request may run continually until it reaches the token limit." OpenAI Text Generation JSON Mode Documentation

Solution

To align with best practices and address the NaN generation issue, I propose updating the prompt for context_precision to explicitly instruct the generation of output in JSON format. This small but impactful change will bring context_precision in line with how context_recall operates and ensure more stable and predictable outcomes when handling larger context volumes.

Impact

By making this explicit switch to JSON output, we not only follow the guideline provided by OpenAI but also prevent the potential uncontrolled behavior that can result in a heavy onslaught of NaN values. This improvement should increase the reliability of calculations within our system and significantly decrease the time spent debugging NaN-related issues.

I look forward to your review and approval of this change, which will help us maintain robustness in our context precision calculations.

Best,
i-w-a

[shahules786]

Hey @i-w-a , this is nice hack man. I tried it , it reduces the issue even with bedrock by 20% avg. I have raised a PR to your fork i-w-a#1, can you merge it?
I added it to every prompt that uses JSON

[i-w-a]

Thank you. I was surprised that bedrock even worked!

[shahules786]

@i-w-a haha :)