Fix MultiQueryRetriever Prompt Rewording Inconsistent Formatting Error

[davidgxue]

Description

• See MultiQueryRetriever Increasing Error Rate due to Prompt Rewording Format  #272 for details of why the error occurs. This won't be explained again in this PR.
• This PR tackles the problem where the LineListOutputParser used in LangChain's MultiQueryRetriever not cleaning enough edge cases such as empty lines generated from LLM between reworded prompt lines causing an error thrown downstream (the empty line, along with other 2 reworded prompts are individually sent downstream to be used for vector db documents retrieval but an empty string will error out during this phase).

Technical Changes

• Add CustomLineListOutputParser, a modified, less error prone implementation of the LineListOutputParser from LangChain. This CustomLineListOutputParser will get rid of lines that are empty lines and also limits the max number of lines or queries the multi-query retriever can generate through rewording.
• Manually replacing the output_parser inside the llm_chain of the multi-query retriever with the newly added CustomLineListOutputParser after its initialization.

Tests

1. Tested to see previous user prompts that causes error downstream due to empty line between reworded prompts generated no longer an issue. I have attempted almost all the errored out prompts I can find and seems like this PR has fixed the issue. See one example below.

• Langsmith link
• Before Fix Screenshot
  
• After Fix Screenshot
  

2. Ran a sanity check on the original test question set. No errors or quality degradation.

closes #272

[pankajastro]

LGTM. But we should address @Lee-W suggestions.

[cloudflare-pages]

Deploying with    Cloudflare Pages

Latest commit:	8f23a2d
Status:	✅  Deploy successful!
Preview URL:	https://51ba7499.ask-astro.pages.dev
Branch Preview URL:	https://fix-multiquery-lines-split-e.ask-astro.pages.dev

View logs

[davidgxue]

Addressed @Lee-W 's comments. Merging this to main. The ruff auto add import in pyproject file will be a separate PR to not bloat this current one.