Do you know of any packages or frameworks similar to promptfoo?

[Keiku]

I was looking for something just like promptfoo. Do you know of any packages or frameworks similar to this? I would like to consider other comparisons.

[ryanpeach]

I also would like to know.

[typpo]

Unfortunately not really. I built this because there wasn't anything else out there that did what I needed it to do. OpenAI does have an Evals framework you can take a look at. Its focus is on testing OpenAI models with heavier test cases, and some of the more advanced test cases require Python implementation.

[ryanpeach]

The main thing I need is this but in python with langchain compatibility. It might be worth cloning and converting.

[Keiku]

As far as I know, QAEvalChain in langchain module might be useful to me. I'm still looking to see if there are other alternatives.

[Keiku]

@typpo Thanks for the link reference.

[typpo]

For those of you working in Python, have a look at the end-to-end LLM chain testing documentation.

Specifically, I've created an example that shows how to evaluate a Python LangChain implementation.

The example compares raw GPT-4 with LangChain's LLM-Math plugin by using the exec provider to run the LangChain script:

# promptfooconfig.yaml
# ...
providers:
  - openai:chat:gpt-4-0613
  - exec:python langchain_example.py
# ...

The result is a side-by-side comparison of GPT-4 and LangChain doing math:

Hope this helps your use cases. If not, interested in learning more.

Side note - QAEvalChain is similar in approach to the llm-rubric assertion type of promptfoo. It can help evaluate whether a specific answer makes sense for a specific question.

[Keiku]

It looks like it was released recently.
hegelai/prompttools: Open-source tools for prompt testing and experimentation

[karrtikiyer]

@typpo : First of all congratulations on the great work in building this library. It would be great if we can have some way to directly compare and contrast promptfoo with prompttools and evals by OpenAi. This will make life easier for consumers to pick & choose best among these based on the usecase.