🤖 Build AI applications with confidence ✅ DSPy Visualizer ✅ Understand how your users are using your LLM-app ✅ Get a full picture of the quality performance of your LLM-app ✅ Collaborate with your stakeholders in ONE platform ✅ Iterate towards the most valuable & reliable LLM-app.
[ICML 2024] TrustLLM: Trustworthiness in Large Language Models
Test your prompts, models, and RAGs. Catch regressions and improve prompt quality. LLM evals for OpenAI, Azure, Anthropic, Gemini, Mistral, Llama, Bedrock, Ollama, and other local & private models with CI/CD integration.
🪢 Open source LLM engineering platform: Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23
An automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.
A task generation and model evaluation system.
Large Language Model Feedback Analysis and Optimization (LLMFAO)
Programming Language Selector based on language metadata and user-specified values.
MixEval, a ground-truth-based dynamic benchmark derived from off-the-shelf benchmark mixtures, which evaluates LLMs with a highly capable model ranking (i.e., 0.96 correlation with Chatbot Arena) while running locally and quickly (6% the time and cost of running MMLU), with its queries being stably updated every month to avoid contamination.
Interpreter for a programming language with basic features
Evaluation of Named Entity Recognition Models for Russian News Texts in the Cultural Domain
Python SDK for running evaluations on LLM generated responses
Comparing Naive- and Advanced RAG to LLM with full-context injection on a Question-answering (QA) dataset based on a narrative text. Evaluation for accuracy, latency and cost.
AI powered Spendenraid evaluation.
[ACL 2024]CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling
OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
The production toolkit for LLMs. Observability, prompt management and evaluations.
Python client for Kolena's machine learning testing platform
Official repo for the paper PHUDGE: Phi-3 as Scalable Judge. Evaluate your LLMs with or without custom rubric, reference answer, absolute, relative and much more. It contains a list of all the available tool, methods, repo, code etc to detect hallucination, LLM evaluation, grading and much more.
Add a description, image, and links to the evaluation topic page so that developers can more easily learn about it.
To associate your repository with the evaluation topic, visit your repo's landing page and select "manage topics."