The LLM Evaluation Framework
evaluation-metrics evaluation-framework llm-evaluation llm-evaluation-framework llm-evaluation-metrics
-
Updated
Jun 18, 2025 - Python
#
llm-evaluation-framework
Here are 29 public repositories matching this topic...
Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.
testing ci evaluation ci-cd pentesting cicd vulnerability-scanners prompts evaluation-framework red-teaming rag llm prompt-engineering llmops prompt-testing llm-eval llm-evaluation llm-evaluation-framework
-
Updated
Jun 18, 2025 - TypeScript
Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪
agent-framework ai-red-team prompt-testing llm-security llm-vulnerabilities llm-evaluation llm-fuzzing llm-evaluation-framework llm-guardrails llm-scanner llm-jailbreaks llm-fuzzer llm-fuzzer-aggregator agent-security
-
Updated
Jun 10, 2025 - Python
The official evaluation suite and dynamic data release for MixEval.
benchmark evaluation benchmarking-suite evaluation-framework benchmarking-framework foundation-models large-language-models large-language-model llm-inference llm-evaluation large-multimodal-models llm-evaluation-framework benchmark-mixture mixeval
-
Updated
Nov 10, 2024 - Python
LangFair is a Python library for conducting use-case level LLM bias and fairness assessments
python ai artificial-intelligence bias fairness ai-safety fairness-testing bias-detection fairness-ai fairness-ml responsible-ai ethical-ai large-language-models llm llm-evaluation llm-evaluation-framework llm-evaluation-metrics
-
Updated
Jun 11, 2025 - Python
Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
metrics good-first-issue llm prompt-engineering generative-ai llmops llm-eval llm-tools llm-evaluation llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework
-
Updated
Feb 13, 2025 - Python
MIT-licensed Framework for LLMs, RAGs, Chatbots testing. Configurable via YAML and integrable into CI pipelines for automated testing.
open-source ci testing-tools chatbot-framework testing-framework chatbot-testing rag ai-chat large-language-models llm ai-testing llm-evaluation llm-evaluation-framework prompt-test llm-testing ai-testing-tool generative-ai-testing rag-testing summarization-testing
-
Updated
Dec 11, 2024 - Python
An easy python package to run quick basic QA evaluations. This package includes standardized QA evaluation metrics and semantic evaluation metrics: Black-box and Open-Source large language model prompting and evaluation, exact match, F1 Score, PEDANT semantic match, transformer match. Our package also supports prompting OPENAI and Anthropic API.
qa-automation-test rl-training llm exact-matching llm-evaluation llm-evaluation-toolkit llm-evaluation-framework reward-modeling
-
Updated
Jun 15, 2025 - Python
Develop reliable AI apps
-
Updated
Jun 18, 2025 - Svelte
Benchmarking Large Language Models for FHIR
-
Updated
May 31, 2025 - TypeScript
[ACL'24] A Knowledge-grounded Interactive Evaluation Framework for Large Language Models
machine-learning explainable-ai llm llm-evaluation llm-evaluation-toolkit llm-evaluation-framework llm-evaluation-metrics acl2024
-
Updated
Jul 19, 2024 - Python
FM-Leaderboard-er allows you to create leaderboard to find the best LLM/prompt for your own business use case based on your data, task, prompts
-
Updated
Oct 31, 2024 - Python
Realign is a testing and simulation framework for AI applications.
ai simulation evaluation alignment red-teaming rag prompt-engineering llms llmops llm-eval llm-evaluation aiengineering llm-evaluation-framework
-
Updated
Dec 4, 2024 - Python
Code for "Prediction-Powered Ranking of Large Language Models", NeurIPS 2024.
ranking-algorithm llm-eval llm-evaluation llm-evaluation-framework prediction-powered-inference rank-sets
-
Updated
Oct 28, 2024 - Jupyter Notebook
Create an evaluation framework for your LLM based app. Incorporate it into your test suite. Lay the monitoring foundation.
workshop llm llms llmops llm-eval llm-test llm-evaluation-framework llm-evaluation-metrics llm-monitoring llm-testing llm-evals
-
Updated
May 6, 2025 - Jupyter Notebook
TypeScript SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
llm prompt-engineering llms llm-eval llm-tools llm-evaluation llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework
-
Updated
Jan 17, 2025 - TypeScript
Multilingual Evaluation Toolkits
-
Updated
Nov 7, 2024 - Python
Community Plugin for Genkit to use Promptfoo
plugin testing firebase ai evaluation prompt prompts evaluation-framework llm llmops prompt-testing llm-eval llm-evaluation llm-evaluation-framework promptfoo genkit genkitx genkit-plugin
-
Updated
Jan 3, 2025 - TypeScript
Shin Rakuda is a comprehensive framework for evaluating and benchmarking Japanese large language models, offering researchers and developers a flexible toolkit for assessing LLM performance across diverse datasets.
-
Updated
Sep 17, 2024 - Python
A set of auxiliary systems designed to provide a measure of estimated confidence for the outputs generated by Large Language Models.
dataset datasets llm llms llm-training llm-evaluation llms-reasoning llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework llm-evaluation-metrics llms-efficency llms-evalution
-
Updated
May 25, 2025 - Python
Improve this page
Add a description, image, and links to the llm-evaluation-framework topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the llm-evaluation-framework topic, visit your repo's landing page and select "manage topics."