The LLM Evaluation Framework
evaluation-metrics evaluation-framework llm-evaluation llm-evaluation-framework llm-evaluation-metrics
-
Updated
May 24, 2025 - Python
#
llm-evaluation-framework
Here are 27 public repositories matching this topic...
Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.
testing ci evaluation ci-cd pentesting cicd vulnerability-scanners prompts evaluation-framework red-teaming rag llm prompt-engineering llmops prompt-testing llm-eval llm-evaluation llm-evaluation-framework
-
Updated
May 25, 2025 - TypeScript
Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪
agent-framework ai-red-team prompt-testing llm-security llm-vulnerabilities llm-evaluation llm-fuzzing llm-evaluation-framework llm-guardrails llm-scanner llm-jailbreaks llm-fuzzer llm-fuzzer-aggregator agent-security
-
Updated
May 23, 2025 - Python
The official evaluation suite and dynamic data release for MixEval.
benchmark evaluation benchmarking-suite evaluation-framework benchmarking-framework foundation-models large-language-models large-language-model llm-inference llm-evaluation large-multimodal-models llm-evaluation-framework benchmark-mixture mixeval
-
Updated
Nov 10, 2024 - Python
LangFair is a Python library for conducting use-case level LLM bias and fairness assessments
python ai artificial-intelligence bias fairness ai-safety fairness-testing bias-detection fairness-ai fairness-ml responsible-ai ethical-ai large-language-models llm llm-evaluation llm-evaluation-framework llm-evaluation-metrics
-
Updated
Apr 25, 2025 - Python
Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
metrics good-first-issue llm prompt-engineering generative-ai llmops llm-eval llm-tools llm-evaluation llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework
-
Updated
Feb 13, 2025 - Python
MIT-licensed Framework for LLMs, RAGs, Chatbots testing. Configurable via YAML and integrable into CI pipelines for automated testing.
open-source ci testing-tools chatbot-framework testing-framework chatbot-testing rag ai-chat large-language-models llm ai-testing llm-evaluation llm-evaluation-framework prompt-test llm-testing ai-testing-tool generative-ai-testing rag-testing summarization-testing
-
Updated
Dec 11, 2024 - Python
Develop reliable AI apps
-
Updated
Apr 17, 2025 - Svelte
[ACL'24] A Knowledge-grounded Interactive Evaluation Framework for Large Language Models
machine-learning explainable-ai llm llm-evaluation llm-evaluation-toolkit llm-evaluation-framework llm-evaluation-metrics acl2024
-
Updated
Jul 19, 2024 - Python
Benchmarking Large Language Models for FHIR
-
Updated
Nov 29, 2024
FM-Leaderboard-er allows you to create leaderboard to find the best LLM/prompt for your own business use case based on your data, task, prompts
-
Updated
Oct 31, 2024 - Python
Realign is a testing and simulation framework for AI applications.
ai simulation evaluation alignment red-teaming rag prompt-engineering llms llmops llm-eval llm-evaluation aiengineering llm-evaluation-framework
-
Updated
Dec 4, 2024 - Python
Code for "Prediction-Powered Ranking of Large Language Models", NeurIPS 2024.
ranking-algorithm llm-eval llm-evaluation llm-evaluation-framework prediction-powered-inference rank-sets
-
Updated
Oct 28, 2024 - Jupyter Notebook
Create an evaluation framework for your LLM based app. Incorporate it into your test suite. Lay the monitoring foundation.
workshop llm llms llmops llm-eval llm-test llm-evaluation-framework llm-evaluation-metrics llm-monitoring llm-testing llm-evals
-
Updated
May 6, 2025 - Jupyter Notebook
TypeScript SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
llm prompt-engineering llms llm-eval llm-tools llm-evaluation llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework
-
Updated
Jan 17, 2025 - TypeScript
Multilingual Evaluation Toolkits
-
Updated
Nov 7, 2024 - Python
Community Plugin for Genkit to use Promptfoo
plugin testing firebase ai evaluation prompt prompts evaluation-framework llm llmops prompt-testing llm-eval llm-evaluation llm-evaluation-framework promptfoo genkit genkitx genkit-plugin
-
Updated
Jan 3, 2025 - TypeScript
Shin Rakuda is a comprehensive framework for evaluating and benchmarking Japanese large language models, offering researchers and developers a flexible toolkit for assessing LLM performance across diverse datasets.
-
Updated
Sep 17, 2024 - Python
An open-source evaluation framework for measuring LLM steerability.
-
Updated
May 17, 2025 - Jupyter Notebook
Hackable, simple, llm evals on preference datasets
-
Updated
May 25, 2025 - Python
Improve this page
Add a description, image, and links to the llm-evaluation-framework topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the llm-evaluation-framework topic, visit your repo's landing page and select "manage topics."