Implements harmful/harmless refusal removal using pure HF Transformers

Distill thinking dataset more compactly and accurately!

Automatic evals for LLMs

Trying to create an rpg with llm being the game master.

Fast Semantic Text Deduplication

Lab Materials for MIT 6.S191: Introduction to Deep Learning

The Open Cookbook for Top-Tier Code Large Language Model

Sharing both practical insights and theoretical knowledge about LLM evaluation that we gathered while managing the Open LLM Leaderboard and designing lighteval!

The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices

A prompting library

A framework for the evaluation of autoregressive code generation language models.

An Open Source Toolkit For LLM Distillation

Code for In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering

Plug in and Play Implementation of Tree of Thoughts: Deliberate Problem Solving with Large Language Models that Elevates Model Reasoning by atleast 70%

Genetics for Language Models

Code and results accompanying the paper "Refusal in Language Models Is Mediated by a Single Direction".

[ICLR 2025] Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing. Your efficient and high-quality synthetic data generation pipeline!

Arena-Hard-Auto: An automatic LLM benchmark.

The official evaluation suite and dynamic data release for MixEval.

The Universe of Evaluation. All about the evaluation for LLMs.

Simple Python library/structure to ablate features in LLMs which are supported by TransformerLens

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.

Manage scalable open LLM inference endpoints in Slurm clusters

Evaluate your LLM's response with Prometheus and GPT4 💯