awesome-interpretability

Mechanistic interpretability

• nnsight

  • To customize a model, instead of running it as a function, you run it as a "with" context. Inside "with" you can write regular pytorch to modify the computation.

  • aim to keep it as simple as bauket eventually, and support remote mechinterp. HuggingFace compatible
• Pyvene (intervention focused)

  • pyvene tries to be HuggingFace-native, supporting pre-defined interventions or customized interventions (below).

• TransformerLens
  • uses jaxtyping, aliases models into a common interface, not as huggingface compatible as other libs

  • an extremely opinionated toolkit for doing whatever you want to specific models,

• BauKit - light, simple, and well loved
• penzai - jax-based, not HuggingFace-native
• Transformer Debugger (OpenAI) - not HuggingFace-native
• Graphpatch - promising but abandoned
• NeuroX
• A tutorial on doing it manually

Explainability, counterfactuals and probing

• captum
• inseq
• Explabox (2022)
• IBM: AIX360 (2019)
• Microsoft: Responsible AI Toolbox (2021)
  • Dashboard that integrates: Error analysis, Fairlearn, InterpretML, DiCE, EconML and Data Balance
• InterpretML
  • SHAP, Mimic and LIME explainers. Permutation feature importance.
• MI2.ai
  • DrWhy (2019)
    • DALEX, survex, Arena, fairmodels,
  • Currently working on: ARES, xSurvival, Large Model Analysis
• XAI (2018)
• ELI5
• NN-SVG
• Neptune-AI blog
• Neptune-AI blog
• AI Ethics tool landscape

Structured output

• jsonformer
  • doesn't do enums
• outlines
• Microsoft Guidance
• lmql.ai
• llama.cpp grammar
• langchain output_parsers
• salute - typescript
• guardrails
• clownfish - 2023 Modifying Transformers to Follow a JSON Schema - not updated
• relm - 2023 Regular Expression engine for Language Models - not updated
• Constrained-Text-Generation-Studio
• kor
• lm-format-enforcer - remote api's
• instructor - for remote api's without logits
• Promptify
• prob_jsonformer - Jsonformer, but it can output the probablity of each choice in a single pass

See more

• s list that inspired this one
• the github interpretability topic
• https://github.com/wangyongjie-ntu/Awesome-explainable-AI
• https://twitter.com/davidbau/status/1785991694279913617