Skip to content
Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

NeuroX Toolkit

Tests Runner API Reference

NeuroX provide all the necessary tooling to perform Interpretation and Analysis of (Deep) Neural Networks centered around Probing. Specifically, the toolkit provides:

  • Support for extraction of activation from popular models including the entirety of transformers, with extended support for other models like OpenNMT-py planned in the near future
  • Support for training linear probes on top of these activations, on the entire activation space of a model, on specific layers, or even on specific set of neurons.
  • Support for neuron extraction related to specific concepts, using the Linear Correlation Analysis method (What is one Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models.). The toolkit can extract either a local ranking of neurons important to a particular target class, or a global ranking of neurons important to all the target classes.
  • Support for ablation analysis by either removing or zeroing out specific neurons to determine their function and importance.
  • Support for subword and character level aggregation across a variety of tokenizers, including BPE and all tokenizers in the transformers library.
  • Support for activation visualization over regular text, to generate qualitative samples of neuron activity over particular sentences.

A demo using a lot of functionality provided by this toolkit is available.

demo picture

Getting Started

This toolkit requires and is tested on Python versions 3.6 and above. It may work with older Python versions with some fiddling, but is currently not tested nor supported. The easiest way to get started is to use the published pip package:

pip install neurox

Manual Installation

If you wish to install this package manually (e.g. to modify or contribute to the code base), you can clone this repository into a directory of your choice:

git clone

Add the directory to your python path. This can be done dynamically at runtime using the sys.path list:

import sys

A Conda environment is provided with all the necessary dependencies for the toolkit. The toolkit primarily relies on PyTorch and NumPy for most of its operations. To create a new environment with all the dependencies, run:

conda env create -f conda-environment.yml -n neurox-toolkit
conda activate neurox-toolkit

If you wish to manage your enviroment in other ways, a standard requirements.txt is also provided for use by pip directly.

Sample Code

A Jupyter notebook with a complete example of extracting activations from BERT, training a toy task, extracting neurons and visualizing them is available in the examples directory for a quick introduction to the main functionality provided by this toolkit.


API Reference contains an API reference for all of the functions exposed by this toolkit. Primarily, the toolkit's functionality is separated into several high-level components:

  • Extraction
  • Data Preprocessing
  • Linear Probing
  • Neuron extraction and interpretation
  • Neuron cluster analysis
  • Visualization


Please cite our paper published at AAAI'19 if you use this toolkit.

  title={NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks},
  author={Dalvi, Fahim
    and Nortonsmith, Avery
    and Bau, D Anthony
    and Belinkov, Yonatan
    and Sajjad, Hassan
    and Durrani, Nadir
    and Glass, James},
  journal={Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)},

Planned features

  • Pip package
  • Support for OpenNMT-py models
  • Support for control tasks and computing metrics like selectivity
  • Support for attention and other module analysis



A Python library that encapsulates various methods for neuron interpretation and analysis in Deep NLP models.





No releases published


No packages published