Skip to content
✂️ Repository for our ICLR 2019 paper: Discovery of Natural Language Concepts in Individual Units of CNNs:
Python Shell JavaScript
Branch: master
Clone or download
Type Name Latest commit message Commit time
Failed to load latest commit information.
code notify abs path of output file. Mar 9, 2019
figures cleanup. Mar 8, 2019
script minor update. Mar 8, 2019
.gitignore initial commit. Mar 7, 2019
LICENSE Initial commit Feb 23, 2019 minor update. Mar 8, 2019
environment.yml remove requirements.txt & add environment.yml Mar 8, 2019


This repository contains an implementation of our ICLR 2019 paper;

TL;DR: Individual units of deep CNNs learned in NLP tasks (e.g. translation, classification) could act as a natural language concept detector.

This work covers the interpretability of Deep Neural Network. We expect that it sheds useful light on how the representation of Deep CNNs learned in language tasks represents the given text.

We show that several information in the given text are not distributed across all units of representation. We observe AND quantify that even a single unit can act as a natural language concept (e.g. morpheme, word, phrase) detector.

Visualizing Individual Units

In this work, we align three natural language concepts per unit. Most units are selectively responsive to the concepts we align. If you want to see the full results, see Optional-Full Visualization Results.

Natural Language Concepts

Concepts that go beyond natural language form

We also discovered that several units tend to capture the concepts that go beyond natural langauge form. Although it is relatively hard to quantify it, we belive that further investigation would be one of interesting future direction. We visualize some units that capture abstract form concepts as follows:


Number + Time

Number + Question



A Demonstrative Pronoun

Similar Meaning or Role

Polarity (Positive)

Polarity (Negative)


If you want to get the results without running the code, skip these parts and see Optional-Full Visualization Results.


  • Python 2.7
  • anaconda (Python 2.7 version, latest version recommended)


  • Clone the code from GitHub.
git clone
  • Create environment via conda & downlaod spacy (english) model
conda env create -f environment.yml
conda activate iclr_19_na
python -m spacy download en
  • Download training data & pretrained models (~160GB space)
cd script

Running Visualization Code

cd script

will save visualization results at visualization/.

or skip to Optional-Full Visualization Results.

Optional-Full Visualization Results

cd script

or google drive link


If you find the code useful, please cite the following paper.

  title = "{Discovery of Natural Language Concepts in Individual Units of CNNs}",
  author = {Seil Na and Yo Joong Choe and Dong-Hyun Lee and Gunhee Kim},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {},


Each model used in our experiments is implemented based on this and this repository. We thank the authors.

We also appreciate Insu Jeon, Jaemin Cho, Sewon Min, Yunseok Jang and the anonymous reviewers for their helpful comments and discussions. This work was supported by Kakao and Kakao Brain corporations, IITP grant funded by the Korea government (MSIT) (No. 2017-0-01772) and Creative Pioneering Researchers Program through Seoul National University.


Have any question? Please contact:

You can’t perform that action at this time.