Branch: master
Find file History
Latest commit 2a36ecc Jul 20, 2017
Type Name Latest commit message Commit time
Failed to load latest commit information.
Session 1 - SENNA Bugfix preprocess Jul 18, 2017
Session 2 - Sentence CNN Ported code of BiLSTM to Keras 2.0.5 Jul 13, 2017
Session 3 - Relation CNN Ported code of BiLSTM to Keras 2.0.5 Jul 13, 2017
Session 4 - LSTM Sequence Classification readme Jul 13, 2017
docker Update Jul 18, 2017
1_Theory_Introduction.pdf Python 3.6 and Keras 2.0 update of scripts Jul 11, 2017
1_Theory_Introduction.pptx Python 3.6 and Keras 2.0 update of scripts Jul 11, 2017
2_Theory_Word_Embeddings.pdf Added PDF Jul 11, 2017
2_Theory_Word_Embeddings.pptx Encoding Jul 11, 2017
3_Theory_Frameworks.pdf Python 3.6 and Keras 2.0 update of scripts Jul 11, 2017
3_Theory_Frameworks.pptx Python 3.6 and Keras 2.0 update of scripts Jul 11, 2017
4_Theory_Convolutional_NN.pdf Python 3.6 and Keras 2.0 update of scripts Jul 11, 2017
4_Theory_Convolutional_NN.pptx Python 3.6 and Keras 2.0 update of scripts Jul 11, 2017
5_Theory_Recurrent_Neural_Networks.pptx Add slides to LSTM Jul 13, 2017 Update Jul 20, 2017
requirements.txt Python 3.6 and Keras 2.0 update of scripts Jul 11, 2017

Deep Learning for NLP - July 2017

This GIT repository accompanies the seminar on Deep Learning for Natural Language Processing.

In contrast to other seminars, this seminar focuses on the usage of deep learning methods. As programming infrastructure we use Python in combination with Keras. The published code can be used with Python 2.7 or Python 3.6, Keras 2.0.5, and Theano (0.9.0) or TensorFlow (1.2.1) backend. You should ensure that you have the frameworks installed in the right version (note: they change quickly).

This seminar is structured into 4 sessions:

  1. Feed-Forward Networks for Sequence Classification (e.g. POS, NER, Chunking)
  2. Convolutional Neural Network for Sentence / Text Classification (e.g. sentiment classification)
  3. Convolutional Neural Network for Relation Extraction (e.g. semantic relation extration)
  4. Long-Short-Term-Memory (LSTM)-Networks for Sequence Classificaiton

The seminar is inspired by an engineering mindset: The beautiful math and complexity of the topic is sometimes neglected to provide instead an easy-to-understand and easy-to-use approach to use Deep Learning for NLP tasks (we use what works without providing a full background on every aspect).

At the end of the seminar you should be able to understand the most important aspect of deep learning for NLP and be able to programm and train your own deep neural networks.

In case of questions, feel free to approach Nils Reimers.

Setting up the Development Environment

The codes in this folder were developed for Python 2.7 (and Python 3.6), Keras 2.0.5. As backend for Keras you can use Theano 0.9.0 or TensorFlow 1.2.1.

You can setup a virtual environment in the following way:

You can install the required packages in the following way:

pip install -r requirements.txt

Alternatively, it should be sufficient to just install Keras in version 2.0.5 and TensorFlow:

pip install Keras==2.0.5 TensorFlow==1.2.1

Virtual Environment

It can be useful, to run Python in a virtual environment for this seminar.

Create a virtualenv in the following way:

virtualenv .env
source .env/bin/activate

If you operate in the virtual environment, you can run pip to install the needed packages in the following way:

.env/bin/pip install -r requirements.txt


The folder docker contains a dockerfile that bundles an environment needed to run the experiments in this folder. It installs Python 3.6, Keras 2.0.5 and Tensorflow 1.2.1.

First, you need to build the docker container:

docker build ./docker -t dl4nlp

Than, in this folder you can start the container and mount files in this container to the docker container:

docker run -it -v ${PWD}:/usr/src/app dl4nlp bash

This will start a bash inside the dl4nlp container, where Python is installed. Through the mounting, you can modify the files in this folder and run them inside the docker container.

Recommended Readings on Deep Learning

The following is a short list with good introductions to different aspects of deep learning.

Theory 1 - Introduction to Deep Learning

Slides: pdf

The first theory lesson covers the fundamentals of deep learning.

Theory 2 - Introduction to Word Embeddings

Slides: pdf

Theory 3 - Introduction to Deep Learning Frameworks

Slides: pdf

The second lesson gives an overview of deep learning frameworks. Hint: Use Keras and have a look at Theano and TensorFlow.

Code Session 1 - SENNA Architecture for Sequence Classification

Slides: pdf

Code: See folder Session 1 - SENNA

The first code session is about the SENNA architecture (Collobert et al., 2011, NLP (almost) from scratch). In the folder you can find Python code for the preprocessing as well as Keras code to train and evaluate a deep learning model. The folder contains an example for Part-of-Speech tagging, which require the English word embeddings from either Levy et al. or from Komninos et al..

You can find in this folder also an example for German NER, based on the GermEval 2014 dataset. To run the German NER code, you need the word embeddings for German from our website.

Recommended Readings:

Theory 4 - Introduction to Convolutional Neural Networks

Slides: pdf

This is an introduction to Convolutional Neural Networks.

Recommended Readings:

Code Session 2 - Convolutional Neural Networks for Text Classification

Slides: pdf

Code: See folder Session 2 - Sentence CNN

This is a Keras implementation of the Kim, 2014, Convolutional Neural Networks for Sentence Classification. We use the same preprocessing as provided by Kim in his github repository but then implement the rest using Keras.

Code Session 3 - Convolutional Neural Networks for Relation Extraction

Slides: pdf

Code: See folder Session 3 - Relation CNN

This is an implementation for relation extraction. We use the SemEval 2010 - Task 8 dataset on semantic relations. We model the task as a pairwise classification task.

Recommended Readings:

Theory 5 - Introduction to LSTM

Slides: pdf

Code: See folder Session 4 - LSTM Sequence Classification

Note: An extended version of this LSTM architecture for sequence tagging can be found in the BiLSTM-CNN-CRF repository.

LSTMs are a powerful model and became very popular in 2015 / 2016.

Recommended Readings:

Code Session 4 - LSTM for Sequence Classification

Slides: pdf

The folder contains a Keras implementation to perfrom sequence classification using LSTM. We use the GermEval 2014 dataset for German NER. But you can adapt the code easily to any other sequence classification problem (POS, NER, Chunking etc.). Check the slides for more information.