Skip to content
No description, website, or topics provided.
Branch: master
Clone or download
Latest commit f261463 Feb 13, 2018
Type Name Latest commit message Commit time
Failed to load latest commit information.
images Decoding Decoders Feb 13, 2018
sent_eval/evaluation Decoding Decoders Feb 13, 2018
skip_thoughts Decoding Decoders Feb 13, 2018
.gitignore Decoding Decoders Feb 13, 2018
LICENSE Decoding Decoders Feb 13, 2018
NOTICE Decoding Decoders Feb 13, 2018 Decoding Decoders Feb 13, 2018
WORKSPACE Decoding Decoders Feb 13, 2018
requirements.txt Decoding Decoders Feb 13, 2018 Decoding Decoders Feb 13, 2018 Decoding Decoders Feb 13, 2018

Decoding Decoders: Finding Optimal Representation Spaces for Unsupervised Similarity Tasks

TensorFlow implementation of the models described in the Decoding Decoders paper.

This codebase builds on top of Tensorflow Skip-Thought implementation by Chris Shallue and uses SentEval from Facebook for evaluations on transfer tasks.

The aim is to study how different choices of decoders affect the performance on unsupervised similarity tasks, such as STS.



This code uses Python 2.7. Please install the requirements in requirements.txt.

Data Preprocessing


You will need to obtain the BookCorpus dataset from this website.

Quick run

# Comma-separated list of globs matching the input files. The format of
# the input files is assumed to be a list of newline-separated sentences, where
# each sentence is already tokenized.
INPUT_FILES=<raw input files>

# Location to save the preprocessed training and validation data.
DATA_DIR=<data directory>

# Run the preprocessing script.
python -m \
    --input_files=${INPUT_FILES} \


Training params

We added a couple of new parameters in the script. The most important ones are described here, please see the code to see additional functionality we have added.

--decoder=SEQxSKGy where x, y can be 0, 1, 2, and 3.

SEQ stands for sequence (recurrent) decoder and SKG stands for bag-of-words (BOW) decoder.

  • 0 - no decoder of this type is present
  • 1 - decoder for the current sentence (Autoencoder)
  • 2 - decoders for the previous and next sentences (Skip-Though/FastSent style)
  • 3 - decoders for previous, current, and next sentences (Skip-Thought + Autoencoder) Note that it is possible to combine SEQ and SKG


  • True The architecture has a bag-of-words (BOW) encoder.
  • False The architecture has a sequence (RNN) encoder.

Defaults to False.

Quick run

# Directory containing the preprocessed data.
DATA_DIR=<data directory>

# Directory to save the model. Note: A new folder will be created in here called run_{unixtimestamp}. Into this folder, the model checkpoints will be saved. Also, the FLAGS wile, as well as its dict and json representations will be stored as `flags.pkl`, `config.pkl` and `config.json` respectively. 
RUN_DIR=<run directory>

# Model decoder configuration (choose one of SEQ0SKG2 SEQ0SKG3 SEQ2SKG2 or SEQ3SKG3)

# Whether to use skipgram (BOW) encoder (choose True or False). Defaults to False.

# Run the training script.

python -m skip_thoughts.train \
    --input_file_pattern="${DATA_DIR}/train-?????-of-00100" \
    --run_dir="${RUN_DIR}" \
    --decoder="${DECODER}" \

This will train a model with an RNN encoder and 2 BOW decoders.

Vocabulary Expansion


You will need to download the pretrained Google News word2vec vectors, found here. Please see the SkipThought readme for more details on vocab expansion.

Quick run

MODEL_DIR=<path to model>
SKIP_THOUGHTS_VOCAB=<path to skipthoughts vocab>
W2VMODEL=<path to W2V model>
LOG_FILE=<path to log file>

python -m skip_thoughts.vocabulary_expansion \
     --skip_thoughts_model="${MODEL_DIR}" \
     --skip_thoughts_vocab="${SKIP_THOUGHTS_VOCAB}" \
     --word2vec_model="${W2VMODEL}" \
     --output_dir="${MODEL_DIR}" \
     > "${LOG_FILE}" 2>&1



You will need to clone the SentEval repo and download the data as instructed there. Then copy our scripts from sent_eval/evaluation to the examples directory to run.

The scripts

The SentEval evaluation scripts either use the encoder output (which we confusingly call context here), or the unrolled decoder (which we less confusingly call unroll for the similarity and transfer tasks.

The similarity scripts and run the STS* tasks (STS12, STS13, STS14, STS15 and STS16) of SentEval.

The transfer scripts and run the transfer tasks (CR, MR, MPQA, SUBJ, SST, TREC, MRPC, SICKRelatedness, SICKEntailment and STSBenchmark) of SentEval.

Each script runs with 10-fold cross validation, and saves the dictionary of all results as a pickle to the desired location. This can then be used for easy generation of plots and other analysis.


The context scripts and work for all decoder types.

The parameters of the context scripts are:

  • --model_dir The path to the saved model you want to evaluate. Specifically, this should include this should be a folder containined checkpoint and decoder configuration information produced by
  • --output_results_path The full path to save the pickle file containing all of the results from this evaluation.


The unroll scripts and only work for RNN decoder types, and use the decoder unrolling mechanism discussed in the Decoding Decoders paper.

In addition to the parameters of the context scripts (above), the unroll scripts require the following parameters:

  • --unroll_length This should be a positive integer, and corresponds to how many time steps each decoder will "unroll" to produce the sentence representation.
  • --decoder_type This should be either 'mean' or 'concat' and corresponds to either taking the sentence representation as the mean or concatentation over the unrolled hidden states respectively.

Quick run

The example below is for running, the exact same process will work for the other evaluation scripts.

# Directory to load the model from
MODEL_DIR=<model directory>

# Which GPU(s) to use (choose from e.g. one of [0 1 0,1])

# Log file
LOG_FILE=<log file>

# Pickle save path
PICKLE_PATH=<pickle path>

# Run the evaluation script.
  python -m sent_eval.evaluation.similarity_context \
    --model_dir="${MODEL_DIR}" \
    --output_results_path="${PICKLE_PATH}" \
    > "${LOG_FILE}" 2>&1


Vitalii Zhelezniak

You can’t perform that action at this time.