Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
..
Failed to load latest commit information.
BUILD
LICENSE
README.md
spinn.py

README.md

SPINN with TensorFlow eager execution

SPINN, or Stack-Augmented Parser-Interpreter Neural Network, is a recursive neural network that utilizes syntactic parse information for natural language understanding.

SPINN was originally described by: Bowman, S.R., Gauthier, J., Rastogi A., Gupta, R., Manning, C.D., & Potts, C. (2016). A Fast Unified Model for Parsing and Sentence Understanding. https://arxiv.org/abs/1603.06021

Our implementation is based on @jekbradbury's PyTorch implementation at: https://github.com/jekbradbury/examples/blob/spinn/snli/spinn.py,

which was released under the BSD 3-Clause License at: https://github.com/jekbradbury/examples/blob/spinn/LICENSE

Other eager execution examples can be found under tensorflow/contrib/eager/python/examples.

Content

  • data.py: Pipeline for loading and preprocessing the SNLI data and GloVe word embedding, written using the tf.data API.
  • spinn.py: Model definition and training routines. This example illustrates how one might perform the following actions with eager execution enabled:
    • defining a model consisting of a dynamic computation graph,
    • assigning operations to the CPU or GPU dependending on device availability,
    • training the model using the data from the tf.data-based pipeline,
    • obtaining metrics such as mean accuracy during training,
    • saving and loading checkpoints,
    • writing summaries for monitoring and visualization in TensorBoard.

To run

  • Make sure you have installed TensorFlow release 1.5 or higher. Alternatively, you can use the latest tf-nightly or tf-nightly-gpu pip package to access the eager execution feature.

  • Download and extract the raw SNLI data and GloVe embedding vectors. For example:

    curl -fSsL https://nlp.stanford.edu/projects/snli/snli_1.0.zip --create-dirs -o /tmp/spinn-data/snli/snli_1.0.zip
    unzip -d /tmp/spinn-data/snli /tmp/spinn-data/snli/snli_1.0.zip
    curl -fSsL http://nlp.stanford.edu/data/glove.42B.300d.zip --create-dirs -o /tmp/spinn-data/glove/glove.42B.300d.zip
    unzip -d /tmp/spinn-data/glove /tmp/spinn-data/glove/glove.42B.300d.zip
  • Train model. E.g.,

    python spinn.py --data_root /tmp/spinn-data --logdir /tmp/spinn-logs

    During training, model checkpoints and TensorBoard summaries will be written periodically to the directory specified with the --logdir flag. The training script will reload a saved checkpoint from the directory if it can find one there.

    To view the summaries with TensorBoard:

    tensorboard --logdir /tmp/spinn-logs
  • After training, you may use the model to perform inference on input data in the SNLI data format. The premise and hypotheses sentences are specified with the command-line flags --inference_premise and --inference_hypothesis, respectively. Each sentence should include the words, as well as parentheses representing a binary parsing of the sentence. The words and parentheses should all be separated by spaces. For instance,

    python spinn.py --data_root /tmp/spinn-data --logdir /tmp/spinn-logs \
        --inference_premise '( ( The dog ) ( ( is running ) . ) )' \
        --inference_hypothesis '( ( The dog ) ( moves . ) )'

    which will generate an output like the following, due to the semantic consistency of the two sentences.

    Inference logits:
      entailment:     1.101249 (winner)
      contradiction:  -2.374171
      neutral:        -0.296733
    

    By contrast, the following sentence pair:

    python spinn.py --data_root /tmp/spinn-data --logdir /tmp/spinn-logs \
        --inference_premise '( ( The dog ) ( ( is running ) . ) )' \
        --inference_hypothesis '( ( The dog ) ( rests . ) )'

    will give you an output like the following, due to the semantic contradiction of the two sentences.

    Inference logits:
      entailment:     -1.070098
      contradiction:  2.798695 (winner)
      neutral:        -1.402287