A Simple Method for Commonsense Reasoning

[flrngel]

https://arxiv.org/abs/1806.02847
google brain

Abstract

• paper's model outperforms previous state-of-the-art methods in Winograd Schema challenges
• paper model uses large RNN Language Model

1. Introduction

• previous models on Winograd Challenge are poor
  • "The trophy doesn't fit in the suitcase because it is too big. What is too big?"
    • Answer0: the trophy [O]
    • Answer1: the suitcase [X]
  • "The trophy doesn't fit in the suitcase because it is too small. What is too small?"
    • Answer0: the trophy [X]
    • Answer1: the suitcase [O]

2. Related Work

• [16] Mikolov et al, Efficient estimation of word representations in vector space
  • similar but Winograd Schema questions require more contextual information
• Previous attempts on solving the Winograd Schema Challenge used these, but it was not good
  • heavy utilization of annotated knowledge bases
  • hand-crafted features
  • rule-based reasoning
• [27] Liu et al, Knowledge Enhanced Embeddings
  • includes supervised and unsupervised models
  • three different knowledge bases

3. Methods

• substitute the pronoun in original sentence with each of the candidate choices
  • this makes LM as solution
• 2 methods for scoring
  
  
  
• Negative sampling
  

5. Main results

6. Analysis

6.2 Partial scoring is better than full scoring

Appendix

• Author tested this paper with 2 version
  • both uses LSTM model
  • word level
    • 1024 dimension
  • character level
    • shape per word (50, 16) with <start of word>, <end of word>
    • use convolution and 2 fully connected layer to encode word (1024 dimension)

Note

• What is "importance negative sampling with negative sample size" means?
  • read http://mccormickml.com/2017/01/11/word2vec-tutorial-part-2-negative-sampling/