Contextualized Word Representations for Multi-Sense Embedding

Summary

Generate multiple word representations for each word in dependency structure relations.

Paper

Contextualized Word Representations for Multi-Sense Embedding

Author/Author

• Kazuki Ashihara (Osaka University)
• Tomoyuki Kajiwara (Osaka University)
• Yuki Arase (Osaka University)
• Satoru Uchida (Kyushu University)

Abstract

We propose methods to generate multiple word representations for each word based on the dependency structure relations. In order to deal with the data sparseness problem due to the increase in the size of vocabulary, the initial value for each word representations is determined using the pre-trained word representations. It is expected that the representations of low frequency words will remain in the vicinity of the initial value, which will in turn reduce the negative effects of data sparseness.

Novelty

• Capture word senses at a finer-grained level.
• Using dependency structure relations.
• Pre-training and Post-training.

Proposed Method

Result

• Context-Aware Word Similarity Task

• Lexical Substitution Task

Comment

I presented the same contents at IPSJNLP.
依存構造に基づく単語から語義の分散表現への細分化
芦原 和樹，梶原 智之，荒瀬 由紀(大阪大)，内田 諭(九州大)

Demo

• Pre-training to obtain a normal word representations.
• Identification of context-word.
• Post-trainig to learn the meaning of each context-word.

Test play

• Prepare a text file that is a morphological analysis / dependency structure analysis of sentences as corpus. Prepare test data for ./corpus/
• Exection make_context_word.py identify context-words from ./corpus/corpus_parse.txt

1. Exection pre_train.py. ./model/pre_train.model is generated.
2. make_context_word.py identifies context-words.
3. post-training makes ./model/post_train.model.

./model/post_train.model contains the word representations generated by context-words.

Attention

• By parsing, we ues Stanford Parser
• The code of word2vec referred to chainer.
• For simplicity this time, part of the code is different from what we actually used.