Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding #959

Open
icoxfog417 opened this issue Oct 12, 2018 · 6 comments

Comments

Projects
None yet
1 participant
@icoxfog417
Copy link
Member

commented Oct 12, 2018

一言でいうと

Bi-directionalのTransformerを事前学習し、QAや文関係推論などのタスクに転移した研究。ELMo(#655)の双方向性と、OpenAIのTransformer転移(#790)をミックスした形になっている。言語モデル学習は双方向でないのでcropした単語を予測する形で学習、文関係学習のため次の文or notを学習させる等の工夫を行っている。

論文リンク

https://arxiv.org/abs/1810.04805

著者/所属機関

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

  • Google AI Language

投稿日付(yyyy/MM/dd)

2018/10/11

概要

新規性・差分

手法

結果

コメント

@icoxfog417

This comment has been minimized.

@icoxfog417

This comment has been minimized.

Copy link
Member Author

commented Nov 1, 2018

@icoxfog417

This comment has been minimized.

Copy link
Member Author

commented Nov 19, 2018

@icoxfog417

This comment has been minimized.

Copy link
Member Author

commented Dec 5, 2018

日本語データセットでの学習
https://techlife.cookpad.com/entry/2018/12/04/093000

@icoxfog417

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.