[EncoderDecoder] Add encoder-decoder for roberta/ vanilla longformer #6411

patrickvonplaten · 2020-08-11T12:40:55Z

This PR adds Roberta to the Encoder Decoder framework. Thus, it automatically makes it possible to use both Roberta2Roberta models and Longformer2Roberta model:

from transformers import EncoderDecoderModel
model = EncoderDecoderModel.from_pretrained("roberta-base", "roberta-base")
input_ids = torch.tensor([10 * [0]])
model(input_ids=input_ids, decoder_input_ids=input_ids)

and

from transformers import EncoderDecoderModel
model = EncoderDecoderModel.from_pretrained("allenai/longformer-base-4096", "roberta-base")
input_ids = torch.tensor([10 * [0]])
model(input_ids=input_ids, decoder_input_ids=input_ids)

Also pinging @ibeltagy and @patil-suraj

src/transformers/modeling_bert.py

tests/test_modeling_bert.py

src/transformers/modeling_encoder_decoder.py

codecov · 2020-08-11T13:02:12Z

Codecov Report

Merging #6411 into master will decrease coverage by 1.93%.
The diff coverage is 92.85%.

@@            Coverage Diff             @@
##           master    #6411      +/-   ##
==========================================
- Coverage   79.77%   77.84%   -1.94%     
==========================================
  Files         150      150              
  Lines       27789    27826      +37     
==========================================
- Hits        22170    21660     -510     
- Misses       5619     6166     +547

Impacted Files	Coverage Δ
src/transformers/__init__.py	`99.25% <ø> (ø)`
src/transformers/modeling_auto.py	`63.95% <ø> (-14.54%)`	⬇️
src/transformers/modeling_encoder_decoder.py	`91.02% <ø> (ø)`
src/transformers/modeling_bert.py	`88.42% <50.00%> (ø)`
src/transformers/modeling_tf_bert.py	`96.22% <50.00%> (-0.36%)`	⬇️
src/transformers/modeling_roberta.py	`95.98% <97.36%> (+0.20%)`	⬆️
src/transformers/optimization.py	`28.94% <0.00%> (-67.11%)`	⬇️
src/transformers/modeling_tf_flaubert.py	`24.53% <0.00%> (-63.20%)`	⬇️
src/transformers/pipelines.py	`26.98% <0.00%> (-52.81%)`	⬇️
src/transformers/optimization_tf.py	`33.33% <0.00%> (-24.33%)`	⬇️
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4047829...ed8414a. Read the comment docs.

sgugger

Great to see the encoder/decoder framework expanded! Thanks for all the work!

src/transformers/modeling_roberta.py

sgugger · 2020-08-11T14:47:21Z

src/transformers/modeling_roberta.py

+        output_attentions=None,
+        output_hidden_states=None,
+        return_dict=None,
+        **kwargs


Are you using those kwargs? If so change the docstrings since there is no legacy arguments here.

Removed them from the corresponding BERT model as well.

tests/test_modeling_bert.py

sshleifer

LGTM. Have people had good ROUGE with the compose two pretrained glue models and finetune for summarization approach?

patrickvonplaten · 2020-08-11T16:18:25Z

LGTM. Have people had good ROUGE with the compose two pretrained glue models and finetune for summarization approach?

Hmm, I think it's very new so not sure if many people have tried out the framework yet. @patil-suraj - do you know if people work a lot with EncoderDecoder by chance?

patil-suraj · 2020-08-11T16:43:18Z

do you know if people work a lot with EncoderDecoder by chance?

Seems like it, seen quite a few issues and questions (on forum as well) regarding EncoderDecoder, but no one has reported any good results yet

ibeltagy · 2020-08-11T17:40:42Z

Looks great. Thanks, @patrickvonplaten.

LGTM. Have people had good ROUGE with the compose two pretrained glue models and finetune for summarization approach?

@sshleifer, was thinking about the same thing. My guess is that numbers won't be great because cross-attention is randomly initialized?

LysandreJik

Great, LGTM!

LysandreJik · 2020-08-12T10:15:27Z

src/transformers/modeling_roberta.py

+@add_start_docstrings(
+    """RoBERTa Model with a `language modeling` head on top for CLM fine-tuning. """, ROBERTA_START_DOCSTRING
+)
+class RobertaForCausalLM(BertPreTrainedModel):


Wouldn't it be more coherent to have it as RobertaLMHeadModel?

(But I do prefer RobertaForCausalLM)

It's the same names for BERT.

Following internal discussion will leave the name as it is more precise and BertLMHeadModel should change in the future.

tests/test_modeling_bert.py

LysandreJik · 2020-08-12T10:18:27Z

src/transformers/modeling_bert.py

-        labels=None,
        encoder_hidden_states=None,
        encoder_attention_mask=None,
+        labels=None,


LysandreJik · 2020-08-12T10:19:25Z

src/transformers/modeling_roberta.py

+        kwargs (:obj:`Dict[str, any]`, optional, defaults to `{}`):
+            Used to hide legacy arguments that have been deprecated.


No need for this I think

src/transformers/modeling_roberta.py

patrickvonplaten · 2020-08-17T09:25:09Z

Looks great. Thanks, @patrickvonplaten.

LGTM. Have people had good ROUGE with the compose two pretrained glue models and finetune for summarization approach?

@sshleifer, was thinking about the same thing. My guess is that numbers won't be great because cross-attention is randomly initialized?

Btw, this paper does some great analysis on reusing checkpoints for Seq2Seq models: https://arxiv.org/pdf/1907.12461.pdf

…gformer (huggingface#6411)" This reverts commit 8b8f41f.

add encoder-decoder for roberta

fb07b7f

patrickvonplaten commented Aug 11, 2020

View reviewed changes

src/transformers/modeling_bert.py Show resolved Hide resolved

patrickvonplaten commented Aug 11, 2020

View reviewed changes

tests/test_modeling_bert.py Show resolved Hide resolved

fix headmask

9cc70d6

patrickvonplaten requested review from sshleifer, sgugger and LysandreJik August 11, 2020 12:57

patrickvonplaten changed the title ~~[EncoderDecoder] Add encoder-decoder for roberta~~ [EncoderDecoder] Add encoder-decoder for roberta/ vanilla longformer Aug 11, 2020

patrickvonplaten commented Aug 11, 2020

View reviewed changes

src/transformers/modeling_encoder_decoder.py Show resolved Hide resolved

sgugger approved these changes Aug 11, 2020

View reviewed changes

sshleifer approved these changes Aug 11, 2020

View reviewed changes

apply Sylvains suggestions

a6b2619

fix typo

756f8b0

LysandreJik approved these changes Aug 12, 2020

View reviewed changes

patrickvonplaten commented Aug 12, 2020

View reviewed changes

src/transformers/modeling_roberta.py Outdated Show resolved Hide resolved

patrickvonplaten commented Aug 12, 2020

View reviewed changes

src/transformers/modeling_roberta.py Outdated Show resolved Hide resolved

Apply suggestions from code review

ed8414a

patrickvonplaten merged commit 0735def into huggingface:master Aug 12, 2020

patil-suraj mentioned this pull request Aug 20, 2020

CamembertForCausalLM #6577

Merged

fabiocapsouza added a commit to fabiocapsouza/transformers that referenced this pull request Nov 15, 2020

Revert "[EncoderDecoder] Add encoder-decoder for roberta/ vanilla lon…

250b2cb

…gformer (huggingface#6411)" This reverts commit 8b8f41f.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[EncoderDecoder] Add encoder-decoder for roberta/ vanilla longformer #6411

[EncoderDecoder] Add encoder-decoder for roberta/ vanilla longformer #6411

patrickvonplaten commented Aug 11, 2020 •

edited

Loading

codecov bot commented Aug 11, 2020 •

edited

Loading

sgugger left a comment

sgugger Aug 11, 2020

patrickvonplaten Aug 11, 2020

sshleifer left a comment

patrickvonplaten commented Aug 11, 2020

patil-suraj commented Aug 11, 2020

ibeltagy commented Aug 11, 2020 •

edited

Loading

LysandreJik left a comment

LysandreJik Aug 12, 2020

LysandreJik Aug 12, 2020

sgugger Aug 12, 2020 •

edited

Loading

patrickvonplaten Aug 12, 2020

LysandreJik Aug 12, 2020

LysandreJik Aug 12, 2020

patrickvonplaten commented Aug 17, 2020 •

edited

Loading

		kwargs (:obj:`Dict[str, any]`, optional, defaults to `{}`):
		Used to hide legacy arguments that have been deprecated.

[EncoderDecoder] Add encoder-decoder for roberta/ vanilla longformer #6411

[EncoderDecoder] Add encoder-decoder for roberta/ vanilla longformer #6411

Conversation

patrickvonplaten commented Aug 11, 2020 • edited Loading

codecov bot commented Aug 11, 2020 • edited Loading

Codecov Report

sgugger left a comment

Choose a reason for hiding this comment

sgugger Aug 11, 2020

Choose a reason for hiding this comment

patrickvonplaten Aug 11, 2020

Choose a reason for hiding this comment

sshleifer left a comment

Choose a reason for hiding this comment

patrickvonplaten commented Aug 11, 2020

patil-suraj commented Aug 11, 2020

ibeltagy commented Aug 11, 2020 • edited Loading

LysandreJik left a comment

Choose a reason for hiding this comment

LysandreJik Aug 12, 2020

Choose a reason for hiding this comment

LysandreJik Aug 12, 2020

Choose a reason for hiding this comment

sgugger Aug 12, 2020 • edited Loading

Choose a reason for hiding this comment

patrickvonplaten Aug 12, 2020

Choose a reason for hiding this comment

LysandreJik Aug 12, 2020

Choose a reason for hiding this comment

LysandreJik Aug 12, 2020

Choose a reason for hiding this comment

patrickvonplaten commented Aug 17, 2020 • edited Loading

patrickvonplaten commented Aug 11, 2020 •

edited

Loading

codecov bot commented Aug 11, 2020 •

edited

Loading

ibeltagy commented Aug 11, 2020 •

edited

Loading

sgugger Aug 12, 2020 •

edited

Loading

patrickvonplaten commented Aug 17, 2020 •

edited

Loading