Skip to content

add classifier_dropout to classification heads - #12794

Merged
sgugger merged 13 commits into
huggingface:masterfrom
German-NLP-Group:add_head_dropout
Jul 26, 2021
Merged

add classifier_dropout to classification heads#12794
sgugger merged 13 commits into
huggingface:masterfrom
German-NLP-Group:add_head_dropout

Conversation

@PhilipMay

@PhilipMay PhilipMay commented Jul 19, 2021

Copy link
Copy Markdown
Contributor

PR for #12781 and maybe a fix for #12792

Logbook

  • added to Electra: SequenceClassification and TokenClassification
  • added to BERT: SequenceClassification and TokenClassification
  • added to RoBERTa: SequenceClassification and TokenClassification
  • added to big_bird: SequenceClassification and TokenClassification
  • added to mobilebert: SequenceClassification and TokenClassification
  • added to reformer: SequenceClassification and TokenClassification
  • added to ConvBERT: SequenceClassification and TokenClassification
  • added to albert: SequenceClassification and TokenClassification

@PhilipMay PhilipMay changed the title add classifier_dropout to classification heads [WIP] add classifier_dropout to classification heads Jul 19, 2021

@sgugger sgugger left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be added to the other models too!

Comment thread src/transformers/models/electra/configuration_electra.py Outdated
Comment thread src/transformers/models/electra/configuration_electra.py Outdated
@sgugger
sgugger requested a review from LysandreJik July 19, 2021 17:48
@PhilipMay
PhilipMay force-pushed the add_head_dropout branch 3 times, most recently from cc8d91e to 4714546 Compare July 19, 2021 18:40
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
@PhilipMay

Copy link
Copy Markdown
Contributor Author

@sgugger should classifier_dropout be only added for <model_name>ForSequenceClassification
or also to <model_name>ForTokenClassification?

@PhilipMay
PhilipMay force-pushed the add_head_dropout branch 2 times, most recently from 250a952 to 4714546 Compare July 19, 2021 19:10
@sgugger

sgugger commented Jul 19, 2021

Copy link
Copy Markdown
Collaborator

It should be used for both, yes.

@LysandreJik LysandreJik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@PhilipMay

Copy link
Copy Markdown
Contributor Author

This is still WIP - I want to add more models. Please do not merge yet ...

@PhilipMay PhilipMay mentioned this pull request Jul 21, 2021
@PhilipMay PhilipMay changed the title [WIP] add classifier_dropout to classification heads add classifier_dropout to classification heads Jul 22, 2021
@PhilipMay

Copy link
Copy Markdown
Contributor Author

I am done with this PR. It is ready for review.
See logbook above for changed model types.

@PhilipMay
PhilipMay requested review from LysandreJik and sgugger July 22, 2021 19:38

@JetRunner JetRunner left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never mind :)

@sgugger sgugger left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again for working on this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants