Refactor Code samples; Test code samples #5036
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #5036 +/- ##
==========================================
+ Coverage 79.08% 79.30% +0.22%
==========================================
Files 138 138
Lines 24078 24265 +187
==========================================
+ Hits 19041 19243 +202
+ Misses 5037 5022 -15
Continue to review full report at Codecov.
|
|
This is amazing! This way we won't do as many mistakes while copy-pasting code for introducing those task-specific models :-) |
thomwolf
approved these changes
Jun 16, 2020
src/transformers/file_utils.py
Outdated
| PYTORCH_MULTIPLE_CHOICE_CODE_SAMPLE_DOCSTRING = r""" | ||
| Examples:: | ||
|
|
||
| from transformers import BertTokenizer, BertForMultipleChoice |
There was a problem hiding this comment.
This should be « {tokenizer_class}, {model_class} «
src/transformers/file_utils.py
Outdated
| from transformers import BertTokenizer, BertForMultipleChoice | ||
| import torch | ||
|
|
||
| tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') |
src/transformers/file_utils.py
Outdated
| choice1 = "It is eaten while held in the hand." | ||
| labels = torch.tensor(0).unsqueeze(0) # choice0 is correct (according to Wikipedia ;)), batch size 1 | ||
|
|
||
| encoding = tokenizer.batch_encode_plus([[prompt, choice0], [prompt, choice1]], return_tensors='pt', pad_to_max_length=True) |
There was a problem hiding this comment.
And we probably want to remove batch_encode_plus here hahah
julien-c
approved these changes
Jun 16, 2020
LysandreJik
force-pushed
the
refactor-code-samples
branch
from
18bd438
to
929682e
Compare
LysandreJik
changed the title
LysandreJik
force-pushed
the
refactor-code-samples
branch
from
929682e
to
574575a
Compare
LysandreJik
changed the title
LysandreJik
commented
Jun 25, 2020
| @@ -633,7 +610,7 @@ def call( | |||
| mc_token_ids = inputs[6] if len(inputs) > 6 else mc_token_ids | |||
| output_attentions = inputs[7] if len(inputs) > 7 else output_attentions | |||
| assert len(inputs) <= 8, "Too many inputs." | |||
| elif isinstance(inputs, dict): | |||
| elif isinstance(inputs, (dict, BatchEncoding)): | |||
Comment on lines
-444
to
+417
| self.roberta = TFBertMainLayer(config, name="roberta") | ||
| self.roberta = TFRobertaMainLayer(config, name="roberta") |
| @@ -863,7 +843,7 @@ def call( | |||
| labels = inputs[4] if len(inputs) > 4 else labels | |||
| output_attentions = inputs[5] if len(inputs) > 5 else output_attentions | |||
| assert len(inputs) <= 6, "Too many inputs." | |||
| elif isinstance(inputs, dict): | |||
| elif isinstance(inputs, (BatchEncoding, dict)): | |||
Comment on lines
-935
to
+895
| assert len(inputs) <= 10, "Too many inputs." | ||
| output_hidden_states = inputs[10] if len(inputs) > 10 else output_hidden_states | ||
| labels = inputs[11] if len(inputs) > 11 else labels | ||
| assert len(inputs) <= 11, "Too many inputs." |
Comment on lines
-947
to
+909
| assert len(inputs) <= 10, "Too many inputs." | ||
| output_hidden_states = inputs.get("output_hidden_states", output_hidden_states) | ||
| labels = inputs.get("labels", labels) | ||
| assert len(inputs) <= 12, "Too many inputs." |
Comment on lines
-1004
to
+966
| config.num_labels, kernel_initializer=get_initializer(config.initializer_range), name="classifier" | ||
| config.num_labels, kernel_initializer=get_initializer(config.init_std), name="classifier" |
Comment on lines
+1008
to
+1014
| if isinstance(inputs, (tuple, list)): | ||
| labels = inputs[11] if len(inputs) > 11 else labels | ||
| if len(inputs) > 11: | ||
| inputs = inputs[:11] | ||
| elif isinstance(inputs, (dict, BatchEncoding)): | ||
| labels = inputs.pop("labels", labels) | ||
|
|
Comment on lines
-1075
to
+1032
| outputs = (logits,) + transformer_outputs[2:] # add hidden states and attention if they are here | ||
| outputs = (logits,) + transformer_outputs[1:] # add hidden states and attention if they are here |
sgugger
approved these changes
Jun 25, 2020
| @@ -978,36 +978,38 @@ def generate( | |||
|
|
|||
| Examples:: | |||
|
|
|||
| from transformers import AutoTokenizer, AutoModelForCausalLM | |||
There was a problem hiding this comment.
Can't manage to have a clean doctest with such a big example
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
jplu
pushed a commit
to jplu/transformers
that referenced
this pull request
Jun 29, 2020
* Refactor code samples * Test docstrings * Style * Tokenization examples * Run rust of tests * First step to testing source docs * Style and BART comment * Test the remainder of the code samples * Style * let to const * Formatting fixes * Ready for merge * Fix fixture + Style * Fix last tests * Update docs/source/quicktour.rst Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Addressing @sgugger's comments + Fix MobileBERT in TF Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactoring the code samples in order to prevent copy/pasting the same code samples across classes while updating the model/tokenizer classes and checkpoint names.