Add use_pretrained attribute for AutoTransformers #3498
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Will add a test |
|
Okay this is probably wrong, going to close and discuss this first. |
arnavgarg1
commented
Aug 4, 2023
tgaddair
reviewed
Aug 4, 2023
| @@ -3092,6 +3097,10 @@ def module_name(): | |||
| description=ENCODER_METADATA["AutoTransformer"]["type"].long_description, | |||
| ) | |||
|
|
|||
| # Always set this to True since we always want to use the pretrained weights | |||
| # We don't currently support training from scratch for AutoTransformers | |||
| use_pretrained: bool = True | |||
There was a problem hiding this comment.
Let's make this a property so the user could never modify it.
There was a problem hiding this comment.
@property
def use_pretrained(self) -> bool:
return True
jeffkinnison
approved these changes
Aug 4, 2023
| @@ -292,3 +289,42 @@ def test_tfidf_encoder(vocab_size: int): | |||
| inputs = torch.randint(2, (batch_size, sequence_length)).to(DEVICE) | |||
| outputs = text_encoder(inputs) | |||
| assert outputs[ENCODER_OUTPUT].shape[1:] == text_encoder.output_shape | |||
|
|
|||
|
|
|||
| def test_hf_auto_transformer_use_pretrained(tmpdir, csv_filename): | |||
There was a problem hiding this comment.
nit: The True case would be tested implicitly elsewhere, correct? If not, maybe we could parametrize the test with both cases.
tgaddair
reviewed
Aug 5, 2023
| text_feature( | ||
| encoder={ | ||
| "type": "auto_transformer", | ||
| "use_pretrained": False, |
There was a problem hiding this comment.
Ideally this should be an error if we were more strict with our config validation rules. We should instead just leave this out of the config.
tgaddair
approved these changes
Aug 5, 2023
dennisrall
pushed a commit
to dennisrall/ludwig
that referenced
this pull request
Aug 9, 2023
dennisrall
pushed a commit
to dennisrall/ludwig
that referenced
this pull request
Aug 9, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes the following error:
when trying to train an custom transformer model from HF using a config that looks like this: