-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[gpt2pre 3] Preprocessor Layer #7794
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good, but can we have a simple test for this where it loads either a fake tokenizer (which you define and register in the test code) or the BytePairTokenizer? The reason for this is because it has a non-trivial fromConfig
function that initializes an additional layer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
* [gpt2pre 3] Preprocessor Layer (#7794) * Add Preprocessor layer * Remove uneeded args * Use LayerArgs * Remove import from src * Add fromConfig method * Serialize tokenizer properly * Add test cases for preprocessor * Preprocessor tests with no set tokenizer --------- Co-authored-by: Linchenn <40653845+Linchenn@users.noreply.github.com> * Remove npm proxy for @tensorflow scoped packages See #7557 --------- Co-authored-by: Piero F Orderique <45519489+pforderique@users.noreply.github.com> Co-authored-by: Linchenn <40653845+Linchenn@users.noreply.github.com> Co-authored-by: Matthew Soulanille <msoulanille@google.com>
Implements the base layer for processor layers, such as the GPT2 Preprocessor layer.
Update:
Some methods were left unimplemented and are pending discussionBased on discussion with @mattsoulanille,
presets
was removed, andfromConfig
andgetConfig
was implemented instead.