Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BPE PR 2.1] BPE Utils #7760

Merged
merged 27 commits into from
Jun 21, 2023
Merged

[BPE PR 2.1] BPE Utils #7760

merged 27 commits into from
Jun 21, 2023

Conversation

pforderique
Copy link
Contributor

Included is bytesToUnicode, createStaticHashTable, and BytePairTokenizerCache with respective tests that will be needed for BPE.

pforderique and others added 24 commits June 14, 2023 21:46
@pforderique
Add Tokenizer base class.
3a655f4
@pforderique
Update licence to 2023
85e34c3
@pforderique
Fix lint errors.
8121219
@pforderique
Only expose WhiteSpaceTokenizer in tests
7b9c1f2
@pforderique
Rename WhitespaceTokenizer to SimpleTokenizer
cd2fc6e
@pforderique
Add example to Tokenizers docstring.
22e7b81
@pforderique
BytePairEncoding implementation started.
992fb99
@pforderique
Update test name to Tokenizer
2f3e944
@pforderique @mattsoulanille
Destructure TokenizerOptions to assign default mode value
cf8c974 
Co-authored-by: Matthew Soulanille <matthew@soulanille.net>
@pforderique
Use destructured mode in call method
9eb393a
@pforderique
Wrap tokenizer in beforeEach clause.
06b4f26
@pforderique
Make TokenizerOptions optional in call()
b4912ab
@pforderique
Add utils file for tokenizers.
d805bb8
@pforderique @mattsoulanille
Update example in Tokenizers to print output.
9a3ef53 
Co-authored-by: Matthew Soulanille <matthew@soulanille.net>
@pforderique
Don't register test SimpleTokenizer
2bd1943
@pforderique
Bring in changes from Tokenizer
af76234
@pforderique
Add bytesToUnicode and HashTable for tensors.
a00f45c
@pforderique
Bring in changes from main
b792a41
@pforderique
Merge in tokenizers_utils file
e5b9a59
@pforderique
Add utils for BPE
0a83a76
@pforderique
Change test name to createStaticHashtable
9b909c2
@pforderique
Add BytePairTokenizer Cache
836a5be
@pforderique
Add tests for BytePairTokenizerCache
f1a72d9
@pforderique
Move BytePairTokenizerCache to correct location
ebed82c
@mattsoulanille mattsoulanille self-requested a review June 16, 2023 20:56
mattsoulanille
mattsoulanille requested changes Jun 16, 2023
View reviewed changes
Copy link
Member

@mattsoulanille mattsoulanille left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests look good! I have a few minor comments about the implementation.

tfjs-layers/src/layers/nlp/tokenizers_utils.ts Outdated Show resolved Hide resolved
tfjs-layers/src/layers/nlp/tokenizers_utils.ts Outdated Show resolved Hide resolved
tfjs-layers/src/layers/nlp/tokenizers_utils.ts Outdated Show resolved Hide resolved
tfjs-layers/src/layers/nlp/tokenizers_utils.ts Outdated Show resolved Hide resolved
tfjs-layers/src/layers/nlp/tokenizers_utils.ts Outdated Show resolved Hide resolved
tfjs-layers/src/layers/nlp/tokenizers_utils.ts Outdated Show resolved Hide resolved
tfjs-layers/src/layers/nlp/tokenizers_utils.ts Outdated Show resolved Hide resolved
tfjs-layers/src/layers/nlp/tokenizers_utils.ts Outdated Show resolved Hide resolved
Linchenn
Linchenn approved these changes Jun 16, 2023
View reviewed changes
Copy link
Collaborator

@Linchenn Linchenn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

mattsoulanille
mattsoulanille approved these changes Jun 21, 2023
View reviewed changes
Copy link
Member

@mattsoulanille mattsoulanille left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@mattsoulanille mattsoulanille enabled auto-merge (squash) June 21, 2023 17:40
@mattsoulanille mattsoulanille merged commit 2ea2292 into tensorflow:master Jun 21, 2023
2 checks passed
@pforderique pforderique deleted the orderique branch June 21, 2023 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mattsoulanille mattsoulanille mattsoulanille approved these changes

@Linchenn Linchenn Linchenn approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@pforderique @mattsoulanille @Linchenn