Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"can't pickle Tokenizer objects" #87

Closed
578123043 opened this issue Jan 19, 2020 · 2 comments
Closed

"can't pickle Tokenizer objects" #87

578123043 opened this issue Jan 19, 2020 · 2 comments

Comments

@578123043
Copy link

Does it work well in multyprecess?
This was referenced Jan 22, 2020
Closed
Closed
@0xamogh
Copy link

0xamogh commented Apr 14, 2020

Facing the same issue with BertWordPieceTokenizer

@n1t0
Copy link
Member

n1t0 commented May 27, 2020

The ability to pickle tokenizers has landed on master and is part of the 0.8.0.dev1 pre-release.

@n1t0 n1t0 closed this as completed May 27, 2020
@rohitgr7 rohitgr7 mentioned this issue Jun 2, 2020
Closed
@jinyongyoo jinyongyoo mentioned this issue Jun 3, 2020
Merged
@sarahwie sarahwie mentioned this issue Jun 9, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@n1t0 @578123043 @0xamogh