New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error loading saved tokenizer #1255
Comments
We don't have the support with |
Sorry for not getting back to you sooner. The notification was buried in my inbox. There was a plan to clean up the code to extract the Python dependency and change the build step to CMake based one, but that has not occurred yet. Before doing so we need to add test for macOS and I was working on this in #1235 and #1300, but got busy. @parmeet If you would like to support this, we can discuss the plan. |
@mthrok sounds good! let's take a look at this. |
Hi @imagineer258 till we do the clean up and find a more convenient way to inject torchtext dependency into C++, you may try to manually link against Below is the sample CMake file that should do the job:
cc: @mthrok |
I believe, this can be resolved now by linking the C++ binary with torchtext library using the CMAKE built cc: @Nayef211 |
❓ Questions and Help
Description
When I try to load a saved tokenizer from torchtext I get the following error :
I saved the tokenizer with the following code :
and trying to load it back with
I'm assuming I have to link the torchtext c++ code correctly somewhere in my CMakeLists.txt but I'm not sure how to do that. I tried adding the following but it didn't help :
The text was updated successfully, but these errors were encountered: