You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
what is the best way to save the vectorizer object for later use? Currently I am trying to use pickle, like so:
with open(f'{path}/vectorizer.pkl', 'wb') as f:
pickle.dump(vectorizer, f)
The resulting pickle file has a size of 6.9gb.
Thanks for your time.
The text was updated successfully, but these errors were encountered:
adkinsty
changed the title
Best way to save a fine-tuned Vectorizer object for later use
Best way to save a fine-tuned vectorizer object for later use
Oct 12, 2022
Ah, actually, perhaps I was confused. I had assumed that the .train() method does some sort of fitting/fine-tuning with the text whereas .infer() merely transforms the text. But if not, then there is no need to save the vectorizer for re-use. I can simply initialize a new vectorizer and use that to transform new text data.
P.S. the pre-trained model I'm using here is fasttext-crawl-subwords-300
Thanks for creating this package! I just have one quick question.
After fine-tuning the vectorizer on my text:
what is the best way to save the
vectorizer
object for later use? Currently I am trying to use pickle, like so:The resulting pickle file has a size of 6.9gb.
Thanks for your time.
The text was updated successfully, but these errors were encountered: