tokenizer save_tokens

void save_tokens()

---

Basic Info

Saves the tokens to a file.

Return Type

Returns void

---

Example

// Calling the save_tokens() function
tokenizer t("bestvocab.txt");
t.save_tokens;