You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
It's harder to support lower-end hardware (with limited memory) particularly with bigger dictionaries.
Describe the solution you'd like
I would like the option to be able to use a memory map instead to refer to an uncompressed dictionary since storage is usually cheaper than memory. The application I need does not need extreme performance so I feel like the IO penalty would be acceptable. If the dictionary gets processed by Vibrato into something else, it would also be nice to be able serialize it to a file and memmap it as well. fst offers something like that: https://docs.rs/fst/latest/fst/#example-stream-to-a-file-and-memory-map-it-for-searching
Describe alternatives you've considered
None that I'm aware of.
Additional context
None
The text was updated successfully, but these errors were encountered:
@kirawi I think the Vaporetto tokenizer is a better choice for small devices.
Lattice based tokenizers (including Vibrato) require large dictionaries, while pointwise tokenizers (including Vaporetto) work with smaller models.
Is your feature request related to a problem? Please describe.
It's harder to support lower-end hardware (with limited memory) particularly with bigger dictionaries.
Describe the solution you'd like
I would like the option to be able to use a memory map instead to refer to an uncompressed dictionary since storage is usually cheaper than memory. The application I need does not need extreme performance so I feel like the IO penalty would be acceptable. If the dictionary gets processed by Vibrato into something else, it would also be nice to be able serialize it to a file and memmap it as well.
fst
offers something like that: https://docs.rs/fst/latest/fst/#example-stream-to-a-file-and-memory-map-it-for-searchingDescribe alternatives you've considered
None that I'm aware of.
Additional context
None
The text was updated successfully, but these errors were encountered: