Obsolete/replace Tokenizer.Position
to make stateless tokenizers possible
#12
Labels
Tokenizer.Position
to make stateless tokenizers possible
#12
While not all tokenizers will end up being thread-safe, this property precludes any instance of
Tokenizer<T>
from being thread-safe.Instead of storing the last-yielded token in a property on
Tokenizer
itself, an overload ofTokenize()
could be provided that accepts an additionalTokenizationState
parameter, or, we could extractStatefulTokenizer
out into a subclass ofTokenizer
and push down the property.The text was updated successfully, but these errors were encountered: