-
Notifications
You must be signed in to change notification settings - Fork 96
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Read token attributes? #100
Comments
Hi - yes, the token list is just a list of tokens that can be iterated over/processed using |
Hi,
And these would be some strings to tokenize and parse
For instance, it should be checked if a string to parse contains tokens from the "placeholder" category |
Thanks for the follow-up. No, category information isn't exposed right now - you'd need to look it up manually based on |
Ok, thanks |
When iterating the tokens generated by the tokenizer, is it possible to check if any TokenAttribute (category, example or description) is set, and, if yes, to get their values ? It would be also interesting to be able to check directly if the token list contains any token from a certain category.
The text was updated successfully, but these errors were encountered: