You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Crossandra.tokenize and Crossandra.tokenize_lines now accept an optional with_positions argument. When set to True, the resulting tokens will be of the (int, T) pair form, with the first element being the starting position of the token relative to the start of the string (or the start of the line when using tokenize_lines), and T being the actual token.