Tokenize new data

[eliotwalt]

Hi, I am trying to use the model on new data and struggle to reproduce the tokenization method to obtain the query_toks and query_toks_no_value fields. I tried using process_sql.tokenize which does not produce the same results as the dataset.
Is the code for this provided somewhere? Thanks.

[FruVirus]

So i ran into the same issue. I believe if you just don't lowercase the word within process_sql.tokenize, that should do the trick.

[FruVirus]

however, i have not figured out how to get the query_toks_no_value field

[listentomi]

however, i have not figured out how to get the query_toks_no_value field

So do I. I want to know if you solve this problem. Can you tell me how to generate the query_toks_no_value field?