Tag queries choke on certain punctuation

[stockholmux]

According to the documentation:

The tokenization is simpler: The user can determine a separator (defaults to a comma) for multiple tags, and we only do whitespace trimming at the end of tags. Thus, tags can contain spaces, punctuation marks, accents, etc. The only two transformations we perform are lower-casing (for latin languages only as of now), and whitespace trimming.

Test schema and a couple documents

127.0.0.1:6379> ft.create testtags schema atag tag
OK
127.0.0.1:6379> ft.add testtags mydoc1 1 fields atag "hello:42"
OK
127.0.0.1:6379> ft.add testtags mydoc2 1 fields atag "hello;42"
OK

So, you can add documents with tag field values that contain punctuation, but you can't query them.

127.0.0.1:6379> ft.search testtags "@atag:{ hello:42 }"
(error) Syntax error at offset 13 near 'hello'
127.0.0.1:6379> ft.search testtags "@atag:{ hello;42 }"
(error) Syntax error at offset 13 near 'hello'

Seems to be choking at the [semi-]colons. Escaping doesn't work either:

127.0.0.1:6379> ft.search testtags "@atag:{ hello\:42 }"
(error) Syntax error at offset 13 near 'hello'
127.0.0.1:6379> ft.search testtags "@atag:{ hello\;42 }"
(error) Syntax error at offset 13 near 'hello'

Some punctuation seems to be OK.

127.0.0.1:6379> ft.search testtags "@atag:{ hello>42 }"
1) (integer) 0
127.0.0.1:6379> ft.search testtags "@atag:{ hello!42 }"
1) (integer) 0
127.0.0.1:6379> ft.search testtags "@atag:{ hello^42 }"
1) (integer) 0
127.0.0.1:6379> ft.search testtags "@atag:{ hello 42 }"
1) (integer) 0

Further testing:

127.0.0.1:6379> ft.search testtags "@name:{ hello:42 }"
(error) Syntax error at offset 13 near 'hello'
127.0.0.1:6379> ft.search testtags "@atag:{ hello;42 }"
(error) Syntax error at offset 13 near 'hello'
127.0.0.1:6379> ft.search testtags "@atag:{ hello$42 }"
(error) Syntax error at offset 13 near '42'
127.0.0.1:6379> ft.search testtags "@atag:{ hello%42 }"
(error) Syntax error at offset 14 near '42'
127.0.0.1:6379> ft.search testtags "@atag:{ hello@42 }"
(error) Syntax error at offset 17 near '42'
127.0.0.1:6379> ft.search testtags "@atag:{ hello*42 }"
(error) Syntax error at offset 17 near '42'
127.0.0.1:6379> ft.search testtags "@atag:{ hello(42 }"
(error) Syntax error at offset 17 near '42'

Other punctuation seems to be all over the place as far as where the error is located.

[stockholmux]

Probably related.

If you do use non-syntax error producing punctuation in the queries, the query parser seemingly still tokenizes the tag:

127.0.0.1:6379> ft.add testtags foo2 1 fields atag "hello!42"
OK
127.0.0.1:6379> ft.search testtags "@atag:{ hello!42 }"
0
127.0.0.1:6379> ft.explain testtags "@atag:{ hello!42 }"
TAG:@atag {
  INTERSECT {
    hello
    42
  }
}

I would expect the explain to be this instead of the INTERSECT:

> ft.explain testtags "@atag:{ hello!42 }"
TAG:@atag {
  hello!42
}

[dvirsky]

but why? all punctuations are tokenizing Kyle. escaping them needs double backslashes. There's no bug here AFAICT

[stockholmux]

Glad that worked! Although we need to change the documentation now - the only place I'm seeing double escaping is on the Tokenization and Escaping page that specifically says it's only for text fields and says to look on the tag page for a description of how tagging tokenization/escaping works - no mention of double escaping there (only single).