Unexpected tokenisation for string datatype

[ju-bezdek]

As a result of this thread
https://weaviate.slack.com/archives/C017EG2SL3H/p1667377648575099

it was discovered that string default tokenisation is word... which is very confusing, since in every programming language compare operators with string field have some expected outcome...

More over... it is not consistent with the docs... here:
https://weaviate.io/developers/weaviate/current/schema/datatypes.html#datatype-string-vs-text

is stated

DataType: string vs. text
There are two datatypes dedicated to saving textual information: string and text. string values are indexed as one token, whereas textvalues are indexed after applying tokenization.

Which is not true according to @etiennedi

Proposed solution:

• the default tokenisation for string should be field
• making the docs more clear about this...

citing @etiennedi:

 The main difference between text/string is whether non-alphanumeric characters are indexed or not.

[etiennedi]

This should become obsolete with the changes outlined in #2751. Do you agree? Then I'll close this one.

[ju-bezdek]

Yes, if it has been already decided to deprecate text, then this issue is irrelevant,
Although having string datatype as a pure value storage make sense to me, I can see that #2751 has been already planned, so this can be closed

[etiennedi]

Although having string datatype as a pure value storage make sense to me

Yes, we will keep the option of turning off any indexing (and, therefore any tokenization).

Thanks, closing this one.