String comparison functions

[etcd]

Is your feature request related to a problem? Please describe.
String comparison functions do not exist in Manticore (other than equality). It is not possible to perform the following query:

CREATE TABLE mytable(mytitle string);
SELECT * FROM `mytable` WHERE `mytitle` < 'foo';

Describe the solution you'd like
The implementation of the following string comparison operators: <, <=, >, >=.

Additional context
This functionality is essential to implement relatively mainstream functionalities, such as cursor-based pagination on string columns.

[sanikolaev]

Thanks for the feature request, @etcd

@tomatolog As discussed, please prepare instructions for @etcd detailing the areas that require modifications, as they would like to implement these changes themselves.

[tomatolog]

In general you need create a new filter instance at sphCreateFilter similar to FilterString_c but one that can compare strings with lesser and greater operators.

You also should add new members to CommonFilterSettings_t for parser to fill. The parser located at bison and flex files I'd start filter_item

And after all works well locally you need to add send and receive of the new filter structure into distributed index SearchRequestBuilder_c::SendQuery and SearchReplyParser_c::ParseReply

Feel free to ask for any help

[etcd]

Hey @tomatolog @sanikolaev, I've spent a few hours familiarizing myself with the codebase. I looked into modeling these string filters after Filter_Range or Filter_FloatRange (both in sphinxfilter.cpp). However there are existing architectural choices that make this route of implementation complicated.

For example, in several places, ranges are assumed to be numerical (either ints or floats - so adding strings to the concept of "ranges" would be challenging). This is a non-exhaustive list:

• jsonqueryfilter.cpp
• sphinxfilter.cpp
• ha_sphinx.cc

So implementing these string filters the same way seems like a somewhat error-prone route, given the number of places that might make this assumption. SPH_FILTER_RANGE appears 81 times in 22 files, and SPH_FILTER_FLOATRANGE appears 54 times in 19 files.

Not to mention there are more subtle cases where ranges are assumed to be numerical, like in EvalRange (sphinxfilter.h), where numerical comparison operators are used - but if we want to compare strings, we need to genericize the comparators so we can pass in strcmp.

There might be simpler ways of adding these string filters that don't involve expanding the existing "range" abstraction and having to ensure that all semantics continue to hold. I'll defer to your judgements.

[tomatolog]

I'd add a new FILTER enum similar to SPH_FILTER_STRING \ SPH_FILTER_STRING_LIST these do not match to SPH_FILTER_VALUES and make separate code that handles only this case - no need to add string into existed SPH_FILTER_RANGE. It could be better to create complete new code without touching most of existed code.

[etcd]

I'd find it helpful if you could also outline which files are deprecated/unused and should not be touched (if there's an easy way to do that).

For example, is mysqlse/ha_sphinx.cc just dead code? It contains a enum ESphFilter, but it seems to me that the enum of the same name in src/sphinxdefs.h is the more complete enum that's actually used.

Also, how about api/ruby/? According to component-licenses/README.md, the Ruby client is old. It is most likely very outdated. Am I supposed to update it? It is supposed to track known filter types, but if this client is unmaintained, then it's probably not worth updating.

In general, it might be a good idea to go through an anti-entropy effort to remove dead code and make development easier (esp for outside contributors).

[etcd]

Hi @tomatolog , please check out the draft PR. I've made updates to all places mentioned in your outline. I've left it as a draft PR because I think there may be additional changes necessary that I would like to double check first.

[tomatolog]

you do not need to implement any API or the mysqlse as it is another kind of the API as we already has features work only via SphinxQL or HTTP JSON

you need just to implement code:

• in the daemon for the local index
• in the daemon for distributed index (pass query into remote agent from master) and increase protocol version

It also worth to add test case as a new test/test_4XX for testing with the uberstest suite

[tomatolog]

implemented at PR#1900