Sep 17, 2018
1.4.1 RC1

@gkorland gkorland released this Aug 20, 2018 · 1 commit to 1.4 since this release

Assets 2

Update urgency: Medium - mainly due to numerous fixes

This version improves overall stability and performance of RediSearch. It also delivers better support for use cases in which documents are continuously updated. New features:

  • Conditional updates
  • Schema modification
  • Query spelling correction
  • Phonetic matching
  • Enhancement: More fuzziness in search

Continuous updates

Like most search engines, RediSearch was designed for maintaining append-mostly indices. To update an existing document, the document is actually replaced - that is deleted and added to the index.

Because RediSearch provides realtime indexing and searching it is very tempting to use for searching near-realtime data (or the products of its processing). In such cases, the number of indexed documents stays mostly static, but their contents are continuously updated.

To support this use case, RediSearch has been reworked internally to use 64-bit internal document IDs so that an index can sustain high update throughputs without overflowing. Furthermore and as a result, significant effort has been put into improving memory management and garbage collection.

Conditional updates

The IF subcommand has been add to FT.ADD. When used with the existing REPLACE [PARTIAL] subcommand, the document will be updated only if the condition provided evaluates to a truth value, otherwise a special NOADD reply is returned.

Schema modification

The FT.ALTER command has been introduced, and provides the ability to add new fields to the definition of an existing index. The contents of such newly-added fields are indexed only for new or updated documents.

Query spelling correction

Query spelling correction, a.k.a "did you mean", is now provided via the FT.SPELLCHECK command. It enables generating suggestions for search terms that could be misspelled. For more details see Query Spelling Correction.

Phonetic matching

Phonetic matching, a.k.a "Jon or John?", is now supported via the PHONETIC text field attribute. The terms in such fields are also indexed by their phonetic equivalents, and search results include these by default. For more details see Phonetic Matching.

More fuzziness in search

The fuzzy match operator, '%', can now be repeated up to three times to specify the Levenshtein distance. That means that the queries %hello%, %%hello%% and %%%hello%%% will perform fuzzy matching on 'hello' for all terms with LD of 1, 2 and 3, respectively.

Pre-release
Pre-release

@MeirShpilraien MeirShpilraien released this Aug 19, 2018 · 6 commits to 1.4 since this release

Assets 2
Merge pull request #438 from RedisLabsModules/fix_issue_437

see #437, spellcheck score on dictionaries will now be zero (and not -1)
Pre-release
Pre-release

@MeirShpilraien MeirShpilraien released this Aug 15, 2018 · 12 commits to 1.4 since this release

Assets 2
Merge pull request #432 from RedisLabsModules/issue-417

Use proper length type for sorting table. Fixes #417
Pre-release
Pre-release

@gkorland gkorland released this Aug 14, 2018 · 99 commits to master since this release

Assets 2
Merge pull request #413 from stockholmux/patch-2

Clarifying double escaping on tag fields (#411)
Aug 6, 2018
RC3 of 1.3.0
Jul 31, 2018
Preview of release 1.3.0
Pre-release
Pre-release

@gkorland gkorland released this Jul 29, 2018 · 126 commits to master since this release

Assets 2
Merge pull request #397 from RedisLabsModules/ld-percent

Allow LD of up to 3 with fuzzy matching via %-repetition

@mnunberg mnunberg released this Jun 19, 2018 · 368 commits to master since this release

Assets 2

This version is the last version that I (@dvirsky) will be releasing, and it includes a lot of new cool features, and actually NO bug-fixes over 1.1.0! So long and thanks for all the fish!

New Features

Aggregation Filters

As an addition to the aggregation API in FT.AGGREGATE, it is possible to do post-index filtering of the pipeline, using the FILTER keyword. e.g.:

FT.AGGREGATE idx "*"
   GROUPBY 1 @foo
     REDUCE count 0 AS num 
   FILTER "@num < 100"

See http://redisearch.io/Aggregations/ for more details.

Query Attributes

It is now possible to apply specific query modifying attributes to specific clauses of the query (see #212).

The syntax is (foo bar) => { $attribute: value; $attribute:value; ...}, e.g:

(foo bar) => { $weight: 2.0; $slop: 1 }
~(bar baz) => { $weight: 0.5; }

The supported attributes are:

1. $weight: determines the weight of the sub-query or token in the overall ranking on the result.

2. $slop: determines the maximum allowed "slop" (space between terms) in the query clause.

3. $inorder: whether or not the terms in a query clause must appear in the same order as in the query. 

Fuzzy Matching

Wrapping a search term with % will cause the index to expand the query to terms that are within an Edit Distance of 1 from the original term. For example, querying for %redis% will expand it to query for redis, jedis, credis, predis, etc (provided the terms appear in documents in the index).

Notice that each query term needs to be wrapped independently, and that we limit the maximum amount of expansions to 200 per term, as this hurts performance significantly.

Conditional Updates

It is now possible to update documents (FT.ADD ... REPLACE [PARTIAL]) only if a certain condition is met regarding the document's state before the updates. So for example, if our document has a timestamp field, and we would like to update its title only if the timestamp is below a certain value, we can do the following:

FT.ADD myIndex myDoc 1.0 
   REPLACE PARTIAL
   IF "@timestamp < 12313134523" 
   FIELDS 
       title "new title"

Backslash Escaping

Following several user requests, it is now possible to escape separator characters in documents (it is already done in the query itself), and avoid tokenization when needed.

For example, indexing the text hello\-world hello world will create the tokenization ["hello-world", "hello", "world"]. Notice that in most languages and in redis-cli, you will need to escape the backslash itself, so instead of hello\-world you will need to send the string hello\\-world.

The same goes to the query string: If the document contains the token hello-world, it can be found by running in redis-cli: `FT.SEARCH idx "hello\-world", which will cause the query not to be separated.

Synonyms Support

It is now possible to provide the index with synonym groups (.e.g boy, child, kid), and have it automatically index synonymous terms so that searching for one will return documents containing another.

See http://redisearch.io/Synonyms/ for more details and examples.

May 30, 2018
- Fix stuck loop at SIGUSR1
- Fix potential invalid memory access in numeric index