2.0 GA (v2.0.0)

This is the GA release for RedisSearch 2.0. This release includes several improvements in performance and usability over RediSearch 1.0. These improvements necessitate a few backward-breaking changes to the API.

Highlights

For this release, we changed the way in which the search indexes are kept in sync with your data. In RediSearch 1.x, you had to manually add data to your indexes using the FT.ADD command. In RediSearch 2.x, your data is indexed automatically based on a key pattern.

These changes are designed to enhance developer productivity, and to ensure that
your search indexes are always kept in sync with your data. To support this, we've
made a few changes to the API.

In addition to simplifying indexing, RediSearch 2.0 allows you to scale a single index over multiple Redis shards using the Redis cluster API.

Finally, RediSearch 2.x keeps its indexes outside of the main Redis key space. Improvements to the indexing code have increased query performance 2.4x.

You can read more details in the RediSearch 2.0 announcement blog post, and you can get started by checking out this quick start blog post.

Details

• When you create an index, you must specify a prefix condition and/or a filter. This determines which hashes RediSearch will index.
• Several RediSearch commands now map to their Redis equivalents: FT.ADD -> HSET, FT.DEL -> DEL (equivalent to FT.DEL with the DD flag in RediSearch 1.x), FT.GET -> HGETALL, FT.MGET -> HGETALL.
• RediSearch indexes no longer reside within the key space, and the indexes are no longer saved to the RDB.
• You can upgrade from RediSearch 1.x to RediSearch 2.x.

Noteworthy changes

• #1246: geodistance function for FT.AGGREGATE APPLY operation.
• #1394: Expired documents (TTL) will be removed from the index.
• #1394: Optimization to avoid reindexing documents when non-indexed fields are updated.
• After index creation, an initial scan starts for existing documents. You can check the status of this scan by calling FT.INFO and looking at the indexing and percent_indexed values. While indexing is true, queries return partial results.
• #1435: NOINITIALINDEX flag on FT.CREATE to skip the initial scan of documents on index creation.
• #1401: Support upgrade from v1.x and for reading RDB's created by RediSearch 1.x (more information).
• #1445: Support for load event. This event indexes documents when they are loaded from RDB, ensuring that indexes are fully available when RDB loading is complete (available from Redis 6.0.7 and above).
• #1384: FT.DROPINDEX, which by default does not delete documents underlying the index (see deprecated FT.DROP).
• #1385: Add index definition to FT.INFO response.
• #1097: Add Hindi snowball stemmer.
• The FT._LIST command returns a list of all available indices. Note that this is a temporary command, as indicated by the _ in the name, so it's not documented. We're working on a SCAN-like command for databases with many indexes.
• The RediSearch version will appear in Redis as 20000, which is equivalent to 2.0.0 in semantic versioning. Since the version of a module in Redis is numeric, we cannot explicitly add an GA flag.
• RediSearch 2.x requires Redis 6.0 or later.

Behavior changes

Please familiarize yourself with these changes before upgrading to RediSearch 2.0:

• #1381: FT.SYNADD is removed; use FT.SYNUPDATE instead. FT.SYNUPDATE requires both
  and index name and a synonym group ID. This ID can be any ASCII string.
• #1437: Documents that expire during query execution time will not appear in the results (but might have been counted in the number of produced documents).
• #1221: Synonyms support for lower case. This can result in a different result set on FT.SEARCH when using synonyms.
• RediSearch will not index hashes whose fields do not match an existing index schema. You can see the number of hashes not indexed using FT.INFO - hash_indexing_failures . The requirement for adding support for partially indexing and blocking is captured here: #1455.
• Removed support for NOSAVE (for details see v1.6 docs).
• RDB loading will take longer due to the index not being persisted.
• Field names in the query syntax are now case-sensitive.
• Deprecated commands:
  • FT.DROP (replaced by FT.DROPINDEX, which by default keeps the documents)
  • FT.ADD (mapped to HSET for backward compatibility)
  • FT.DEL (mapped to DEL for backward compatibility)
  • FT.GET (mapped to HGETALL for backward compatibility)
  • FT.MGET (mapped to HGETALL for backward compatibility)
• Removed commands:
  • FT.ADDHASH (no longer makes sense)
  • FT.SYNADD (see #1381)
  • FT.OPTIMIZE (see v1.6 docs)

Scaling a single index over multiple shards with the open source Redis cluster API

Previously, a single RediSearch index, and its documents, had to reside on a single shard. This meant that dataset size and throughput was bound to what a single Redis process could handle.

Redis Enterprise offered the ability to distribute documents in a clustered database and aggregate the results at query time. This fan-out and aggregation is handled by a component called the “coordinator” that is now also available under the same [Redis Source Available License] for all Redis OSS users in it's own repository RSCoordinator.

Notes:

• The version inside Redis will be 20000 or 2.0.0 in semantic versioning. Since the version of a module in Redis is numeric, we could not add an GA flag.
• Requires Redis v6 or above.

1.6.14

This is a maintenance release for version 1.6.
Update urgency: Medium

• Minor features:

  • #1420 The hard limit of the number of results produced by FT.SEARCH is now configurable with MAXSEARCHRESULTS.

• Bug fixes:

  • #1313 Wrong error of unsupported phonetic field.
  • #1286 Possible crash on optional search.
  • #1449 Rare file descriptor leak on FORK GC.
  • #1469 Endless loop when reaching internal docid above uint32_max.

2.0.0-M5

This is the fifth milestone release for RedisSearch 2.0.

Highlights:

• Support upgrade from v1.x (more information).
• geodistance function for FT.AGGREGATE APPLY operation.

Details:

• Features:

  • #1435 NOINITIALINDEX flag on FT.CREATE to skip the initial scan of documents on index creation.
  • #1401 Support for reading legacy (created by RediSearch 1.x) RDB (more information)
  • #1445 Support for load event, this event allows to index documents when they are loaded from RDB and so to promise that the index is fully available when RDB loading is finalised (available from Redis 6.0.7 and above)
  • #1246 geodistance function for FT.AGGREGATE APPLY operation.

• Behaviour changes:

  • #1437 Documents that expired during query execution time will not appear in the results (but might have been counted in the number of produced documents).
  • #1221 [breaking change compared to 1.x] Synonyms support for lower case. This can result in a different result set on FT.SEARCH when using synonyms.

• Bug fixes:

  • #1448 Rare file descriptor leak on FORK GC.
  • #1461 Bad memory access when cursor continued to read an inverted index that was freed by Garbage Collection.
  • #1469 Endless loop when reaching internal docid above uint32_max.
  • #1483, #1425 Delete document using rm_call on drop index command so the document will be deleted from all indexes that indexed this document.

Notes:

• This is not the GA version of 2.0. The version inside Redis will be 19905 or 1.99.5 in semantic versioning. Since the version of a module in Redis is numeric, we could not add an M5 flag.
• Requires Redis v6 or above.

2.0.0-M3

This is the third milestone release for RedisSearch 2.0.

Highlights:

• #1424 Performance optimisation if no index exists
• #1428 clean empty terms on gc

Details:

• Minor changes:

  • #1413 return _ft.safeadd
  • #1411, #1427 Minor documentation updates.

• Bug fixes:

  • #1417 Evicted + fix expired
  • #1416 Fix issue #1414 remove hard limit on FT.SEARCH
  • #1407 Fix to llapi (the fix was already applied on 1.8 branch)

• Known issues in this milestone release:

  • Upgrade is not possible from 1.X versions or from 2.0.0-M1/2

Notes:

• This is not the GA version of 2.0. The version inside Redis will be 19903 or 1.99.3 in semantic versioning. Since the version of a module in Redis is numeric, we could not add an M3 flag.
• Requires Redis v6 or above.

2.0.0-M2

This is the second milestone release for RedisSearch 2.0

Highlights:

• #1394 - Expired documents (TTL) will be removed from the index.
• #1394 - Optimization for not reindexing documents where non-indexed fields were updated.

Details:

• Minor changes:

  • #1384 - Renamed FT.DELETE (introduced in 2.0-M1) to FT.DROPINDEX
  • #1385 - Add index definition to FT.INFO response.
  • #1381 - FT.SYNADD is removed, use FT.SYNUPDATE instead. Also the group id can now be a string (next to a number).

• Bugfixes

  • #1389 - Fix argument name to PAYLOAD_FIELD (was PAYLOAD).
  • #1403 - Fix issue where reindexing stats were reported wrong.

• Known issues in this milestone release:

  • Upgrade is not possible from 1.X versions or from 2.0.0-M1

Notes:

• This is not the GA version of 2.0. The version inside Redis will be 19902 or 1.99.2 in semantic versioning. Since the version of a module in Redis is numeric, we could not add an M2 flag.
• Requires Redis v6 or above

2.0.0-M1

This is the first milestone release for RedisSearch 2.0

Headlines:
This milestone re-architects the way indices are kept in sync with the data. Instead of having to write data through the index (using FT.ADD), RediSearch will now follow the data written in hashes and automatically index it. Read more about it in this blog.

Details:

• The index no longer resides within the key space.
• Indexes need be created with an IndexCondition (prefix matcher and filter)
• The deprecated FT.X commands were mapped to their Redis equivalents commands
• The inverted index itself is no longer saved to the RDB.

• Other notable Features:

  • #1097 Add hindi snowball stemmer.

• Bugfixes:

  • #1282 Crash when querying with some combinations of optional terms.
  • #1290 Crash when calling RediSearch_IndexAddDocument and NewAddDocumentCtx fails.
  • #1313,#1330 Unexpected: "field does not support phonetics".
  • #1255 Change ForkGC threshold to 100 to reduce empty GC cycles.

• Known issues in this milestone release:

  • Upgrade is not possible from 1.X versions

Notes:

• This is not the GA version of 2.0. The version inside Redis will be 19901 or 1.99.1 in semantic versioning. Since the version of a module in Redis is numeric, we could not add an M1 flag.
• Requires Redis v6 or above

1.6.13

This is a maintenance release for version 1.6.
Update urgency: Medium

Headlines:

• This release improves overall stability and provides fixes for issues found after the previous release.

Details:

• Bugfixes:
  • #1186 #1188 incorrect values for inverted_sz_mb and num_records in FT.INFO command.

1.4.28

This is a maintenance release for version 1.4.
Update urgency: Medium

Headlines:

• This release improves overall stability and provides fixes for issues found after the previous release.

Details:

• Bugfixes:
  • #1218 Potential crash when running without concurrency and using the cursor API.

1.6.12

This is a maintenance release for version 1.6.
Update urgency: Medium

Headlines:

• This release improves overall stability and provides fixes for issues found after the previous release.

Details:

• Features:
  • #1172 Added exists function that can be used on conditional updates (REPLACE PARTIAL) to check if a field exists in the document.
• Minor Enhancements:
  • #1172 Lazy evaluation of the right side of 'or'/'and' clauses in IF condition.
  • #1134 Remove hard limit on LIMIT when using FT.SEARCH.
• Bugfixes:
  • #1124 NOINDEX tag fields could not be updated on UPDATE PARTIAL with no indexed fields.
  • #1120 Release loop in II_GetCriteriaTester which released the same criteria tester multiple times.
  • #1161 Case where setting MAXDOCTABLESIZE had no effect.
  • #1169 FIRST_VALUE reducer crashed when value did not exist.
  • #1159 Infinite loop on NOT criteria tester.

1.4.27

This is a maintenance release for version 1.4.
Update urgency: Medium

Headlines:

• This release improves overall stability and provides fixes for issues found after the previous release.

Details:

• Features:
  • #1172 Added exists function that can be used on conditional updates (REPLACE PARTIAL) to check if a field exists in the document.
• Minor Enhancements:
  • #1172 Lazy evaluation of the right side of 'or'/'and' clauses in IF condition.
• Bugfixes:
  • #1110 Rare GC failure when accessing uninitialized variable.
  • #1131 Crash on highlighting a search query where the document no longer exists.

Note: we skipped 1.4.26 for technical reasons.