Skip to content

Commit

Permalink
[1.6] Cherry pick from master for 1.6.15 (#1761)
Browse files Browse the repository at this point in the history 
* Replace assert on limit of sortable fields (#1757)

(cherry picked from commit 5f7f105)

* Enforce 0 value for count reducer (#1745)

* Enforce 0 values for count

* fix rstest

* fix for centos

(cherry picked from commit 0ab57d6)

* feat: Add hindi snowball stemmer

(cherry picked from commit eac3594)

* Update snowball (#1708)

* Update snowball

* use rm allocator

(cherry picked from commit b3e86c2)

* [fix] parsing float scientific representation - Issue 1208 (#1225)

* fix ragel file

* test

* more precise

(cherry picked from commit 5eb145b)

* add STOPWORD to TAGLIST in query parser (#1668)

* add STOPWORD to TAGLIST

* use templete for parser.c

(cherry picked from commit fda9566)

* Revert "Update snowball (#1708)"

This reverts commit b2b49ea.

* Revert "feat: Add hindi snowball stemmer"

This reverts commit 4ee9026.

* return error for ft.alter w/o schema (#1754)

* return error for ft.alter w/o schema

* Add test

(cherry picked from commit 97f9d94)

* minTermPrefix & maxPrefixExpansions (#1774)

* minTermPrefix & maxPrefixExpansions

* fix rstest

(cherry picked from commit afc7e3c)

* Fix depth issue for score explain

(cherry picked from commit acdb772)

* Update Configuring.md

* Revert "Update Configuring.md"

This reverts commit 96e5a00.

* fix slop when no diff between offsets + test (#1605)

unclear why there

(cherry picked from commit 9de8286)

* use pointer instead of struct (#1591)

Co-authored-by: Rafi Einstein <rafi@redislabs.com>
(cherry picked from commit 56864b3)

* Sortby for non-sortable fields issue 1540 (#1574)

* sortby for non sortable fields issue 1540

(cherry picked from commit 1cd66d8)

* explicit error for EXPLAINSCORE w/o WITHSCORES (#1550)

* explicit error for EXPLAINSCORE w/o WITHSCORES

* per review

(cherry picked from commit 21b3d7f)

* avoid loop at DocTable_Set (#1498)

* avoid loop at DocTable_Set

(cherry picked from commit 06f52bf)

* Add test for LIMIT offset num (#1672)

* test

* updated test for coordinator

(cherry picked from commit 685929a)

* upgrade mkdocs (#1755)

(cherry picked from commit 84529a6)

* Fixed testCriteriaTesterDeactivated on macOS (#1660)

(cherry picked from commit 1e25d9e)

* fix tests

* Revert "Add test for LIMIT offset num (#1672)"

This reverts commit 8c6be73.

* Insert to suggest heap only if score larger than minimum - (#1689)

(cherry picked from commit ac0973d)

* Fix wrong arity for syndump (#1570)

(cherry picked from commit afbc834)

* skip or remove unrelated tests

* bump version to 1.6.15

* Fix stats report (#1517)

* Fix stats report

* remove unneeded ';'

* Update lastblkBytesCollected correctly

(cherry picked from commit 9c67a13)

* remove loom and rax - unused (#1521)

Co-authored-by: Rafi Einstein <rafi@redislabs.com>
(cherry picked from commit be7d847)

* Fixed build instructions (#1535)

(cherry picked from commit fef72dc)

* See RED-47209, fix crash when using WITHSORTKEYS without SORTBY (#1572)

(cherry picked from commit 11a6218)

* Prevent multiple sortby steps (#1614)

(cherry picked from commit 89095a6)

* added thread local support to rwlock (#1328) (#1643)

* added thread local support to rwlock

* fix msan failure

* fix msan failure

* fix compilation error

* fix compilation error

* fix msan failure

* review fixes

(cherry picked from commit 5977ef0)

Co-authored-by: Joe Paul <joeirimpan@gmail.com>
Co-authored-by: Emmanuel Keller <74923777+emmanuelkeller@users.noreply.github.com>
Co-authored-by: Emmanuel Keller <emmanuel.keller@redislabs.com>
Co-authored-by: Meir Shpilraien (Spielrein) <meir@redislabs.com>
Co-authored-by: Guy Korland <gkorland@gmail.com>
Co-authored-by: Rafi Einstein <rafi@redislabs.com>
  • Loading branch information
@ashtul @joeirimpan
@emmanuelkeller @MeirShpilraien @gkorland @rafie
7 people committed Jan 21, 2021
1 parent aeca816 commit c57f828
Show file tree
Hide file tree
Showing 49 changed files with 691 additions and 1,129 deletions.
28 changes: 17 additions & 11 deletions docs/Development.md
View file
@@ -1,6 +1,6 @@
# Developing RediSearch

Developing RediSearch involves setting up the development environment (which can be either Linux-based or macOS-based), building RediSearch, running tests and benchmarks, and debuugging both the RediSearch module and its tests.
Developing RediSearch involves setting up the development environment (which can be either Linux-based or macOS-based), building RediSearch, running tests and benchmarks, and debugging both the RediSearch module and its tests.

## Cloning the git repository
By invoking the following command, RediSearch module and its submodules are cloned:
Expand All @@ -10,42 +10,48 @@ git clone --recursive https://github.com/RediSearch/RediSearch.git
## Working in an isolated environment
There are several reasons to develop in an isolated environment, like keeping your workstation clean, and developing for a different Linux distribution.
The most general option for an isolated environment is a virtual machine (it's very easy to set one up using [Vagrant](https://www.vagrantup.com)).
Docker is even a more agile solution, as it offers an almost instant solution:
Docker is even a more agile, as it offers an almost instant solution:

```
search=$(docker run -d -it -v $PWD:/build debian:buster bash)
docker exec -it $search bash
```
Then, from whithin the container, ```cd /build``` and go on as usual.
Then, from within the container, ```cd /build``` and go on as usual.

In this mode, all installations remain in the scope of the Docker container.
Upon exiting the container, you can either re-invoke the container with the above ```docker exec``` or commit the state of the container to an image and re-invoke it on a later stage:
Upon exiting the container, you can either re-invoke it with the above ```docker exec``` or commit the state of the container to an image and re-invoke it on a later stage:

```
docker commit $search redisearch1
docker stop $search
search=$(docker run -d -it -v $PWD:/build rediseatch1 bash)
docker exec -it $search bash
```

You can replace `debian:buster` with your OS of choice, with the host OS being the best choice (so you can run the RediSearch binary on your host once it is built).

## Installing prerequisites
To build and test RediSearch one needs to install serveral packages, depending on the underlying OS. Currently, we support the Ubuntu/Debian, CentOS, Fedora, and macOS.

To build and test RediSearch one needs to install several packages, depending on the underlying OS. Currently, we support the Ubuntu/Debian, CentOS, Fedora, and macOS.

If you have ```gnu make``` installed, you can execute
```
cd RediSearch
make setup
sudo make setup
```
Alternatively, just invoke the following:
```
cd RediSearch
./deps/readies/bin/getpy2
./system-setup.py
sudo ./deps/readies/bin/getpy2
sudo ./system-setup.py
```
Note that ```system-setup.py``` **will install various packages on your system** using the native package manager and pip. This requires root permissions (i.e. sudo) on Linux.

If you prefer to avoid that, you can:

* Review system-setup.py and install packages manually,
* Use an isolated environment like explained above,
* Utilize a Python virtual environment, as Python installations known to be sensitive when not used in isolation.
* Use a Python virtual environment, as Python installations are known to be sensitive when not used in isolation: `python2 -m virtualenv venv; . ./venv/bin/activate`

Next, execute the following, to complete dependency acquisition:
```
Expand All @@ -55,9 +61,9 @@ make fetch
## Installing Redis
As a rule of thumb, you're better off running the latest Redis version.

If your OS has a Redis 5.x package, you can install it using the OS package manager.
If your OS has a Redis 6.x package, you can install it using the OS package manager.

Otherwise, you can invoke ```./deps/readies/getredis5```.
Otherwise, you can invoke ```sudo ./deps/readies/bin/getredis```.

## Getting help
```make help``` provides a quick summary of the development features.
Expand Down
15 changes: 8 additions & 7 deletions docs/Quick_Start.md
View file
Expand Up @@ -3,7 +3,7 @@

## Redis Cloud

RediSearch is available on all Redis Cloud managed services. Redis Cloud Essentials offers a completely free managed databbases up to 30MB.
RediSearch is available on all Redis Cloud managed services. Redis Cloud Essentials offers a completely free managed databases up to 30MB.

[Get started here](https://redislabs.com/try-free/)

Expand All @@ -19,27 +19,28 @@ First download the pre-compiled version from [RedisLabs download center](https:/

Next, run Redis with RediSearch:

```
```sh
$ redis-server --loadmodule /path/to/module/src/redisearch.so
```

## Building and running from source

First, clone the git repo:
First, clone the git repo (make sure not to omit the `--recursive` option, to properly clone submodules):

```
```sh
git clone --recursive https://github.com/RediSearch/RediSearch.git
```

Next, build:
Next, install dependencies and build:

```
```sh
sudo make setup
make build
```

Finally, run Redis with RediSearch:

```
```sh
make run
```

Expand Down
6 changes: 3 additions & 3 deletions docs/requirements.txt
View file
@@ -1,3 +1,3 @@
mkdocs==1.0.4
mkdocs-material==4.6.3
-e git+https://github.com/RedisLabs/mkdocs-versions-menu.git#egg=mkdocs-versions-menu
mkdocs==1.1.2
mkdocs-material==6.2.2
-e git+https://github.com/RedisLabs/mkdocs-versions-menu.git#egg=mkdocs-versions-menu
2 changes: 1 addition & 1 deletion src/aggregate/aggregate_exec.c
View file
Expand Up @@ -14,7 +14,7 @@ static void runCursor(RedisModuleCtx *outputCtx, Cursor *cursor, size_t num);
* RLookup registry. Returns NULL if there is no sorting key
*/
static const RSValue *getSortKey(AREQ *req, const SearchResult *r, const PLN_ArrangeStep *astp) {
if (!astp) {
if (!astp || !(astp->sortkeysLK)) {
return NULL;
}
const RLookupKey *kk = astp->sortkeysLK[0];
Expand Down
11 changes: 11 additions & 0 deletions src/aggregate/aggregate_request.c
View file
Expand Up @@ -173,6 +173,12 @@ static int handleCommonArgs(AREQ *req, ArgsCursor *ac, QueryError *status, int a
}

static int parseSortby(PLN_ArrangeStep *arng, ArgsCursor *ac, QueryError *status, int isLegacy) {
// Prevent multiple SORTBY steps
if (arng->sortKeys != NULL) {
QERR_MKBADARGS_FMT(status, "Multiple SORTBY steps are not allowed. Sort multiple fields in a single step");
return REDISMODULE_ERR;
}

// Assume argument is at 'SORTBY'
ArgsCursor subArgs = {0};
int rv;
Expand Down Expand Up @@ -361,6 +367,11 @@ static int parseQueryArgs(ArgsCursor *ac, AREQ *req, RSSearchOptions *searchOpts
}
}

if ((req->reqflags & QEXEC_F_SEND_SCOREEXPLAIN) && !(req->reqflags & QEXEC_F_SEND_SCORES)) {
QERR_MKBADARGS_FMT(status, "EXPLAINSCORE must be accompanied with WITHSCORES");
return REDISMODULE_ERR;
}

searchOpts->inkeys = (const char **)inKeys.objs;
searchOpts->ninkeys = inKeys.argc;
searchOpts->legacy.infields = (const char **)inFields.objs;
Expand Down
143 changes: 73 additions & 70 deletions src/aggregate/expr/lexer.c
View file
Expand Up @@ -39,119 +39,122 @@ static const char _expr_actions[] = {

static const unsigned char _expr_key_offsets[] = {
0, 0, 3, 5, 8, 10, 14, 18,
19, 20, 22, 32, 73, 74, 77, 78,
82, 85, 86, 89, 92, 94, 95, 96,
97, 107, 114, 122, 129
19, 20, 22, 25, 27, 37, 78, 79,
82, 83, 87, 90, 91, 94, 99, 103,
105, 106, 107, 108, 118, 125, 133, 140
};

static const char _expr_trans_keys[] = {
10, 34, 92, 34, 92, 10, 39, 92,
39, 92, 10, 39, 92, 110, 10, 39,
92, 102, 110, 102, 48, 57, 9, 13,
32, 47, 58, 64, 91, 96, 123, 126,
32, 33, 34, 37, 38, 39, 40, 41,
42, 43, 44, 45, 47, 60, 61, 62,
64, 94, 105, 124, 127, 0, 8, 9,
13, 14, 31, 35, 46, 48, 57, 58,
63, 65, 90, 91, 96, 97, 122, 123,
126, 61, 10, 34, 92, 38, 10, 39,
92, 105, 10, 39, 92, 105, 105, 48,
57, 46, 48, 57, 48, 57, 61, 61,
61, 92, 96, 0, 47, 58, 64, 91,
94, 123, 127, 95, 48, 57, 65, 90,
97, 122, 95, 110, 48, 57, 65, 90,
97, 122, 95, 48, 57, 65, 90, 97,
122, 124, 0
92, 102, 110, 102, 48, 57, 45, 48,
57, 48, 57, 9, 13, 32, 47, 58,
64, 91, 96, 123, 126, 32, 33, 34,
37, 38, 39, 40, 41, 42, 43, 44,
45, 47, 60, 61, 62, 64, 94, 105,
124, 127, 0, 8, 9, 13, 14, 31,
35, 46, 48, 57, 58, 63, 65, 90,
91, 96, 97, 122, 123, 126, 61, 10,
34, 92, 38, 10, 39, 92, 105, 10,
39, 92, 105, 105, 48, 57, 46, 69,
101, 48, 57, 69, 101, 48, 57, 48,
57, 61, 61, 61, 92, 96, 0, 47,
58, 64, 91, 94, 123, 127, 95, 48,
57, 65, 90, 97, 122, 95, 110, 48,
57, 65, 90, 97, 122, 95, 48, 57,
65, 90, 97, 122, 124, 0
};

static const char _expr_single_lengths[] = {
0, 3, 2, 3, 2, 4, 4, 1,
1, 0, 0, 21, 1, 3, 1, 4,
3, 1, 1, 1, 0, 1, 1, 1,
2, 1, 2, 1, 1
1, 0, 1, 0, 0, 21, 1, 3,
1, 4, 3, 1, 1, 3, 2, 0,
1, 1, 1, 2, 1, 2, 1, 1
};

static const char _expr_range_lengths[] = {
0, 0, 0, 0, 0, 0, 0, 0,
0, 1, 5, 10, 0, 0, 0, 0,
0, 0, 1, 1, 1, 0, 0, 0,
4, 3, 3, 3, 0
0, 1, 1, 1, 5, 10, 0, 0,
0, 0, 0, 0, 1, 1, 1, 1,
0, 0, 0, 4, 3, 3, 3, 0
};

static const unsigned char _expr_index_offsets[] = {
0, 0, 4, 7, 11, 14, 19, 24,
26, 28, 30, 36, 68, 70, 74, 76,
81, 85, 87, 90, 93, 95, 97, 99,
101, 108, 113, 119, 124
26, 28, 30, 33, 35, 41, 73, 75,
79, 81, 86, 90, 92, 95, 100, 104,
106, 108, 110, 112, 119, 124, 130, 135
};

static const char _expr_indicies[] = {
0, 2, 3, 1, 4, 3, 1, 0,
2, 6, 5, 7, 6, 5, 8, 2,
6, 9, 5, 8, 2, 6, 10, 5,
11, 0, 12, 0, 14, 13, 15, 15,
15, 15, 15, 0, 18, 19, 20, 22,
23, 24, 25, 26, 27, 28, 29, 30,
31, 33, 34, 35, 36, 38, 39, 40,
16, 16, 18, 16, 21, 32, 21, 37,
21, 37, 21, 17, 42, 41, 0, 2,
3, 1, 44, 43, 43, 2, 6, 45,
5, 0, 2, 6, 5, 47, 46, 47,
32, 48, 50, 32, 49, 14, 49, 52,
51, 53, 43, 55, 54, 56, 0, 0,
0, 0, 0, 15, 37, 37, 37, 37,
57, 37, 58, 37, 37, 37, 57, 37,
37, 37, 37, 57, 59, 43, 0
11, 0, 12, 0, 14, 13, 15, 16,
13, 16, 13, 17, 17, 17, 17, 17,
0, 20, 21, 22, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 35, 36,
37, 38, 40, 41, 42, 18, 18, 20,
18, 23, 34, 23, 39, 23, 39, 23,
19, 44, 43, 0, 2, 3, 1, 46,
45, 45, 2, 6, 47, 5, 0, 2,
6, 5, 49, 48, 49, 34, 50, 52,
53, 53, 34, 51, 53, 53, 14, 51,
16, 51, 55, 54, 56, 45, 58, 57,
59, 0, 0, 0, 0, 0, 17, 39,
39, 39, 39, 60, 39, 61, 39, 39,
39, 60, 39, 39, 39, 39, 60, 62,
45, 0
};

static const char _expr_trans_targs[] = {
11, 1, 11, 2, 13, 3, 4, 16,
11, 6, 16, 8, 11, 11, 20, 24,
11, 0, 11, 12, 13, 11, 11, 14,
15, 11, 11, 11, 17, 11, 18, 11,
19, 21, 22, 23, 24, 25, 11, 26,
28, 11, 11, 11, 11, 5, 11, 7,
11, 11, 9, 11, 11, 11, 11, 11,
10, 11, 27, 11
13, 1, 13, 2, 15, 3, 4, 18,
13, 6, 18, 8, 13, 13, 22, 11,
23, 27, 13, 0, 13, 14, 15, 13,
13, 16, 17, 13, 13, 13, 19, 13,
20, 13, 21, 24, 25, 26, 27, 28,
13, 29, 31, 13, 13, 13, 13, 5,
13, 7, 13, 13, 9, 10, 13, 13,
13, 13, 13, 12, 13, 30, 13
};

static const char _expr_trans_actions[] = {
63, 0, 35, 0, 77, 0, 0, 77,
61, 0, 68, 0, 7, 59, 0, 65,
41, 0, 37, 0, 80, 39, 25, 0,
80, 9, 11, 31, 74, 33, 71, 29,
5, 0, 0, 0, 80, 0, 27, 0,
0, 53, 19, 57, 21, 0, 55, 0,
47, 43, 0, 49, 13, 17, 51, 15,
0, 45, 0, 23
61, 0, 68, 0, 7, 59, 5, 0,
0, 65, 41, 0, 37, 0, 80, 39,
25, 0, 80, 9, 11, 31, 74, 33,
71, 29, 5, 0, 0, 0, 80, 0,
27, 0, 0, 53, 19, 57, 21, 0,
55, 0, 47, 43, 0, 0, 49, 13,
17, 51, 15, 0, 45, 0, 23
};

static const char _expr_to_state_actions[] = {
0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 1, 0, 0, 0, 0,
0, 0, 0, 0, 0, 1, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0
0, 0, 0, 0, 0, 0, 0, 0
};

static const char _expr_from_state_actions[] = {
0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 3, 0, 0, 0, 0,
0, 0, 0, 0, 0, 3, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0
0, 0, 0, 0, 0, 0, 0, 0
};

static const unsigned char _expr_eof_trans[] = {
0, 1, 1, 1, 1, 9, 9, 1,
1, 14, 1, 0, 42, 1, 44, 44,
1, 47, 49, 50, 50, 52, 44, 55,
1, 58, 58, 58, 44
1, 14, 14, 14, 1, 0, 44, 1,
46, 46, 1, 49, 51, 52, 52, 52,
55, 46, 58, 1, 61, 61, 61, 46
};

static const int expr_start = 11;
static const int expr_first_final = 11;
static const int expr_start = 13;
static const int expr_first_final = 13;
static const int expr_error = 0;

static const int expr_en_main = 11;
static const int expr_en_main = 13;


/* #line 250 "lexer.rl" */
Expand All @@ -173,7 +176,7 @@ RSExpr *RSExpr_Parse(const char *expr, size_t len, char **err) {
const char* ts = ctx.raw;
const char* te = ctx.raw + ctx.len;

/* #line 177 "lexer.c" */
/* #line 180 "lexer.c" */
{
cs = expr_start;
ts = 0;
Expand All @@ -190,7 +193,7 @@ RSExpr *RSExpr_Parse(const char *expr, size_t len, char **err) {
const char* eof = pe;


/* #line 194 "lexer.c" */
/* #line 197 "lexer.c" */
{
int _klen;
unsigned int _trans;
Expand All @@ -211,7 +214,7 @@ RSExpr *RSExpr_Parse(const char *expr, size_t len, char **err) {
/* #line 1 "NONE" */
{ts = p;}
break;
/* #line 215 "lexer.c" */
/* #line 218 "lexer.c" */
}
}

Expand Down Expand Up @@ -641,7 +644,7 @@ RSExpr *RSExpr_Parse(const char *expr, size_t len, char **err) {
}
}
break;
/* #line 645 "lexer.c" */
/* #line 648 "lexer.c" */
}
}

Expand All @@ -654,7 +657,7 @@ RSExpr *RSExpr_Parse(const char *expr, size_t len, char **err) {
/* #line 1 "NONE" */
{ts = 0;}
break;
/* #line 658 "lexer.c" */
/* #line 661 "lexer.c" */
}
}

Expand Down

0 comments on commit c57f828

Please sign in to comment.