Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .ci/doc/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ version: '3'

services:
kuzzle:
image: kuzzleio/kuzzle
image: kuzzleio/kuzzle:1
ports:
- '7512:7512'
cap_add:
Expand All @@ -17,7 +17,7 @@ services:
- NODE_ENV=production

redis:
image: redis:3.2
image: redis:5

elasticsearch:
image: kuzzleio/elasticsearch:5.6.10
Expand Down
5 changes: 2 additions & 3 deletions .ci/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ version: '3'

services:
kuzzle:
image: kuzzleio/kuzzle
image: kuzzleio/kuzzle:1
ports:
- "7512:7512"
cap_add:
Expand All @@ -17,7 +17,7 @@ services:
- NODE_ENV=production

redis:
image: redis:3.2
image: redis:5

elasticsearch:
image: kuzzleio/elasticsearch:5.6.10
Expand All @@ -29,4 +29,3 @@ services:

volumes:
snippets:

8 changes: 7 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,10 @@ examples/node_modules/
coverage/

doc/framework
dead_links.json
dead_links.json

# Cypress debug
doc/6/getting-started/.react/cypress/screenshots
doc/6/getting-started/.react/cypress/videos
doc/6/getting-started/.vuejs/cypress/screenshots
doc/6/getting-started/.vuejs/cypress/videos
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,14 @@ Our teams will be able to meet your needs in terms of expertise and multi-techno

## Usage

## Compatibility matrice

| Kuzzle Version | SDK Version |
| -------------- | -------------- |
| 1.x.x | 5.x.x |
| 1.x.x | 6.x.x |
| 2.x.x | 7.x.x |

### Installation

This SDK can be used either in NodeJS or in a browser.
Expand Down Expand Up @@ -140,4 +148,3 @@ try {
console.error(error);
}
```

15 changes: 0 additions & 15 deletions doc/6/core-classes/search-result/introduction/index.md

This file was deleted.

49 changes: 25 additions & 24 deletions doc/6/core-classes/search-result/next/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,58 +6,59 @@ description: SearchResult next method
order: 200
---

# SearchResult
# next

Returns a new `SearchResult` object which contain the subsequent results of the search.
Advances through the search results and returns the next page of items.

## Arguments

```js
next();
```

## Behaviour of the next method
## Resolve

In order to be able to compute the next search page, some initial conditions must be met.
Resolves to a `SearchResult` object, or to `null` if no more pages are available.

Depending on the arguments given to the initial search, thhe `next` method will pick one of the following policies, by decreasing order of priority (i.e. a search including a `scroll`, `sort` and `size` will use the `scroll` policy).
## Throw

If no policy is applicable, the `next` method will throw an exception.
This method throws an exception if:

:::info
When processing a large number of documents (i.e. more than 1000), it is advised to use a scroll cursor.
- No pagination strategy can be applied (see below)
- If invoking it would lead to more than 10 000 items being retrieved with the `from/size` strategy

It is also the only method guaranteeing that all matching documents will be retrieved and no duplicates will be included.
:::
## Pagination strategies

## Usage with scroll
Depending on the arguments given to the initial search, the `next` method will pick one of the following strategies, by decreasing order of priority.

**This is the preferred way to get some paginated results**.
### Strategy: scroll cursor

If the original search is given a `scroll` parameter, the `next` method will use a cursor to paginate results.
If the original search query is given a `scroll` parameter, the `next` method uses a cursor to paginate results.

The results that are returned from a scroll request reflect the state of the index at the time the initial `search` request was performed, like a snapshot in time.
The results from a scroll request are frozen, and reflect the state of the index at the time the initial `search` request.
For that reason, this method is guaranteed to return consistent results, even if documents are updated or deleted in the database between two pages retrieval.

As such, even if some documents are added or deleted from the database between two calls to `next`, the result is garanteed to include all items matching the query at the time the initial `search` was sent and to not get any duplicate between two search pages.
This is the most consistent way to paginate results, however, this comes at a higher computing cost for the server.

<<< ./snippets/scroll.js

## Usage with sort / size
### Strategy: sort / size

If the initial search is given some `sort` and `size` parameters, the `next` method will retrieve the next items matching the sort.
If the initial search contains `sort` and `size` parameters, the `next` method retrieves the next page of results following the sort order, the last item of the current page acting as a live cursor.

To avoid too many duplicates, it is advised to provide a sort combination that will always identify one item only. The recommended way is to use the field `_uid` which is certain to contain one unique value for each document.

Because this method does not freeze the research between two calls, if some updates are applied to the database between two calls, it is still possible to miss some documents and/or to get some duplicates between search pages.
Because this method does not freeze the search results between two calls, there can be missing or duplicated documents between two result pages.

## Usage with from / size
This method efficiently mitigates the costs of scroll searches, but returns less consistent results: it's a middle ground, ideal for real-time search requests.

If the initial search is given some `from` and `size` parameters, the `next` method will increment the `from` parameter to retrieved the next results.
### Strategy: from / size

Because this method does not freeze the research between two calls, if some updates are applied to the database between two calls, it is possible to miss some documents and/or to get some duplicates between search pages.
If the initial search contains `from` and `size` parameters, the `next` method retrieves the next page of result by incrementing the `from` offset.

:::info
It is not possible to retrieve more than 10000 items using this method. Above that limit, any call to `next` will throw an Exception.
:::
Because this method does not freeze the search results between two calls, there can be missing or duplicated documents between two result pages.

It's the fastest pagination method available, but also the less consistent, and it is not possible to retrieve more than 10000 items using it.
Above that limit, any call to `next` throws an Exception.

<<< ./snippets/fromsize.js
84 changes: 29 additions & 55 deletions doc/6/core-classes/search-result/next/snippets/fromsize.js
Original file line number Diff line number Diff line change
@@ -1,69 +1,43 @@
const suv = { category: 'suv' };

try {
const requests = [];
const documents = [];

for (let i = 0; i < 10; i++) {
requests.push(kuzzle.document.create('nyc-open-data', 'yellow-taxi', suv));
for (let i = 0; i < 100; i++) {
documents.push({ _id: `suv_no${i}`, body: { category: 'suv' } });
}

await Promise.all(requests);

// Waits documents to be indexed
await kuzzle.index.refresh('nyc-open-data');

const searchOptions = {
from: 1,
size: 5
};
await kuzzle.document.mCreate('nyc-open-data', 'yellow-taxi', documents, {
refresh: 'wait_for'
});

const results = await kuzzle.document.search(
let results = await kuzzle.document.search(
'nyc-open-data',
'yellow-taxi',
{
query: {
match: {
category: 'suv'
}
}
},
searchOptions
);
{ query: { match: { category: 'suv' } } },
{ from: 1, size: 5 });

const nextResults = await results.next();
// Fetch the matched items by advancing through the result pages
const matched = [];

while (results) {
matched.push(...results.hits);
results = await results.next();
}

console.log(`Successfully retrieved ${nextResults.fetched} documents`);
console.log(nextResults);
console.log(matched[0]);
/*
{
"aggregations": undefined,
"hits": [
{
"_index": "nyc-open-data",
"_type": "yellow-taxi",
"_id": "AWgi6A1POQUM6ucJ3q06",
"_score": 0.046520017,
"_source": {
"category": "suv",
"_kuzzle_info": {
"author": "-1",
"createdAt": 1546773859655,
"updatedAt": null,
"updater": null,
"active": true,
"deletedAt": null
}
}
},
...
]
},
"total": 10,
"fetched": 10,
"scroll_id": undefined,
"from": 1,
"size": 5
{ _id: 'suv_no1',
_score: 0.03390155,
_source:
{ _kuzzle_info:
{ active: true,
author: '-1',
updater: null,
updatedAt: null,
deletedAt: null,
createdAt: 1570093133057 },
category: 'suv' } }
*/
console.log(`Successfully retrieved ${matched.length} documents`);
} catch (error) {
console.error(error.message);
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@ hooks:
after: |
curl -XDELETE kuzzle:7512/nyc-open-data
template: default
expected: Successfully retrieved 10 documents
expected: Successfully retrieved 100 documents
sdk: js
version: 6
78 changes: 24 additions & 54 deletions doc/6/core-classes/search-result/next/snippets/scroll.js
Original file line number Diff line number Diff line change
@@ -1,74 +1,44 @@
const suv = { category: 'suv' };

try {
const requests = [];
const documents = [];

for (let i = 0; i < 10; i++) {
requests.push(kuzzle.document.create('nyc-open-data', 'yellow-taxi', suv));
for (let i = 0; i < 100; i++) {
documents.push({ _id: `suv_no${i}`, body: { category: 'suv' } });
}

await Promise.all(requests);

// Waits documents to be indexed
await kuzzle.index.refresh('nyc-open-data');

const searchOptions = {
scroll: '1m',
size: 5
};
await kuzzle.document.mCreate('nyc-open-data', 'yellow-taxi', documents, {
refresh: 'wait_for'
});

let results = await kuzzle.document.search(
'nyc-open-data',
'yellow-taxi',
{
query: {
match: {
category: 'suv'
}
}
},
searchOptions
);
{ query: { match: { category: 'suv' } } },
{ scroll: '1m', size: 10 });

// Fetch the next fetch results and push them into the 'documents' array
const documents = [];
// Fetch the matched items by advancing through the result pages
const matched = [];

while (results) {
results.hits.forEach(hit => documents.push(hit._source));
matched.push(...results.hits);
results = await results.next();
}

console.log(results);
console.log(matched[0]);
/*
{
"aggregations": undefined,
"hits": [
{
"_index": "nyc-open-data",
"_type": "yellow-taxi",
"_id": "AWgi6A1POQUM6ucJ3q06",
"_score": 0.046520017,
"_source": {
"category": "suv",
"_kuzzle_info": {
"author": "-1",
"createdAt": 1546773859655,
"updatedAt": null,
"updater": null,
"active": true,
"deletedAt": null
}
}
},
...
]
},
"total": 10,
"fetched": 10,
"scroll_id": "DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAAAqFnQ5NU9sZWFaUTRhd2VHNU5KZzVEQ"
{ _id: 'suv_no1',
_score: 0.03390155,
_source:
{ _kuzzle_info:
{ active: true,
author: '-1',
updater: null,
updatedAt: null,
deletedAt: null,
createdAt: 1570093133057 },
category: 'suv' } }
*/

console.log(`Successfully retrieved ${documents.length} documents`);
console.log(`Successfully retrieved ${matched.length} documents`);
} catch (error) {
console.error(error.message);
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@ hooks:
after: |
curl -XDELETE kuzzle:7512/nyc-open-data
template: default
expected: Successfully retrieved 10 documents
expected: Successfully retrieved 100 documents
sdk: js
version: 6
Loading