Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Change pagination system to use search_after instead of from #974

Closed
richard-julien opened this issue Jan 1, 2021 · 0 comments
Closed

Change pagination system to use search_after instead of from #974

richard-julien opened this issue Jan 1, 2021 · 0 comments
Assignees
@richard-julien
Labels
feature use for describing a new feature to develop solved use to identify issue that has been solved (must be linked to the solving PR)
Milestone
Release 4.1.0

Comments

@richard-julien
Copy link
Member

Description

OpenCTI is now a platform that can handle billion of information. And so the current pagination approach with from/limit is no longer working for huge list of information

Environment

All

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Inject more than 100 000 malware
  2. Paginate on it

Expected Output

Pagination from 0 to 100 000 without any error/warning

Actual Output

Elastic complains about max result window

Additional information

Move to search_after that use pages instead of in memory pagination
@richard-julien richard-julien self-assigned this Jan 1, 2021
@richard-julien richard-julien added the feature use for describing a new feature to develop label Jan 1, 2021
@richard-julien richard-julien added this to the Release 4.1.0 milestone Jan 1, 2021
@richard-julien richard-julien mentioned this issue Jan 26, 2021
Merged
SamuelHassine pushed a commit that referenced this issue Jan 26, 2021
@richard-julien
[api] Improve delete and merge for huge volume / pagination search af…
ed2fdcf 
…ter (#974, #1002)

* [api] Refactor loading and pagination
* [api] Fix tests using old pagination offset
* [api] Default ordering by created_at
* [api] Try usage of standard_id as default sorting
* [api] Refactor to get all related relations without recursive func
* [api] Improve merge and deletion for large volume
* [api] Fix size of minio file
@SamuelHassine SamuelHassine added the solved use to identify issue that has been solved (must be linked to the solving PR) label Jan 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@richard-julien richard-julien

Labels
feature use for describing a new feature to develop solved use to identify issue that has been solved (must be linked to the solving PR)
Projects
None yet
Milestone
Release 4.1.0
Development

No branches or pull requests
2 participants
@richard-julien @SamuelHassine