Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[backend] Improvement to limit engine memory usage #5642

Merged
merged 10 commits into from
Jan 26, 2024
Merged

[backend] Improvement to limit engine memory usage #5642

merged 10 commits into from
Jan 26, 2024

Conversation

richard-julien
Copy link
Member

@richard-julien richard-julien commented Jan 22, 2024 •
edited

The idea is to limit the memory footprint of the engine query.
First round before oob/batch_list_through_refactor

  • Limit the pagination size to 500 elements
  • Limit the window results of id resolution to minimum
  • Infer the indices to read from the types if possible
  • Remove deprecated ignore_throttled option

@richard-julien richard-julien self-assigned this Jan 22, 2024
@richard-julien richard-julien added the filigran team use to identify PR from the Filigran team label Jan 22, 2024
@codecov Codecov
Copy link

codecov bot commented Jan 22, 2024 •
edited

Codecov Report

Attention: 60 lines in your changes are missing coverage. Please review.

Comparison is base (f12bf41) 66.09% compared to head (2e5aa8d) 66.11%.

❗ Current head 2e5aa8d differs from pull request most recent head 1261e1e. Consider uploading reports for the commit 1261e1e to get more accurate results

Files Patch % Lines
...ti-platform/opencti-graphql/src/database/engine.js 85.89% 22 Missing ⚠️
...rm/opencti-graphql/src/resolvers/stixCoreObject.js 0.00% 7 Missing ⚠️
...pencti-graphql/src/domain/backgroundTask-common.js 0.00% 6 Missing ⚠️
...encti-graphql/src/resolvers/stixCyberObservable.js 0.00% 6 Missing ⚠️
...rm/opencti-graphql/src/manager/fileIndexManager.ts 0.00% 5 Missing ⚠️
...latform/opencti-graphql/src/database/middleware.js 89.18% 4 Missing ⚠️
...opencti-graphql/src/resolvers/externalReference.js 0.00% 3 Missing ⚠️
...ncti-graphql/src/resolvers/stixCoreRelationship.js 50.00% 2 Missing ⚠️
.../opencti-graphql/src/resolvers/stixDomainObject.js 0.00% 2 Missing ⚠️
...orm/opencti-graphql/src/domain/stixRelationship.js 50.00% 1 Missing ⚠️
... and 2 more
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #5642      +/-   ##
==========================================
+ Coverage   66.09%   66.11%   +0.02%     
==========================================
  Files         513      513              
  Lines       60817    60752      -65     
  Branches     4414     4452      +38     
==========================================
- Hits        40196    40167      -29     
+ Misses      20621    20585      -36

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

SouadHadjiat
SouadHadjiat reviewed Jan 23, 2024
View reviewed changes
opencti-platform/opencti-graphql/src/database/engine.js Outdated
@@ -1006,7 +1012,7 @@ export const RUNTIME_ATTRIBUTES = {
// eslint-disable-next-line no-use-before-define
const users = await elPaginate(context, user, READ_INDEX_INTERNAL_OBJECTS, {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we load users from cache too here ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

SouadHadjiat
SouadHadjiat reviewed Jan 23, 2024
View reviewed changes
opencti-platform/opencti-graphql/src/database/engine.js Outdated
if (types && types.length > 0) {
const typesShould = types.map((typeShould) => (
[
{ match_phrase: { 'entity_type.keyword': typeShould } },
{ match_phrase: { 'parent_types.keyword': typeShould } }
{ term: { 'entity_type.keyword': typeShould } },
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here we could also do a terms query with the list of types instead of a list of should with one term.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

@richard-julien richard-julien force-pushed the oob/memory_improvment branch 2 times, most recently from bd9d238 to 2e5aa8d Compare January 25, 2024 21:38
@richard-julien richard-julien linked an issue Jan 25, 2024 that may be closed by this pull request
[backend] Improvement to limit engine memory usage #5701
Closed
@SouadHadjiat
Copy link
Member

I tested locally and didn't find any bug related to this PR, but I'm not sure I tested all the cases.
It seems already more performant even with my small db.

@SamuelHassine SamuelHassine merged commit d3df58b into master Jan 26, 2024
4 of 6 checks passed
@SamuelHassine SamuelHassine deleted the oob/memory_improvment branch January 26, 2024 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SouadHadjiat SouadHadjiat SouadHadjiat approved these changes

Assignees

@richard-julien richard-julien

Labels
filigran team use to identify PR from the Filigran team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[backend] Improvement to limit engine memory usage
3 participants
@richard-julien @SouadHadjiat @SamuelHassine