Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[api] Improve merging memory footprint #2366

Closed
richard-julien opened this issue Sep 18, 2022 · 0 comments
Closed

[api] Improve merging memory footprint #2366

richard-julien opened this issue Sep 18, 2022 · 0 comments
Assignees
@richard-julien
Labels
feature use for describing a new feature to develop solved use to identify issue that has been solved (must be linked to the solving PR)
Milestone
Release 5.3.13

Comments

@richard-julien
Copy link
Member

Use case

Current footprint for merging entities is pretty big.
Currently merging Emotet with 1 million relationships take more than 10Go of memory. This lead to oom for a lot of users.

Result

Improvement able to do the merge with 1Go footprint.
Time to process the million elements and do the merge
Start: "timestamp":"2022-09-18T20:34:50.410Z",
Stop: "timestamp":"2022-09-18T20:41:51.471Z"
=> 7 minutes
@richard-julien richard-julien added the feature use for describing a new feature to develop label Sep 18, 2022
@richard-julien richard-julien added this to the Release 5.3.13 milestone Sep 18, 2022
@richard-julien richard-julien self-assigned this Sep 18, 2022
richard-julien added a commit that referenced this issue Sep 18, 2022
@richard-julien
[api] Improve merging memory footprint (#2366)
de77e0d
richard-julien added a commit that referenced this issue Sep 18, 2022
@richard-julien
[api] Use the filtered participants list for locking (#2366)
c27aa7e
richard-julien added a commit that referenced this issue Sep 18, 2022
@richard-julien
[api] Improve merging memory footprint (#2366)
92a1336
richard-julien added a commit that referenced this issue Sep 18, 2022
@richard-julien
[api] Use the filtered participants list for locking (#2366)
d8835da
richard-julien added a commit that referenced this issue Sep 18, 2022
@richard-julien
[api] Adapt test to remove fully resolve function usage (#2366)
7a1e572
richard-julien added a commit that referenced this issue Sep 18, 2022
@richard-julien
[api] Improve merging memory footprint (#2366)
56ee33a
@SamuelHassine SamuelHassine added the solved use to identify issue that has been solved (must be linked to the solving PR) label Sep 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@richard-julien richard-julien

Labels
feature use for describing a new feature to develop solved use to identify issue that has been solved (must be linked to the solving PR)
Projects
None yet
Milestone
Release 5.3.13
Development

No branches or pull requests
2 participants
@richard-julien @SamuelHassine