Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve expansions on disk #1335

Merged
merged 49 commits into from Oct 11, 2023
Merged

Improve expansions on disk #1335

merged 49 commits into from Oct 11, 2023

Conversation

as51340
Copy link
Contributor

@as51340 as51340 commented Oct 6, 2023

Docs:

memgraph/documentation#252

NOTE:

Some e2e disk tests are disabled since a lot of things changes in this PR. All commented tests aren't that important for our workloads (query modules interacting with writing to DB, mock API, triggers...). This PR makes it possible to run 1-hop expansions with 30 Q/s. Currently, the performance of 1-hop expansion is 0.5 Q/s. Next PRs will deal with further improvements.

Tasks:

Currently, serializing edges across multiple column families should be solved. That means creating edges and modifying edges should work.

Create out_edges and in_edges column families ✅
Serializing edges ✅
Remove prefetching edges ✅
Modify edge creation ✅
Modify edge deletion ✅
Delete vertex ✅
Detach delete vertex ✅
Refactored indices ✅
Made thinner vertex and edge accessors by sending pointer to the storage. ✅
How to call OutEdges? ✅
How to call InEdges? ✅
fix all unit tests ✅
revert changes from some unit tests now that we handle changes from the current transaction ✅

[master < Epic] PR

  • Check, and update documentation if necessary
  • Write E2E tests
  • Compare the benchmarking results between the master branch and the Epic branch
  • Provide the full content or a guide for the final git message

[master < Task] PR

  • Check, and update documentation if necessary
  • Provide the full content or a guide for the final git message

To keep docs changelog up to date, one more thing to do:

  • Write a release note here, including added/changed clauses
  • Tag someone from docs team in the comments

@as51340 as51340 added feature feature Docs needed Docs needed on disk labels Oct 6, 2023
@as51340 as51340 self-assigned this Oct 6, 2023
@as51340 as51340 added this to the mg-v2.12.0 milestone Oct 6, 2023
@as51340 as51340 marked this pull request as ready for review October 6, 2023 11:03
src/storage/v2/disk/storage.cpp Outdated Show resolved Hide resolved
src/storage/v2/disk/storage.cpp Show resolved Hide resolved
src/storage/v2/disk/storage.cpp Show resolved Hide resolved
src/storage/v2/disk/storage.cpp Show resolved Hide resolved
src/storage/v2/storage.cpp Show resolved Hide resolved
tests/unit/clearing_old_disk_data.cpp Show resolved Hide resolved
tests/unit/storage_v2_edge_ondisk.cpp Outdated Show resolved Hide resolved
tests/unit/query_dump.cpp Show resolved Hide resolved
src/storage/v2/vertex_accessor.cpp Outdated Show resolved Hide resolved
@as51340 as51340 merged commit 1a3c5af into master Oct 11, 2023
6 checks passed
@as51340 as51340 deleted the new-merge branch October 11, 2023 08:18
as51340 added a commit that referenced this pull request Nov 10, 2023
* Improve disk expansions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
No open projects
Status: Done
Development

Successfully merging this pull request may close these issues.

None yet

3 participants