Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[master < ] Parallelize recovery #868

Merged
merged 33 commits into from
Apr 25, 2023
Merged

[master < ] Parallelize recovery #868

merged 33 commits into from
Apr 25, 2023

Conversation

antaljanosbenjamin
Copy link
Contributor

[master < Epic] PR

  • Check, and update documentation if necessary
  • Write E2E tests
  • Compare the benchmarking results between the master branch and the Epic branch
  • Provide the full content or a guide for the final git message

[master < Task] PR

  • Check, and update documentation if necessary
  • Provide the full content or a guide for the final git message

To keep docs changelog up to date, one more thing to do:

  • Write a release note here
  • Tag someone from docs team in the comments

@antaljanosbenjamin antaljanosbenjamin self-assigned this Apr 6, 2023
@gitbuda gitbuda mentioned this pull request Apr 7, 2023
Closed
1 task
@antaljanosbenjamin antaljanosbenjamin linked an issue Apr 7, 2023 that may be closed by this pull request
Improve database recovery time #859
Closed
1 task
@antaljanosbenjamin antaljanosbenjamin removed a link to an issue Apr 7, 2023
Improve database recovery time #859
Closed
1 task
@antaljanosbenjamin
Copy link
Contributor Author

antaljanosbenjamin commented Apr 7, 2023 •
edited
Loading

Here some measurements from my laptop and from wolverine. The important intervals:

  • between Recovering edges. and the first Recovering 3750000 edges.: the first pass to figure out offsets for the multiple threads to recover edges from (+ starting the first thread)
  • between the first Recovering 3750000 edges. and Edges are recovered.: constructing the edges skiplist and destroying the threads
  • between Recovering vertices. and the first Recovering 1250000 vertices. the first pass to figure out offsets for the multiple threads to recover vertices from (+ starting the first thread)
  • between the first Recovering 1250000 vertices. and Vertices are recovered.: constructing the vertices skiplist with the labels and properties of the vertices and destroying the threads
  • Recover connectivity. and Connectivity is recovered.: adding in and out edges to vertices
My laptop

Original version

[2023-04-04 22:20:57.197] [memgraph_log] [warning] Invalid license key string. To use Enterprise features please set it to a valid string using the following query:
SET DATABASE SETTING "enterprise.license" TO "your-license-key"
[2023-04-04 22:20:57.208] [memgraph_log] [info] Recovering persisted data using snapshot ("/home/kovi/Repos/memgraph/build/data/snapshots") and WAL directory ("/home/kovi/Repos/memgraph/build/data/wal").
[2023-04-04 22:20:57.208] [memgraph_log] [info] Try recovering from snapshot directory "/home/kovi/Repos/memgraph/build/data/snapshots".
[2023-04-04 22:20:57.208] [memgraph_log] [info] Starting snapshot recovery from "/home/kovi/Repos/memgraph/build/data/snapshots/20230404132023784337_timestamp_6".
[2023-04-04 22:20:57.209] [memgraph_log] [info] Recovering 1000000 vertices and 3000000 edges.
[2023-04-04 22:20:57.209] [memgraph_log] [info] Recovering mapper metadata.
[2023-04-04 22:20:57.209] [memgraph_log] [info] Recovering 3000000 edges.
[2023-04-04 22:21:00.673] [memgraph_log] [info] Edges are recovered.
[2023-04-04 22:21:00.673] [memgraph_log] [info] Recovering 1000000 vertices.
[2023-04-04 22:21:02.199] [memgraph_log] [info] Vertices are recovered.
[2023-04-04 22:21:02.199] [memgraph_log] [info] Recovering connectivity.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Connectivity is recovered.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Recovering metadata of indices.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Recovering metadata of 0 label indices.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Metadata of label indices are recovered.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Recovering metadata of 1 label+property indices.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Metadata of label+property indices are recovered.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Metadata of indices are recovered.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Recovering metadata of constraints.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Recovering metadata of 0 existence constraints.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Metadata of existence constraints are recovered.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Recovering metadata of 0 unique constraints.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Metadata of unique constraints are recovered.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Metadata of constraints are recovered.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Recovering metadata.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Metadata recovered.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Snapshot recovery successful!
[2023-04-04 22:21:16.552] [memgraph_log] [info] Recreating indices from metadata.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Recreating 0 label indices from metadata.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Label indices are recreated.
[2023-04-04 22:21:16.552] [memgraph_log] [info] Recreating 1 label+property indices from metadata.
[2023-04-04 22:21:18.192] [memgraph_log] [info] A label+property index is recreated from metadata.
[2023-04-04 22:21:18.192] [memgraph_log] [info] Label+property indices are recreated.
[2023-04-04 22:21:18.192] [memgraph_log] [info] Indices are recreated.
[2023-04-04 22:21:18.192] [memgraph_log] [info] Recreating constraints from metadata.
[2023-04-04 22:21:18.192] [memgraph_log] [info] Recreating 0 existence constraints from metadata.
[2023-04-04 22:21:18.192] [memgraph_log] [info] Existence constraints are recreated from metadata.
[2023-04-04 22:21:18.192] [memgraph_log] [info] Recreating 0 unique constraints from metadata.
[2023-04-04 22:21:18.192] [memgraph_log] [info] Unique constraints are recreated from metadata.
[2023-04-04 22:21:18.192] [memgraph_log] [info] Constraints are recreated from metadata.
[2023-04-04 22:21:18.192] [memgraph_log] [info] Replica's configuration will be stored and will be automatically restored in case of a crash.
[2023-04-04 22:21:18.231] [memgraph_log] [info] Restoring replicas.
[2023-04-04 22:21:18.351] [memgraph_log] [info] Loading triggers...
[2023-04-04 22:21:18.351] [memgraph_log] [info] Loading streams...
[2023-04-04 22:21:18.351] [memgraph_log] [warning] Using non-secure Bolt connection (without SSL). For more details, visit https://memgr.ph/ssl.

8 threads

[2023-04-04 22:23:41.871] [memgraph_log] [warning] Invalid license key string. To use Enterprise features please set it to a valid string using the following query:
SET DATABASE SETTING "enterprise.license" TO "your-license-key"
[2023-04-04 22:23:41.882] [memgraph_log] [info] Recovering persisted data using snapshot ("/home/kovi/Repos/memgraph/build/data/snapshots") and WAL directory ("/home/kovi/Repos/memgraph/build/data/wal").
[2023-04-04 22:23:41.882] [memgraph_log] [info] Try recovering from snapshot directory "/home/kovi/Repos/memgraph/build/data/snapshots".
[2023-04-04 22:23:41.882] [memgraph_log] [info] Starting snapshot recovery from "/home/kovi/Repos/memgraph/build/data/snapshots/20230404132023784337_timestamp_6".
[2023-04-04 22:23:41.882] [memgraph_log] [info] Recovering 1000000 vertices and 3000000 edges.
[2023-04-04 22:23:41.882] [memgraph_log] [info] Recovering mapper metadata.
[2023-04-04 22:23:41.882] [memgraph_log] [info] Recovering edges.
[2023-04-04 22:23:42.811] [memgraph_log] [info] Recovering 375000 edges.
[2023-04-04 22:23:42.811] [memgraph_log] [info] Recovering 375000 edges.
[2023-04-04 22:23:42.811] [memgraph_log] [info] Recovering 375000 edges.
[2023-04-04 22:23:42.811] [memgraph_log] [info] Recovering 375000 edges.
[2023-04-04 22:23:42.811] [memgraph_log] [info] Recovering 375000 edges.
[2023-04-04 22:23:42.811] [memgraph_log] [info] Recovering 375000 edges.
[2023-04-04 22:23:42.811] [memgraph_log] [info] Recovering 375000 edges.
[2023-04-04 22:23:42.811] [memgraph_log] [info] Recovering 375000 edges.
[2023-04-04 22:23:43.645] [memgraph_log] [info] Partial edges are recovered.
[2023-04-04 22:23:43.665] [memgraph_log] [info] Partial edges are recovered.
[2023-04-04 22:23:43.680] [memgraph_log] [info] Partial edges are recovered.
[2023-04-04 22:23:43.726] [memgraph_log] [info] Partial edges are recovered.
[2023-04-04 22:23:43.752] [memgraph_log] [info] Partial edges are recovered.
[2023-04-04 22:23:43.769] [memgraph_log] [info] Partial edges are recovered.
[2023-04-04 22:23:43.801] [memgraph_log] [info] Partial edges are recovered.
[2023-04-04 22:23:43.817] [memgraph_log] [info] Partial edges are recovered.
[2023-04-04 22:23:43.817] [memgraph_log] [info] Edges are recovered.
[2023-04-04 22:23:43.817] [memgraph_log] [info] Recovering vertices.
[2023-04-04 22:23:44.395] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:44.395] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:44.395] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:44.395] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:44.395] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:44.395] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:44.395] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:44.395] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:44.702] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:44.704] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:44.716] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:44.717] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:44.718] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:44.719] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:44.785] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:44.790] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:44.791] [memgraph_log] [info] Vertices are recovered.
[2023-04-04 22:23:44.791] [memgraph_log] [info] Recover connectivity.
[2023-04-04 22:23:44.791] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:44.791] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:44.791] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:44.791] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:44.791] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:44.791] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:44.791] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:44.792] [memgraph_log] [info] Recovering 125000 vertices.
[2023-04-04 22:23:47.580] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:47.594] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:47.630] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:47.644] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:47.674] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:47.681] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:47.685] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:47.744] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Connectivity is recovered.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Recovering metadata of indices.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Recovering metadata of 0 label indices.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Metadata of label indices are recovered.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Recovering metadata of 1 label+property indices.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Metadata of label+property indices are recovered.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Metadata of indices are recovered.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Recovering metadata of constraints.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Recovering metadata of 0 existence constraints.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Metadata of existence constraints are recovered.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Recovering metadata of 0 unique constraints.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Metadata of unique constraints are recovered.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Metadata of constraints are recovered.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Recovering metadata.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Metadata recovered.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Snapshot recovery successful!
[2023-04-04 22:23:47.745] [memgraph_log] [info] Recreating indices from metadata.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Recreating 0 label indices from metadata.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Label indices are recreated.
[2023-04-04 22:23:47.745] [memgraph_log] [info] Recreating 1 label+property indices from metadata.
[2023-04-04 22:23:49.601] [memgraph_log] [info] A label+property index is recreated from metadata.
[2023-04-04 22:23:49.601] [memgraph_log] [info] Label+property indices are recreated.
[2023-04-04 22:23:49.601] [memgraph_log] [info] Indices are recreated.
[2023-04-04 22:23:49.601] [memgraph_log] [info] Recreating constraints from metadata.
[2023-04-04 22:23:49.601] [memgraph_log] [info] Recreating 0 existence constraints from metadata.
[2023-04-04 22:23:49.601] [memgraph_log] [info] Existence constraints are recreated from metadata.
[2023-04-04 22:23:49.601] [memgraph_log] [info] Recreating 0 unique constraints from metadata.
[2023-04-04 22:23:49.601] [memgraph_log] [info] Unique constraints are recreated from metadata.
[2023-04-04 22:23:49.601] [memgraph_log] [info] Constraints are recreated from metadata.
[2023-04-04 22:23:49.601] [memgraph_log] [info] Replica's configuration will be stored and will be automatically restored in case of a crash.
[2023-04-04 22:23:49.613] [memgraph_log] [info] Restoring replicas.
[2023-04-04 22:23:49.733] [memgraph_log] [info] Loading triggers...
[2023-04-04 22:23:49.733] [memgraph_log] [info] Loading streams...
[2023-04-04 22:23:49.733] [memgraph_log] [warning] Using non-secure Bolt connection (without SSL). For more details, visit https://memgr.ph/ssl.

4 threads

[2023-04-04 22:27:28.201] [memgraph_log] [warning] Invalid license key string. To use Enterprise features please set it to a valid string using the following query:
SET DATABASE SETTING "enterprise.license" TO "your-license-key"
[2023-04-04 22:27:28.212] [memgraph_log] [info] Recovering persisted data using snapshot ("/home/kovi/Repos/memgraph/build/data/snapshots") and WAL directory ("/home/kovi/Repos/memgraph/build/data/wal").
[2023-04-04 22:27:28.212] [memgraph_log] [info] Try recovering from snapshot directory "/home/kovi/Repos/memgraph/build/data/snapshots".
[2023-04-04 22:27:28.212] [memgraph_log] [info] Starting snapshot recovery from "/home/kovi/Repos/memgraph/build/data/snapshots/20230404132023784337_timestamp_6".
[2023-04-04 22:27:28.213] [memgraph_log] [info] Recovering 1000000 vertices and 3000000 edges.
[2023-04-04 22:27:28.213] [memgraph_log] [info] Recovering mapper metadata.
[2023-04-04 22:27:28.213] [memgraph_log] [info] Recovering edges.
[2023-04-04 22:27:29.017] [memgraph_log] [info] Recovering 750000 edges.
[2023-04-04 22:27:29.017] [memgraph_log] [info] Recovering 750000 edges.
[2023-04-04 22:27:29.017] [memgraph_log] [info] Recovering 750000 edges.
[2023-04-04 22:27:29.017] [memgraph_log] [info] Recovering 750000 edges.
[2023-04-04 22:27:30.345] [memgraph_log] [info] Partial edges are recovered.
[2023-04-04 22:27:30.354] [memgraph_log] [info] Partial edges are recovered.
[2023-04-04 22:27:30.366] [memgraph_log] [info] Partial edges are recovered.
[2023-04-04 22:27:30.381] [memgraph_log] [info] Partial edges are recovered.
[2023-04-04 22:27:30.381] [memgraph_log] [info] Edges are recovered.
[2023-04-04 22:27:30.381] [memgraph_log] [info] Recovering vertices.
[2023-04-04 22:27:30.891] [memgraph_log] [info] Recovering 250000 vertices.
[2023-04-04 22:27:30.891] [memgraph_log] [info] Recovering 250000 vertices.
[2023-04-04 22:27:30.891] [memgraph_log] [info] Recovering 250000 vertices.
[2023-04-04 22:27:30.891] [memgraph_log] [info] Recovering 250000 vertices.
[2023-04-04 22:27:31.469] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:27:31.471] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:27:31.472] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:27:31.488] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:27:31.488] [memgraph_log] [info] Vertices are recovered.
[2023-04-04 22:27:31.488] [memgraph_log] [info] Recover connectivity.
[2023-04-04 22:27:31.489] [memgraph_log] [info] Recovering 250000 vertices.
[2023-04-04 22:27:31.489] [memgraph_log] [info] Recovering 250000 vertices.
[2023-04-04 22:27:31.489] [memgraph_log] [info] Recovering 250000 vertices.
[2023-04-04 22:27:31.489] [memgraph_log] [info] Recovering 250000 vertices.
[2023-04-04 22:27:36.654] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:27:36.674] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:27:36.688] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:27:36.696] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-04 22:27:36.696] [memgraph_log] [info] Connectivity is recovered.
[2023-04-04 22:27:36.696] [memgraph_log] [info] Recovering metadata of indices.
[2023-04-04 22:27:36.696] [memgraph_log] [info] Recovering metadata of 0 label indices.
[2023-04-04 22:27:36.696] [memgraph_log] [info] Metadata of label indices are recovered.
[2023-04-04 22:27:36.696] [memgraph_log] [info] Recovering metadata of 1 label+property indices.
[2023-04-04 22:27:36.696] [memgraph_log] [info] Metadata of label+property indices are recovered.
[2023-04-04 22:27:36.696] [memgraph_log] [info] Metadata of indices are recovered.
[2023-04-04 22:27:36.696] [memgraph_log] [info] Recovering metadata of constraints.
[2023-04-04 22:27:36.696] [memgraph_log] [info] Recovering metadata of 0 existence constraints.
[2023-04-04 22:27:36.696] [memgraph_log] [info] Metadata of existence constraints are recovered.
[2023-04-04 22:27:36.697] [memgraph_log] [info] Recovering metadata of 0 unique constraints.
[2023-04-04 22:27:36.697] [memgraph_log] [info] Metadata of unique constraints are recovered.
[2023-04-04 22:27:36.697] [memgraph_log] [info] Metadata of constraints are recovered.
[2023-04-04 22:27:36.697] [memgraph_log] [info] Recovering metadata.
[2023-04-04 22:27:36.697] [memgraph_log] [info] Metadata recovered.
[2023-04-04 22:27:36.697] [memgraph_log] [info] Snapshot recovery successful!
[2023-04-04 22:27:36.697] [memgraph_log] [info] Recreating indices from metadata.
[2023-04-04 22:27:36.697] [memgraph_log] [info] Recreating 0 label indices from metadata.
[2023-04-04 22:27:36.697] [memgraph_log] [info] Label indices are recreated.
[2023-04-04 22:27:36.697] [memgraph_log] [info] Recreating 1 label+property indices from metadata.
[2023-04-04 22:27:38.558] [memgraph_log] [info] A label+property index is recreated from metadata.
[2023-04-04 22:27:38.558] [memgraph_log] [info] Label+property indices are recreated.
[2023-04-04 22:27:38.558] [memgraph_log] [info] Indices are recreated.
[2023-04-04 22:27:38.558] [memgraph_log] [info] Recreating constraints from metadata.
[2023-04-04 22:27:38.558] [memgraph_log] [info] Recreating 0 existence constraints from metadata.
[2023-04-04 22:27:38.558] [memgraph_log] [info] Existence constraints are recreated from metadata.
[2023-04-04 22:27:38.558] [memgraph_log] [info] Recreating 0 unique constraints from metadata.
[2023-04-04 22:27:38.558] [memgraph_log] [info] Unique constraints are recreated from metadata.
[2023-04-04 22:27:38.558] [memgraph_log] [info] Constraints are recreated from metadata.
[2023-04-04 22:27:38.558] [memgraph_log] [info] Replica's configuration will be stored and will be automatically restored in case of a crash.
[2023-04-04 22:27:38.573] [memgraph_log] [info] Restoring replicas.
[2023-04-04 22:27:38.694] [memgraph_log] [info] Loading triggers...
[2023-04-04 22:27:38.694] [memgraph_log] [info] Loading streams...
[2023-04-04 22:27:38.694] [memgraph_log] [warning] Using non-secure Bolt connection (without SSL). For more details, visit https://memgr.ph/ssl.
wolverine

LOAD CSV FROM "/home/mg/benjo/generated_text_nodes_big.csv" NO HEADER AS row
CREATE (:Label {arg0:row[0],arg1:row[1],arg2:row[2],arg3:row[3]});
Query for edges creation:
LOAD CSV FROM "/home/mg/benjo/generated_text_edges_big.csv" NO HEADER AS row
MATCH (a:Label {arg0:row[0]}), (b:Label {arg0:row[1]}) CREATE (a)-[:EDGE_TYPE {arg2:row[2],arg3:row[3],arg4:row[4],arg5:row[5],arg6:row[6]}]->(b);

Original version

[2023-04-05 13:28:17.871] [memgraph_log] [warning] Invalid license key string. To use Enterprise features please set it to a valid string using the following query:
SET DATABASE SETTING "enterprise.license" TO "your-license-key"
[2023-04-05 13:28:17.964] [memgraph_log] [info] Recovering persisted data using snapshot ("/home/mg/benjo/memgraph/build/data/snapshots") and WAL directory ("/home/mg/benjo/memgraph/build/data/wal").
[2023-04-05 13:28:17.964] [memgraph_log] [info] Try recovering from snapshot directory "/home/mg/benjo/memgraph/build/data/snapshots".
[2023-04-05 13:28:17.964] [memgraph_log] [info] Starting snapshot recovery from "/home/mg/benjo/memgraph/build/data/snapshots/20230405112629149696_timestamp_10".
[2023-04-05 13:28:17.964] [memgraph_log] [info] Recovering 5000000 vertices and 15000000 edges.
[2023-04-05 13:28:17.964] [memgraph_log] [info] Recovering mapper metadata.
[2023-04-05 13:28:17.964] [memgraph_log] [info] Recovering 15000000 edges.
[2023-04-05 13:28:46.614] [memgraph_log] [info] Edges are recovered.
[2023-04-05 13:28:46.614] [memgraph_log] [info] Recovering 5000000 vertices.
[2023-04-05 13:28:57.853] [memgraph_log] [info] Vertices are recovered.
[2023-04-05 13:28:57.853] [memgraph_log] [info] Recovering connectivity.
[2023-04-05 13:30:31.291] [memgraph_log] [info] Connectivity is recovered.
[2023-04-05 13:30:31.291] [memgraph_log] [info] Recovering metadata of indices.
[2023-04-05 13:30:31.291] [memgraph_log] [info] Recovering metadata of 0 label indices.
[2023-04-05 13:30:31.291] [memgraph_log] [info] Metadata of label indices are recovered.
[2023-04-05 13:30:31.291] [memgraph_log] [info] Recovering metadata of 1 label+property indices.
[2023-04-05 13:30:31.292] [memgraph_log] [info] Metadata of label+property indices are recovered.
[2023-04-05 13:30:31.292] [memgraph_log] [info] Metadata of indices are recovered.
[2023-04-05 13:30:31.292] [memgraph_log] [info] Recovering metadata of constraints.
[2023-04-05 13:30:31.292] [memgraph_log] [info] Recovering metadata of 0 existence constraints.
[2023-04-05 13:30:31.292] [memgraph_log] [info] Metadata of existence constraints are recovered.
[2023-04-05 13:30:31.292] [memgraph_log] [info] Recovering metadata of 0 unique constraints.
[2023-04-05 13:30:31.292] [memgraph_log] [info] Metadata of unique constraints are recovered.
[2023-04-05 13:30:31.292] [memgraph_log] [info] Metadata of constraints are recovered.
[2023-04-05 13:30:31.292] [memgraph_log] [info] Recovering metadata.
[2023-04-05 13:30:31.292] [memgraph_log] [info] Metadata recovered.
[2023-04-05 13:30:31.292] [memgraph_log] [info] Snapshot recovery successful!
[2023-04-05 13:30:31.292] [memgraph_log] [info] Recreating indices from metadata.
[2023-04-05 13:30:31.292] [memgraph_log] [info] Recreating 0 label indices from metadata.
[2023-04-05 13:30:31.292] [memgraph_log] [info] Label indices are recreated.
[2023-04-05 13:30:31.292] [memgraph_log] [info] Recreating 1 label+property indices from metadata.
[2023-04-05 13:30:44.553] [memgraph_log] [info] A label+property index is recreated from metadata.
[2023-04-05 13:30:44.553] [memgraph_log] [info] Label+property indices are recreated.
[2023-04-05 13:30:44.553] [memgraph_log] [info] Indices are recreated.
[2023-04-05 13:30:44.553] [memgraph_log] [info] Recreating constraints from metadata.
[2023-04-05 13:30:44.553] [memgraph_log] [info] Recreating 0 existence constraints from metadata.
[2023-04-05 13:30:44.553] [memgraph_log] [info] Existence constraints are recreated from metadata.
[2023-04-05 13:30:44.553] [memgraph_log] [info] Recreating 0 unique constraints from metadata.
[2023-04-05 13:30:44.553] [memgraph_log] [info] Unique constraints are recreated from metadata.
[2023-04-05 13:30:44.553] [memgraph_log] [info] Constraints are recreated from metadata.
[2023-04-05 13:30:44.553] [memgraph_log] [info] Replica's configuration will be stored and will be automatically restored in case of a crash.
[2023-04-05 13:30:44.634] [memgraph_log] [info] Restoring replicas.
[2023-04-05 13:30:44.908] [memgraph_log] [info] Loading triggers...
[2023-04-05 13:30:44.908] [memgraph_log] [info] Loading streams...
[2023-04-05 13:30:44.908] [memgraph_log] [warning] Using non-secure Bolt connection (without SSL). For more details, visit https://memgr.ph/ssl.

16 threads

[2023-04-05 14:06:05.868] [memgraph_log] [warning] Invalid license key string. To use Enterprise features please set it to a valid string using the following query:
SET DATABASE SETTING "enterprise.license" TO "your-license-key"
[2023-04-05 14:06:05.954] [memgraph_log] [info] Recovering persisted data using snapshot ("/home/mg/benjo/memgraph/build/data/snapshots") and WAL directory ("/home/mg/benjo/memgraph/build/data/wal").
[2023-04-05 14:06:05.954] [memgraph_log] [info] Try recovering from snapshot directory "/home/mg/benjo/memgraph/build/data/snapshots".
[2023-04-05 14:06:05.954] [memgraph_log] [info] Starting snapshot recovery from "/home/mg/benjo/memgraph/build/data/snapshots/20230405112629149696_timestamp_10".
[2023-04-05 14:06:05.955] [memgraph_log] [info] Recovering 5000000 vertices and 15000000 edges.
[2023-04-05 14:06:05.955] [memgraph_log] [info] Recovering mapper metadata.
[2023-04-05 14:06:05.955] [memgraph_log] [info] Recovering edges.
[2023-04-05 14:06:15.323] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:15.323] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:15.325] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:15.329] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:15.329] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:15.329] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:15.329] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:15.330] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:15.331] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:15.332] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:15.332] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:15.333] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:15.333] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:15.335] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:15.335] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:15.336] [memgraph_log] [info] Recovering 937500 edges.
[2023-04-05 14:06:21.339] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:21.343] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:21.391] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:21.396] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:21.468] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:21.568] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:21.588] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:21.785] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:22.465] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:22.498] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:22.562] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:22.579] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:22.633] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:22.636] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:22.664] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:22.672] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 14:06:22.672] [memgraph_log] [info] Edges are recovered.
[2023-04-05 14:06:22.672] [memgraph_log] [info] Recovering vertices.
[2023-04-05 14:06:28.361] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:28.361] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:28.361] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:28.361] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:28.361] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:28.361] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:28.361] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:28.362] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:28.363] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:28.364] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:28.364] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:28.364] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:28.365] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:28.365] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:28.367] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:28.367] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.283] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.323] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.419] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.549] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.626] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.656] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.678] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.698] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.722] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.770] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.788] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.809] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.813] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.832] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.840] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.880] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:31.880] [memgraph_log] [info] Vertices are recovered.
[2023-04-05 14:06:31.880] [memgraph_log] [info] Recover connectivity.
[2023-04-05 14:06:31.881] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.882] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.882] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.882] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.882] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.882] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.882] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.884] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.887] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.889] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.891] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.893] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.895] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.895] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.896] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:31.897] [memgraph_log] [info] Recovering 312500 vertices.
[2023-04-05 14:06:41.118] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:41.126] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:41.211] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:41.280] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:41.367] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:41.399] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:41.486] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:41.570] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:42.070] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:42.121] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:42.166] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:42.261] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:42.343] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:42.467] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:42.479] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:42.558] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 14:06:42.558] [memgraph_log] [info] Connectivity is recovered.
[2023-04-05 14:06:42.558] [memgraph_log] [info] Recovering metadata of indices.
[2023-04-05 14:06:42.558] [memgraph_log] [info] Recovering metadata of 0 label indices.
[2023-04-05 14:06:42.558] [memgraph_log] [info] Metadata of label indices are recovered.
[2023-04-05 14:06:42.558] [memgraph_log] [info] Recovering metadata of 1 label+property indices.
[2023-04-05 14:06:42.558] [memgraph_log] [info] Metadata of label+property indices are recovered.
[2023-04-05 14:06:42.558] [memgraph_log] [info] Metadata of indices are recovered.
[2023-04-05 14:06:42.558] [memgraph_log] [info] Recovering metadata of constraints.
[2023-04-05 14:06:42.558] [memgraph_log] [info] Recovering metadata of 0 existence constraints.
[2023-04-05 14:06:42.558] [memgraph_log] [info] Metadata of existence constraints are recovered.
[2023-04-05 14:06:42.559] [memgraph_log] [info] Recovering metadata of 0 unique constraints.
[2023-04-05 14:06:42.559] [memgraph_log] [info] Metadata of unique constraints are recovered.
[2023-04-05 14:06:42.559] [memgraph_log] [info] Metadata of constraints are recovered.
[2023-04-05 14:06:42.559] [memgraph_log] [info] Recovering metadata.
[2023-04-05 14:06:42.559] [memgraph_log] [info] Metadata recovered.
[2023-04-05 14:06:42.559] [memgraph_log] [info] Snapshot recovery successful!
[2023-04-05 14:06:42.559] [memgraph_log] [info] Recreating indices from metadata.
[2023-04-05 14:06:42.559] [memgraph_log] [info] Recreating 0 label indices from metadata.
[2023-04-05 14:06:42.559] [memgraph_log] [info] Label indices are recreated.
[2023-04-05 14:06:42.559] [memgraph_log] [info] Recreating 1 label+property indices from metadata.
[2023-04-05 14:06:56.853] [memgraph_log] [info] A label+property index is recreated from metadata.
[2023-04-05 14:06:56.853] [memgraph_log] [info] Label+property indices are recreated.
[2023-04-05 14:06:56.853] [memgraph_log] [info] Indices are recreated.
[2023-04-05 14:06:56.853] [memgraph_log] [info] Recreating constraints from metadata.
[2023-04-05 14:06:56.853] [memgraph_log] [info] Recreating 0 existence constraints from metadata.
[2023-04-05 14:06:56.853] [memgraph_log] [info] Existence constraints are recreated from metadata.
[2023-04-05 14:06:56.853] [memgraph_log] [info] Recreating 0 unique constraints from metadata.
[2023-04-05 14:06:56.853] [memgraph_log] [info] Unique constraints are recreated from metadata.
[2023-04-05 14:06:56.853] [memgraph_log] [info] Constraints are recreated from metadata.
[2023-04-05 14:06:56.853] [memgraph_log] [info] Replica's configuration will be stored and will be automatically restored in case of a crash.
[2023-04-05 14:06:56.945] [memgraph_log] [info] Restoring replicas.
[2023-04-05 14:06:57.232] [memgraph_log] [info] Loading triggers...
[2023-04-05 14:06:57.232] [memgraph_log] [info] Loading streams...
[2023-04-05 14:06:57.232] [memgraph_log] [warning] Using non-secure Bolt connection (without SSL). For more details, visit https://memgr.ph/ssl.

10 threads

[2023-04-05 13:57:26.092] [memgraph_log] [warning] Invalid license key string. To use Enterprise features please set it to a valid string using the following query:
SET DATABASE SETTING "enterprise.license" TO "your-license-key"
[2023-04-05 13:57:26.184] [memgraph_log] [info] Recovering persisted data using snapshot ("/home/mg/benjo/memgraph/build/data/snapshots") and WAL directory ("/home/mg/benjo/memgraph/build/data/wal").
[2023-04-05 13:57:26.184] [memgraph_log] [info] Try recovering from snapshot directory "/home/mg/benjo/memgraph/build/data/snapshots".
[2023-04-05 13:57:26.185] [memgraph_log] [info] Starting snapshot recovery from "/home/mg/benjo/memgraph/build/data/snapshots/20230405112629149696_timestamp_10".
[2023-04-05 13:57:26.185] [memgraph_log] [info] Recovering 5000000 vertices and 15000000 edges.
[2023-04-05 13:57:26.185] [memgraph_log] [info] Recovering mapper metadata.
[2023-04-05 13:57:26.185] [memgraph_log] [info] Recovering edges.
[2023-04-05 13:57:35.135] [memgraph_log] [info] Recovering 1500000 edges.
[2023-04-05 13:57:35.138] [memgraph_log] [info] Recovering 1500000 edges.
[2023-04-05 13:57:35.138] [memgraph_log] [info] Recovering 1500000 edges.
[2023-04-05 13:57:35.138] [memgraph_log] [info] Recovering 1500000 edges.
[2023-04-05 13:57:35.139] [memgraph_log] [info] Recovering 1500000 edges.
[2023-04-05 13:57:35.139] [memgraph_log] [info] Recovering 1500000 edges.
[2023-04-05 13:57:35.140] [memgraph_log] [info] Recovering 1500000 edges.
[2023-04-05 13:57:35.141] [memgraph_log] [info] Recovering 1500000 edges.
[2023-04-05 13:57:35.142] [memgraph_log] [info] Recovering 1500000 edges.
[2023-04-05 13:57:35.143] [memgraph_log] [info] Recovering 1500000 edges.
[2023-04-05 13:57:41.823] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:57:41.831] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:57:41.849] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:57:41.856] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:57:41.864] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:57:41.902] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:57:41.925] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:57:41.925] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:57:41.932] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:57:41.982] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:57:41.982] [memgraph_log] [info] Edges are recovered.
[2023-04-05 13:57:41.982] [memgraph_log] [info] Recovering vertices.
[2023-04-05 13:57:47.422] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:47.422] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:47.422] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:47.422] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:47.422] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:47.422] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:47.423] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:47.425] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:47.425] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:47.425] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:49.465] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:57:49.483] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:57:49.494] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:57:49.511] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:57:49.514] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:57:49.534] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:57:49.571] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:57:49.574] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:57:49.586] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:57:49.602] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:57:49.603] [memgraph_log] [info] Vertices are recovered.
[2023-04-05 13:57:49.603] [memgraph_log] [info] Recover connectivity.
[2023-04-05 13:57:49.604] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:49.604] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:49.604] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:49.604] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:49.604] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:49.604] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:49.608] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:49.611] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:49.614] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:57:49.615] [memgraph_log] [info] Recovering 500000 vertices.
[2023-04-05 13:58:02.534] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:58:02.648] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:58:02.683] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:58:02.691] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:58:02.941] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:58:03.015] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:58:03.346] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:58:03.349] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:58:03.360] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:58:03.373] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Connectivity is recovered.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Recovering metadata of indices.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Recovering metadata of 0 label indices.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Metadata of label indices are recovered.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Recovering metadata of 1 label+property indices.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Metadata of label+property indices are recovered.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Metadata of indices are recovered.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Recovering metadata of constraints.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Recovering metadata of 0 existence constraints.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Metadata of existence constraints are recovered.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Recovering metadata of 0 unique constraints.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Metadata of unique constraints are recovered.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Metadata of constraints are recovered.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Recovering metadata.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Metadata recovered.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Snapshot recovery successful!
[2023-04-05 13:58:03.374] [memgraph_log] [info] Recreating indices from metadata.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Recreating 0 label indices from metadata.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Label indices are recreated.
[2023-04-05 13:58:03.374] [memgraph_log] [info] Recreating 1 label+property indices from metadata.
[2023-04-05 13:58:17.232] [memgraph_log] [info] A label+property index is recreated from metadata.
[2023-04-05 13:58:17.232] [memgraph_log] [info] Label+property indices are recreated.
[2023-04-05 13:58:17.232] [memgraph_log] [info] Indices are recreated.
[2023-04-05 13:58:17.232] [memgraph_log] [info] Recreating constraints from metadata.
[2023-04-05 13:58:17.232] [memgraph_log] [info] Recreating 0 existence constraints from metadata.
[2023-04-05 13:58:17.232] [memgraph_log] [info] Existence constraints are recreated from metadata.
[2023-04-05 13:58:17.232] [memgraph_log] [info] Recreating 0 unique constraints from metadata.
[2023-04-05 13:58:17.232] [memgraph_log] [info] Unique constraints are recreated from metadata.
[2023-04-05 13:58:17.232] [memgraph_log] [info] Constraints are recreated from metadata.
[2023-04-05 13:58:17.232] [memgraph_log] [info] Replica's configuration will be stored and will be automatically restored in case of a crash.
[2023-04-05 13:58:17.319] [memgraph_log] [info] Restoring replicas.
[2023-04-05 13:58:17.593] [memgraph_log] [info] Loading triggers...
[2023-04-05 13:58:17.593] [memgraph_log] [info] Loading streams...
[2023-04-05 13:58:17.593] [memgraph_log] [warning] Using non-secure Bolt connection (without SSL). For more details, visit https://memgr.ph/ssl.

8 threads

[2023-04-05 13:35:28.216] [memgraph_log] [warning] Invalid license key string. To use Enterprise features please set it to a valid string using the following query:
SET DATABASE SETTING "enterprise.license" TO "your-license-key"
[2023-04-05 13:35:28.303] [memgraph_log] [info] Recovering persisted data using snapshot ("/home/mg/benjo/memgraph/build/data/snapshots") and WAL directory ("/home/mg/benjo/memgraph/build/data/wal").
[2023-04-05 13:35:28.303] [memgraph_log] [info] Try recovering from snapshot directory "/home/mg/benjo/memgraph/build/data/snapshots".
[2023-04-05 13:35:28.303] [memgraph_log] [info] Starting snapshot recovery from "/home/mg/benjo/memgraph/build/data/snapshots/20230405112629149696_timestamp_10".
[2023-04-05 13:35:28.303] [memgraph_log] [info] Recovering 5000000 vertices and 15000000 edges.
[2023-04-05 13:35:28.303] [memgraph_log] [info] Recovering mapper metadata.
[2023-04-05 13:35:28.303] [memgraph_log] [info] Recovering edges.
[2023-04-05 13:35:37.340] [memgraph_log] [info] Recovering 1875000 edges.
[2023-04-05 13:35:37.340] [memgraph_log] [info] Recovering 1875000 edges.
[2023-04-05 13:35:37.341] [memgraph_log] [info] Recovering 1875000 edges.
[2023-04-05 13:35:37.342] [memgraph_log] [info] Recovering 1875000 edges.
[2023-04-05 13:35:37.343] [memgraph_log] [info] Recovering 1875000 edges.
[2023-04-05 13:35:37.343] [memgraph_log] [info] Recovering 1875000 edges.
[2023-04-05 13:35:37.344] [memgraph_log] [info] Recovering 1875000 edges.
[2023-04-05 13:35:37.344] [memgraph_log] [info] Recovering 1875000 edges.
[2023-04-05 13:35:44.436] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:35:44.468] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:35:44.490] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:35:44.493] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:35:44.503] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:35:44.505] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:35:44.524] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:35:44.555] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:35:44.555] [memgraph_log] [info] Edges are recovered.
[2023-04-05 13:35:44.555] [memgraph_log] [info] Recovering vertices.
[2023-04-05 13:35:49.899] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:35:49.899] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:35:49.899] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:35:49.899] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:35:49.899] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:35:49.899] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:35:49.899] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:35:49.899] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:35:52.124] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:35:52.126] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:35:52.160] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:35:52.167] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:35:52.171] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:35:52.173] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:35:52.192] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:35:52.213] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:35:52.213] [memgraph_log] [info] Vertices are recovered.
[2023-04-05 13:35:52.213] [memgraph_log] [info] Recover connectivity.
[2023-04-05 13:35:52.214] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:35:52.214] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:35:52.214] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:35:52.214] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:35:52.214] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:35:52.214] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:35:52.216] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:35:52.216] [memgraph_log] [info] Recovering 625000 vertices.
[2023-04-05 13:36:07.709] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:36:07.726] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:36:07.871] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:36:07.897] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:36:08.124] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:36:08.313] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:36:08.345] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Connectivity is recovered.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Recovering metadata of indices.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Recovering metadata of 0 label indices.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Metadata of label indices are recovered.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Recovering metadata of 1 label+property indices.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Metadata of label+property indices are recovered.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Metadata of indices are recovered.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Recovering metadata of constraints.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Recovering metadata of 0 existence constraints.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Metadata of existence constraints are recovered.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Recovering metadata of 0 unique constraints.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Metadata of unique constraints are recovered.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Metadata of constraints are recovered.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Recovering metadata.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Metadata recovered.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Snapshot recovery successful!
[2023-04-05 13:36:08.384] [memgraph_log] [info] Recreating indices from metadata.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Recreating 0 label indices from metadata.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Label indices are recreated.
[2023-04-05 13:36:08.384] [memgraph_log] [info] Recreating 1 label+property indices from metadata.
[2023-04-05 13:36:21.660] [memgraph_log] [info] A label+property index is recreated from metadata.
[2023-04-05 13:36:21.660] [memgraph_log] [info] Label+property indices are recreated.
[2023-04-05 13:36:21.660] [memgraph_log] [info] Indices are recreated.
[2023-04-05 13:36:21.660] [memgraph_log] [info] Recreating constraints from metadata.
[2023-04-05 13:36:21.660] [memgraph_log] [info] Recreating 0 existence constraints from metadata.
[2023-04-05 13:36:21.660] [memgraph_log] [info] Existence constraints are recreated from metadata.
[2023-04-05 13:36:21.660] [memgraph_log] [info] Recreating 0 unique constraints from metadata.
[2023-04-05 13:36:21.660] [memgraph_log] [info] Unique constraints are recreated from metadata.
[2023-04-05 13:36:21.660] [memgraph_log] [info] Constraints are recreated from metadata.
[2023-04-05 13:36:21.660] [memgraph_log] [info] Replica's configuration will be stored and will be automatically restored in case of a crash.
[2023-04-05 13:36:21.752] [memgraph_log] [info] Restoring replicas.
[2023-04-05 13:36:22.033] [memgraph_log] [info] Loading triggers...
[2023-04-05 13:36:22.033] [memgraph_log] [info] Loading streams...
[2023-04-05 13:36:22.033] [memgraph_log] [warning] Using non-secure Bolt connection (without SSL). For more details, visit https://memgr.ph/ssl.

4 threads

[2023-04-05 13:48:46.950] [memgraph_log] [warning] Invalid license key string. To use Enterprise features please set it to a valid string using the following query:
SET DATABASE SETTING "enterprise.license" TO "your-license-key"
[2023-04-05 13:48:47.037] [memgraph_log] [info] Recovering persisted data using snapshot ("/home/mg/benjo/memgraph/build/data/snapshots") and WAL directory ("/home/mg/benjo/memgraph/build/data/wal").
[2023-04-05 13:48:47.037] [memgraph_log] [info] Try recovering from snapshot directory "/home/mg/benjo/memgraph/build/data/snapshots".
[2023-04-05 13:48:47.037] [memgraph_log] [info] Starting snapshot recovery from "/home/mg/benjo/memgraph/build/data/snapshots/20230405112629149696_timestamp_10".
[2023-04-05 13:48:47.037] [memgraph_log] [info] Recovering 5000000 vertices and 15000000 edges.
[2023-04-05 13:48:47.037] [memgraph_log] [info] Recovering mapper metadata.
[2023-04-05 13:48:47.037] [memgraph_log] [info] Recovering edges.
[2023-04-05 13:48:54.802] [memgraph_log] [info] Recovering 3750000 edges.
[2023-04-05 13:48:54.803] [memgraph_log] [info] Recovering 3750000 edges.
[2023-04-05 13:48:54.804] [memgraph_log] [info] Recovering 3750000 edges.
[2023-04-05 13:48:54.804] [memgraph_log] [info] Recovering 3750000 edges.
[2023-04-05 13:49:04.343] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:49:04.394] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:49:04.656] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:49:04.660] [memgraph_log] [info] Partial edges are recovered.
[2023-04-05 13:49:04.660] [memgraph_log] [info] Edges are recovered.
[2023-04-05 13:49:04.660] [memgraph_log] [info] Recovering vertices.
[2023-04-05 13:49:09.411] [memgraph_log] [info] Recovering 1250000 vertices.
[2023-04-05 13:49:09.411] [memgraph_log] [info] Recovering 1250000 vertices.
[2023-04-05 13:49:09.411] [memgraph_log] [info] Recovering 1250000 vertices.
[2023-04-05 13:49:09.411] [memgraph_log] [info] Recovering 1250000 vertices.
[2023-04-05 13:49:12.900] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:49:12.934] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:49:12.987] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:49:12.995] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:49:12.995] [memgraph_log] [info] Vertices are recovered.
[2023-04-05 13:49:12.995] [memgraph_log] [info]  
[2023-04-05 13:49:12.996] [memgraph_log] [info] Recovering 1250000 vertices.
[2023-04-05 13:49:12.996] [memgraph_log] [info] Recovering 1250000 vertices.
[2023-04-05 13:49:12.996] [memgraph_log] [info] Recovering 1250000 vertices.
[2023-04-05 13:49:12.996] [memgraph_log] [info] Recovering 1250000 vertices.
[2023-04-05 13:49:41.867] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:49:41.880] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:49:42.820] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Partial vertices are recovered.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Connectivity is recovered.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Recovering metadata of indices.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Recovering metadata of 0 label indices.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Metadata of label indices are recovered.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Recovering metadata of 1 label+property indices.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Metadata of label+property indices are recovered.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Metadata of indices are recovered.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Recovering metadata of constraints.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Recovering metadata of 0 existence constraints.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Metadata of existence constraints are recovered.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Recovering metadata of 0 unique constraints.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Metadata of unique constraints are recovered.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Metadata of constraints are recovered.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Recovering metadata.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Metadata recovered.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Snapshot recovery successful!
[2023-04-05 13:49:42.863] [memgraph_log] [info] Recreating indices from metadata.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Recreating 0 label indices from metadata.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Label indices are recreated.
[2023-04-05 13:49:42.863] [memgraph_log] [info] Recreating 1 label+property indices from metadata.
[2023-04-05 13:49:56.318] [memgraph_log] [info] A label+property index is recreated from metadata.
[2023-04-05 13:49:56.318] [memgraph_log] [info] Label+property indices are recreated.
[2023-04-05 13:49:56.318] [memgraph_log] [info] Indices are recreated.
[2023-04-05 13:49:56.318] [memgraph_log] [info] Recreating constraints from metadata.
[2023-04-05 13:49:56.318] [memgraph_log] [info] Recreating 0 existence constraints from metadata.
[2023-04-05 13:49:56.318] [memgraph_log] [info] Existence constraints are recreated from metadata.
[2023-04-05 13:49:56.318] [memgraph_log] [info] Recreating 0 unique constraints from metadata.
[2023-04-05 13:49:56.318] [memgraph_log] [info] Unique constraints are recreated from metadata.
[2023-04-05 13:49:56.318] [memgraph_log] [info] Constraints are recreated from metadata.
[2023-04-05 13:49:56.318] [memgraph_log] [info] Replica's configuration will be stored and will be automatically restored in case of a crash.
[2023-04-05 13:49:56.408] [memgraph_log] [info] Restoring replicas.
[2023-04-05 13:49:56.694] [memgraph_log] [info] Loading triggers...
[2023-04-05 13:49:56.694] [memgraph_log] [info] Loading streams...
[2023-04-05 13:49:56.694] [memgraph_log] [warning] Using non-secure Bolt connection (without SSL). For more details, visit https://memgr.ph/ssl.

The measurements from wolverine visualized:
image

@gitbuda gitbuda added this to the mg-v2.8.0 milestone Apr 7, 2023
@gitbuda gitbuda changed the title Parallelize recovery [master < ] Parallelize recovery Apr 9, 2023
@gitbuda gitbuda added the feature feature label Apr 9, 2023
János Benjamin Antal added 15 commits April 11, 2023 18:39
antoniofilipovic
antoniofilipovic reviewed Apr 18, 2023
View reviewed changes
Copy link
Collaborator

@antoniofilipovic antoniofilipovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to check tests still, will do that tomorrow morning

@antaljanosbenjamin antaljanosbenjamin added the Ready for review PR is ready for review label Apr 19, 2023
antoniofilipovic
antoniofilipovic reviewed Apr 19, 2023
View reviewed changes
Copy link
Collaborator

@antoniofilipovic antoniofilipovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at tests, all seems good to me

@antoniofilipovic antoniofilipovic removed the Ready for review PR is ready for review label Apr 19, 2023
@gitbuda
Copy link
Member

gitbuda commented Apr 24, 2023

@antoniofilipovic I think first #882 has to be merged here (before we merge this to master)? 👀

antoniofilipovic
antoniofilipovic approved these changes Apr 24, 2023
View reviewed changes
Copy link
Collaborator

@antoniofilipovic antoniofilipovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good 🚀

@antaljanosbenjamin antaljanosbenjamin merged commit 7dec982 into master Apr 25, 2023
@antaljanosbenjamin antaljanosbenjamin deleted the MG-parallelize-recovery branch April 25, 2023 14:15
antaljanosbenjamin added a commit that referenced this pull request Apr 25, 2023
@antaljanosbenjamin
Revert "[master < ] Parallelize recovery (#868)"
603f1a0 
This reverts commit 7dec982.
@antaljanosbenjamin antaljanosbenjamin restored the MG-parallelize-recovery branch April 25, 2023 14:19
antaljanosbenjamin pushed a commit that referenced this pull request Apr 25, 2023
Parallelize recovery (#868)
6c94794 
* Parallelize edge recovery

* Load vertex labels and properties parallel

* Add parallel connectivity loading

* Add batches information to snapshot

* Introduce `items_per_batch` and `recovery_thread_count` flags

* Make possible to load snapshots with old version

* Add vertex batches to `RecoveryInfo`

* Extend durability integration tests with v15 test cases

* Add `std::vector` based `InitProperties`

* Use `InitProperties` in snapshot loading
@antaljanosbenjamin
Copy link
Contributor Author

I force-pushed master back and then did the merge manually on my machine, because I accidentally left the [master <] part in the commit message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@antoniofilipovic antoniofilipovic antoniofilipovic approved these changes

@gvolfing gvolfing gvolfing approved these changes

Assignees

@antaljanosbenjamin antaljanosbenjamin

Labels
feature feature
Projects
None yet
Milestone
mg-v2.8.0
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@antaljanosbenjamin @gitbuda @antoniofilipovic @gvolfing