Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Peak memory observation in the system #788

Open
wants to merge 6 commits into
base: memgraph-2-18
Choose a base branch
from
Open

Peak memory observation in the system #788

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c88bd4d
Peak memory observation in the system
Josipmrden May 22, 2024
9b2a537
Merge branch 'main' into observe-peak-memory-docs
Josipmrden May 29, 2024
a0c727e
Update storage info where needed
Josipmrden May 29, 2024
2cabff4
Update best practics
Josipmrden May 29, 2024
1e06655
Merge branch 'memgraph-2-18' into observe-peak-memory-docs
kgolubic Jun 4, 2024
0b05e61
Add information from PR review
Josipmrden Jun 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
15 changes: 15 additions & 0 deletions pages/data-migration/best-practices.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -540,6 +540,21 @@ of the `--storage-gc-cycle-sec` flag. That means that Memgraph's garbage
collector potentially still didn't deallocate unused objects and free the
memory. You can free up memory by running the `FREE MEMORY` query.

## Monitoring memory during import

Often times, the user needs to batch transactions in the `IN_MEMORY_TRANSACTIONAL` storage mode.
This is due to large-update transactions that are creating a lot of `Delta` objects during the
transaction lifetime. `Delta` objects are necessary for reverting the query if an error during
query execution happens. They are approximately of 56 bytes in size, but large-update transactions
can make the number of delta objects be huge - therefore resulting in Memgraph getting out of memory.

We have exposed the `peak_memory_res` variable in the `SHOW STORAGE INFO;` command, with which you
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Explain that res is an abbreviation for "resident"

can monitor when the peak memory rises in the system. It will help you diagnose bottlenecks, and queries
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awkward. Too many comma. Try something like:
It will help you diagnose bottlenecks and high-memory queries which can be optimized.

which utilize a lot of memory, and can be optimized. That is most common in import queries because users
would want to import the whole dataset with one command only.

For more information, check our [storage memory usage](/fundamentals/storage-memory-usage).

## Do you have more questions?

Schedule a 30 min session with one of our engineers to discuss how Memgraph fits
Expand Down
4 changes: 3 additions & 1 deletion pages/database-management/server-stats.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,15 @@ The result will contain the following fields:

| Field | Description |
|------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| name | Name of the current database. |
| vertex_count | The number of stored nodes (vertices). |
| edge_count | The number of stored relationships (edges). |
| average_degree | The average number of relationships of a single node. |
| vm_max_map_count | The number of memory-mapped areas that the kernel allows a process to have. If it is unknown, returns -1. </br> For more info, check out [virtual memory section of the docs](/fundamentals/storage-memory-usage#virtual-memory). |
| memory_res | The non-swapped physical RAM memory a task has used, reported by the OS (in B, KiB, MiB, GiB or TiB). |
| peak_memory_res | Peak RAM memory usage in the system during the whole run. |
| disk_usage | The amount of disk space used by the data directory (in B, KiB, MiB, GiB or TiB). |
| memory_tracked | The amount of RAM allocated in the system and tracked by Memgraph (in B, KiB, MiB, GiB or TiB).<br/>For more info, check out [memory control](/fundamentals/storage-memory-usage). |
| vm_max_map_count | The number of memory-mapped areas that the kernel allows a process to have. If it is unknown, returns -1. </br> For more info, check out [virtual memory section of the docs](/fundamentals/storage-memory-usage#virtual-memory). |
| allocation_limit | The current allocation limit set for this instance (in B, KiB, MiB, GiB or TiB).<br/>For more info, check out the [memory control](/fundamentals/storage-memory-usage#control-memory-usage). |
| global_isolation_level | The current `global` isolation level.<br/>For more info, check out [isolation levels](/fundamentals/transactions#isolation-levels). |
| session_isolation_level | The current `session` isolation level. |
Expand Down
16 changes: 9 additions & 7 deletions pages/fundamentals/storage-memory-usage.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -729,13 +729,15 @@ SHOW STORAGE INFO;
| storage info | value |
+--------------------------------+--------------------------------+
| "name" | "memgraph" |
| "vertex_count" | 2677 |
| "edge_count" | 11967 |
| "average_degree" | 8.94061 |
| "memory_usage" | "44.32MiB" |
| "disk_usage" | "133.63KiB" |
| "memory_allocated" | "6.74MiB" |
| "allocation_limit" | "15.28GiB" |
| "vertex_count" | 0 |
| "edge_count" | 0 |
| "average_degree" | 0 |
| "vm_max_map_count" | 453125 |
| "memory_res" | "43.16MiB" |
| "peak_memory_res" | "43.16MiB" |
| "disk_usage" | "104.46KiB" |
| "memory_tracked" | "8.52MiB" |
| "allocation_limit" | "58.55GiB" |
| "global_isolation_level" | "SNAPSHOT_ISOLATION" |
| "session_isolation_level" | "" |
| "next_session_isolation_level" | "" |
Expand Down