New journal disk based indexing for agent memory reduction #13885
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
This pull request introduces 11 alerts when merging 4c281aa into b1bc23d - view on LGTM.com new alerts:
|
|
|
|
|
|
This pull request introduces 11 alerts when merging f6af2c0 into 9e89ac7 - view on LGTM.com new alerts:
|
|
@stelfrag I fixed both crashes. |
|
This pull request introduces 11 alerts when merging 0ada1bb into 9e89ac7 - view on LGTM.com new alerts:
|
stelfrag
force-pushed
the
0_journal_v2
branch
from
0ada1bb
to
6c63ade
Compare
|
This pull request introduces 10 alerts when merging 6c63ade into 185f5a1 - view on LGTM.com new alerts:
|
stelfrag
force-pushed
the
0_journal_v2
branch
from
6c63ade
to
76838d0
Compare
|
This pull request introduces 10 alerts when merging 76838d0 into 88c03ca - view on LGTM.com new alerts:
|
|
This pull request introduces 10 alerts when merging 052e645 into 88c03ca - view on LGTM.com new alerts:
|
|
This pull request introduces 10 alerts when merging ee45261 into e4cc55d - view on LGTM.com new alerts:
|
|
This pull request introduces 10 alerts when merging 376d5f5 into 3bbfa17 - view on LGTM.com new alerts:
|
|
This pull request introduces 10 alerts when merging f1ba97b into 3bbfa17 - view on LGTM.com new alerts:
|
stelfrag
force-pushed
the
0_journal_v2
branch
from
3741718
to
fb8cb7f
Compare
|
This pull request introduces 10 alerts when merging fb8cb7f into 3bbfa17 - view on LGTM.com new alerts:
|
|
This pull request introduces 11 alerts when merging 66e28a1 into fced0fc - view on LGTM.com new alerts:
|
|
This pull request introduces 11 alerts when merging b643d8f into 0e3c365 - view on LGTM.com new alerts:
|
|
This pull request introduces 11 alerts when merging d0e2bae into 1ab62f3 - view on LGTM.com new alerts:
|
stelfrag
force-pushed
the
0_journal_v2
branch
from
a6811f0
to
cc48dee
Compare
|
This pull request introduces 11 alerts when merging cc48dee into f9ea1f2 - view on LGTM.com new alerts:
|
|
This pull request introduces 10 alerts when merging 1946c41 into 4dd4bd7 - view on LGTM.com new alerts:
|
|
This pull request introduces 8 alerts when merging 0f686cd into 4dd4bd7 - view on LGTM.com new alerts:
|
|
This pull request introduces 8 alerts when merging 46112ad into 4dd4bd7 - view on LGTM.com new alerts:
|
|
This pull request introduces 8 alerts when merging c968dd9 into 9e85dc2 - view on LGTM.com new alerts:
|
|
This pull request introduces 8 alerts when merging 7a49304 into df87a53 - view on LGTM.com new alerts:
|
|
This pull request introduces 1 alert when merging 7994128 into d15f36d - view on LGTM.com new alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog. |
…when replication commands are sent while there are others inflight
|
This pull request introduces 1 alert when merging ba1b2f7 into 558db52 - view on LGTM.com new alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog. |
|
This pull request introduces 1 alert when merging d4f72ea into 558db52 - view on LGTM.com new alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog. |
|
This pull request introduces 1 alert when merging 9635927 into 558db52 - view on LGTM.com new alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog. |
Add config parameter for detailed journal integrity check (Metric chain validation check during startup) pg cache insert drop check for existing page Fix crc calculation for metric headers
|
This pull request introduces 1 alert and fixes 1 when merging e4b51f2 into b4a0298 - view on LGTM.com new alerts:
fixed alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog. |
|
This pull request introduces 1 alert and fixes 1 when merging c8def16 into b4a0298 - view on LGTM.com new alerts:
fixed alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog. |
|
This pull request introduces 1 alert and fixes 1 when merging c1c1c20 into b4a0298 - view on LGTM.com new alerts:
fixed alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog. |
|
This pull request fixes 1 alert when merging 074113e into b4a0298 - view on LGTM.com fixed alerts:
Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog. |
Summary
The agent requires a lot of memory to index pages and how they map to the actual files that store metrics
File structure
The new file based index has a structure that allows quick access of the needed metadata. The file structure consists
of
During the agent start up, the journal replay only needs to create the necessary pages (unique metrics) which is very fast (initial tests indicate that is ~x100 faster than the current journal replay). This is aided by the fast that individual pages are not created in memory during startup but only when needed (during data queries).
Pages that are no longer needed (evicted from the cache) are removed. They will also be removed when unused for more than 10 minutes.
You can see the number of descriptors in memory under under netdata.dbengine_long_term_page_stats, journal v2 descriptors
Creation of new journal index files
When the agent starts it will check if a new index file exists for each journal file that needs to be processed. If it exists, it will use that instead. If the index file does not exist, it will replay the old journal file and use that information to create the new journal file and start using that immediately.
The agent will generated new index files for all journals except the last (active) one
New datafiles while the agent is running
When a new datafile / journal pair is created the agent will check and create a new journal index file for the journal that was just completed.
Known issues
Other fixes
This PR also fixes:
Test Plan