PostgreSQL - Reduce memory usage by not tracking metadata of the files backed up #1101
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
krnaveen14
force-pushed
the
reduce-memory-usage
branch
3 times, most recently
from
de8a0b9
to
9e641b1
Compare
|
@usernamedt I feel naming of |
usernamedt
reviewed
Sep 29, 2021
|
@usernamedt Can you review and approve the workflows? |
@usernamedt A Gentle Reminder! |
Hi! Currently, we plan to merge this PR first and then proceed to this one. |
|
Hi! We merged the #1114, can you please rebase this PR onto it? |
Sure |
…s backed up By default, WAL-G tracks metadata of the files backed up. If millions of files are backed up (typically in case of hundreds of databases and thousands of tables in each database), tracking this metadata alone would require GBs of memory. If `--reduce-memory-usage` or `WALG_REDUCE_MEMORY_USAGE` is enabled, WAL-G does not track metadata of the files backed up. This significantly reduces the memory usage on instances with `> 100k` files. Limitations * Cannot be used with `rating-composer`, `copy-composer` * Cannot be used with `delta-from-user-data`, `delta-from-name`, `add-user-data`
…hout-files-metadata (WALG_WITHOUT_FILES_METADATA)
usernamedt
force-pushed
the
reduce-memory-usage
branch
from
4859943
to
70260bc
Compare
usernamedt
reviewed
Nov 24, 2021
| @@ -62,9 +62,8 @@ func chooseTablespaceSpecification(sentinelDtoSpec, spec *TablespaceSpec) *Table | |||
|
|
|||
| // TODO : unit tests | |||
| // deltaFetchRecursion function composes Backup object and recursively searches for necessary base backup | |||
| func deltaFetchRecursionOld(backupName string, folder storage.Folder, dbDataDirectory string, | |||
| func deltaFetchRecursionOld(backup Backup, folder storage.Folder, dbDataDirectory string, | |||
There was a problem hiding this comment.
This change is not related to the files metadata uploading, but helps to avoid downloading it twice.
x4m
approved these changes
Nov 24, 2021
usernamedt
force-pushed
the
reduce-memory-usage
branch
from
70260bc
to
b7abd11
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Database name
PostgreSQL
Pull request description
Similar to Issue #738 , WAL-G consumes huge memory on backing up instance with millions of files. This is due to tracking the metadata of each file, maintaining the list of files in each TarFileSet and finally marshalling the data structure to text for creating
sentinel.jsonfile. Even though PR #740 tries to address the marshalling part by streaming / pipe it, significant memory usage onBackupSentinelDtodata structure still persists.Based on the
BackupSentinelDtousage, it is needed only for configurations such asDELTA,INCREMENTAL,RATING_COMPOSER,catchup-pushetc... Since we don't use any of the above mentioned configs,FilesandTarFileSetsdata inBackupSentinelDtosimply unused.To reduce the memory usage of WAL-G on instances with millions of files and to make it available & configurable for the community as whole,
NopTarBallComposer&NopBundleFilesis implemented which does not track any of the file metadata, list of files in each TarFileSet and the list of TarFileSets itself.reduce-memory-usageis supported on both local and remote backups.Usage :
wal-g backup-push [path] --reduce-memory-usageorWALG_REDUCE_MEMORY_USAGE=trueenvironment variable( Successor to PR #1037 )
Please provide steps to reproduce (if it's a bug)
Create 500+ Databases with 1000+ Tables in each Database (not necessary to insert any data). Execute WAL-G
backup-pushto see the memory usage ofwal-ggradually increases over time and failing with OOM after exhausting available memory (10GB in our case).