PostgreSQL - Simple Composer for reduced memory usage

[krnaveen14]

Database name

PostgreSQL

Pull request description

Simple Composer for reduced memory usage on instances with > 100k files

• Does not track metadata of files backed up (Files and TarFileSets)
• Does not support WALG_DELTA_MAX_STEPS, WALG_DELTA_ORIGIN, WALG_USE_REVERSE_UNPACK, WALG_SKIP_REDUNDANT_TARS, catchup-push

Describe what this PR fix

Similar to Issue #738 , WAL-G consumes huge memory on backing up instance with millions of files. This is due to tracking the metadata of each file, maintaining the list of files in each TarFileSet and finally marshalling the data structure to text for creating sentinel.json file. Even though PR #740 tries to address the marshalling part by streaming / pipe it, significant memory usage on BackupSentinelDto data structure still persists.

Based on the BackupSentinelDto usage, it is needed only for configurations such as DELTA, INCREMENTAL, RATING_COMPOSER, catchup-push etc... Since we don't use any of the above mentioned configs, Files and TarFileSets data in BackupSentinelDto simply unused.

To reduce the memory usage of WAL-G on instances with millions of files and to make it available & configurable for the community as whole, SimpleTarBallComposer is implemented which does not track any of the file metadata, list of files in each TarFileSet and the list of TarFileSets itself.

Simple Composer is supported on both local and remote backups.

Usage : wal-g backup-push [path] --simple-composer or WALG_USE_SIMPLE_COMPOSER=true environment variable

Please provide steps to reproduce

Create 500+ Databases with 1000+ Tables in each Database (not necessary to insert any data). Execute WAL-G backup-push to see the memory usage of wal-g gradually increases over time and failing with OOM after exhausting available memory (10GB in our case).

[krnaveen14]

@usernamedt waiting on approval for test execution and expecting code review suggestions

[usernamedt]

Hi!

This is definitely a cool feature, but it needs some adjustments.

[usernamedt]

Hi!

Looks like this unit test is failing:

2021-08-17T11:25:48.0366780Z === RUN   TestGetFilesToUnwrap_UtilityFiles
2021-08-17T11:25:48.0367313Z     backup_test.go:49: 
2021-08-17T11:25:48.0367713Z         	Error Trace:	backup_test.go:49
2021-08-17T11:25:48.0378206Z         	Error:      	Not equal: 
2021-08-17T11:25:48.0378808Z         	            	expected: map[string]bool{"/global/pg_control":true, "backup_label":true, "tablespace_map":true}
2021-08-17T11:25:48.0379442Z         	            	actual  : map[string]bool(nil)
2021-08-17T11:25:48.0379793Z         	            	
2021-08-17T11:25:48.0380100Z         	            	Diff:
2021-08-17T11:25:48.0380664Z         	            	--- Expected
2021-08-17T11:25:48.0381016Z         	            	+++ Actual
2021-08-17T11:25:48.0381447Z         	            	@@ -1,6 +1,2 @@
2021-08-17T11:25:48.0381949Z         	            	-(map[string]bool) (len=3) {
2021-08-17T11:25:48.0382567Z         	            	- (string) (len=18) "/global/pg_control": (bool) true,
2021-08-17T11:25:48.0383204Z         	            	- (string) (len=12) "backup_label": (bool) true,
2021-08-17T11:25:48.0383852Z         	            	- (string) (len=14) "tablespace_map": (bool) true
2021-08-17T11:25:48.0384349Z         	            	-}
2021-08-17T11:25:48.0384701Z         	            	+(map[string]bool) <nil>
2021-08-17T11:25:48.0385035Z         	            	 
2021-08-17T11:25:48.0385506Z         	Test:       	TestGetFilesToUnwrap_UtilityFiles
2021-08-17T11:25:48.0386267Z --- FAIL: TestGetFilesToUnwrap_UtilityFiles (0.00s)
2021-08-17T11:25:48.0386909Z === RUN   TestGetFilesToUnwrap_NoMoreFiles
2021-08-17T11:25:48.0387661Z --- PASS: TestGetFilesToUnwrap_NoMoreFiles (0.00s)

[usernamedt]

Seems OK, but need to change the flag

[krnaveen14]

Closing this in favor new PR #1101