Bloating data and orioledb_data on s3 with cluster start-stops

[pashkinelfe]

I've initialized cluster with 10K partitions table, inserted around 6Gb tuples then did cluster start-stops without any modifications and got gradual increase of S3 directory size from 11.8Gb 174K files to 25Gb 475K files over 16 start-stops.

Directory structure of s3 (after 16 start-stops):

Local pgdata catalog size grew only insignificantly 5.9 to 6.2 Gb

Stop and start time first time were: 7m0.841s (stop) and 3m7.716 (start)
Next 15 times they stabilized at around ~1:45 (stop) and ~2:05 (start)

Checked at be17f3a (due to segfaults #333 at 234115c)

[pashkinelfe]

at 02bc078
Bucket size increased from 18.1Gb to 26.1 GB (1.4x) over 16 start-stops with the number of files 180000

I.e. increase of s3 bucket is lower than previously (1.4x vs 2.1x) , but exists with just start-stops (no inserts)

[pashkinelfe]

at 9c34d00
Bucket size after inserts

data/ 5.1 GB / 1209 files
orioledb_data/ 4.0 GB / 100233 files
wal/ 5.2 GB / 335 files

After 16 start-stops:

data/  13.9 GB | 4790 files
orioledb_data/  4.4 GB | 120474 files
wal/ 5.5 GB | 352 files

/data contains subdirs 1...30. orioledb_undo occupies most in each of them (~300Mb each)

[akorotkov]

@pashkinelfe, please recheck on b225ece

[pashkinelfe]

Rechecked. orioledb_undo bloating disappeared.
Also, it's very nice that cluster start and stop times decreased significantly.
First cluster stop ~50s, then start ~10s, then all start-stops (with no data inserted between) ~2s.

Now /data/NN contents on s3 contains ~180Mb per checkpoint. (was 600Mb/checkpoint)

It consists of two parts:
(1) orioledb_data/small_files_* (90Mb)
they are not completely similar in different checkpoints, but diff is quite small (two sample files 10 checkpoints apart are attached)
small_files_89_1.txt
small_files_89.txt

(2) base/ (90 Mb)
looks like identical data in each checkpoints

[pashkinelfe]

On s3 bucket dir structure:

• /data//orioledb_undo is empty (maybe doesn't need to be created)
• Orioledb part of checkpoint is split into /data//orioledb_data (1) and /orioledb_data/ (2) (maybe move small_files, control and xid into (2) and delete (1)