Changed the way backups stored in cloud. #31
Conversation
xbcloud store each chink as separate object with name database/table.ibd.NNNNNNNNNNNNNNNNNNNN, where NNN... is 0-padded serial number of chunk within file. Size of chunk produced by xtrabackup and xbstream changed to 10M. Each chunk read fully from input stream into the buffer. Then it uploaded via HTTP. If input chunk is corrupted, then upload aborted and partially uploaded backup will remain in cloud storage. Following has been implemented along the way: 1. xbcloud refuses to overwrite the backup with the same name. 2. added ability to download only selected files.
|
+1 - I can't really comment much on the protocol interactions correctness as I am pretty ignorant of the real format, but the remainder of things look good to me. The only thing I would bring up is the larger hard coded value of page to read in a chunk. At one point in the past, we had talked about allowing this to be a configurable option. In some tests, a much larger value proved to improve overall backup performance and in others, a smaller value was desired for other reasons. If you start getting into an XB w/ multiple threads + compression + encryption + xbstream + xbcloud, the amount of memory required as a direct result of the size of this buffer can grow to be pretty big due to the fact that IIRC there is one instance of this buffer per thread, then another buffer of this size for each encryption and compression thread and I think another for xbstream, then more in xbcoud. I do not remember though why we abandoned the idea, maybe just not enough demand or lost in the noise of other work. So I would maybe suggest adding another option to allow changing the default size of this block (in pages) as part of a new feature. |
Changed the way backups stored in cloud.
xbcloud store each chink as separate object with name
backup_name/database/table.ibd.NNNNNNNNNNNNNNNNNNNN,
where NNN... is 0-padded serial number of chunk within file.
Size of chunk produced by xtrabackup and xbstream changed
to 10M.
Each chunk read fully from input stream into the buffer. Then it
uploaded via HTTP. If input chunk is corrupted, then upload aborted
and partially uploaded backup will remain in cloud storage.
Following has been implemented along the way:
jenkins: http://jenkins.percona.com/view/PXB%202.3/job/percona-xtrabackup-2.3-param/62/
Jenkins currently can not test xbcloud. I am checking the status of the task to set up the swift for testing with Jenkins.