ZDP-Triage:

• relates to Allow configuring request timeout for InstallRequest #12793 in being a more long term solution => reduce size of chunks tranferred to not hit timeouts on slow networks

ZDP-Planning:

• already mitigated by Allow configuring request timeout for InstallRequest #12793
• Improve raft snapshot replication failure handling #11496 is more valuable to do

Notes:

The implementation of this will come in two parts the sender PR and the receiver PR.

The sender:
A ChunkedFileBasedSnapshotChunkReader will be created an will replace the current FileBasedSnapshotChunkReader however the chunk size designated will be Integer.MAX_VALUE so therefore no functional change should occur each chunk will still contain the full file. In addition the chunkName has been changed, previously this was the file name however since there could be multiple chunks per file the chunkName will now be appended with the file part e.g. file-1, file-2,file-3. As a result a fileName field has been added to the snapshot chunk implementation used instead of relying on string splits to extract (There is a fair amount of changes required to add the fileName filed maybe it would be simpler to rely on string split but this locks in the chunk naming scheme so a specific format so not ideal)

The receiver:

This involves adding support for chunks which only contain parts of files, currently correctness checks occur where file existence is taken as a failure however with chunked file snapshots this no longer holds these checks will be removed or refactored. In addition there is the question of checksums, before each chunk was a file and the file snapshot was provided. Currently the new implementation would involve snapshots of the file chunks. I think this is a fine solution, if we know all the file parts are not corrupted the full file should be correct? Open to thoughts on this. Lastly there will need to be changes to how files are written per chunk as now a chunk could represent not just a new file to create and write but also appending content to a file.

Have you also looked into how to make these changes with out breaking rolling update?

Have you also looked into how to make these changes with out breaking rolling update?

Yes changing chunkName to be a chunk identifier and adding a fileName field will break rolling update. Instead keep chunkName to be the fileName (not a very apt name this can be changed in a later version). and add a field called chunkIdentifier? Which has format file_name-<file_part> for use during debugging (Do you think this field is even needed?)