restore: limit number of files restore span entries #119785
Labels
C-bug
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
O-postmortem
Originated from a Postmortem action item.
O-support
Originated from a customer
P-1
Issues/test failures with a fix SLA of 1 month
T-disaster-recovery
Projects
Today we size restore span entries solely based on the sum size of the files grouped into that entry, to hit the target size (384mb by default).
However, if a backup contains many small files, this can mean that a single span, to hit its target size, includes many files. Since all files in a span are allowed to overlap, we are obligated to open all of them to merge them. This, however, combines poorly with the above heuristic that can put a very large number (hundreds or thousands) of files into one span, as the restore processor then needs to open thousands of files concurrently to process that span.
We should have an additional secondary target limit on the number of files (64? 128?) in a span when constructing these spans.
Jira issue: CRDB-36313
The text was updated successfully, but these errors were encountered: