Sync should read "batches" of chunks at each level using `ReadManyValues` #3182

ghost · 2017-02-11T00:24:27Z

So as to maximize read perf from nbs (by maximizing usage of chunk locality)

In an NBS world, bulk 'has' checks are waaaay cheaper than they used to be. In light of this, we can toss out the complex logic we were using in Pull() -- which basically existed for no reason other than to avoid doing 'has' checks. Now, the code basically just descends down a tree of chunks breadth-first, using HasMany() at each level to figure out which chunks are not yet in the sink all at once, and GetMany() to pull them from the source in bulk. Fixes attic-labs#3182, Towards attic-labs#3384

In an NBS world, bulk 'has' checks are waaaay cheaper than they used to be. In light of this, we can toss out the complex logic we were using in Pull() -- which basically existed for no reason other than to avoid doing 'has' checks. Now, the code basically just descends down a tree of chunks breadth-first, using HasMany() at each level to figure out which chunks are not yet in the sink all at once, and GetMany() to pull them from the source in bulk. Fixes #3182, Towards #3384

ghost added P0 Sync labels Feb 11, 2017

ghost mentioned this issue Feb 11, 2017

Improve Sync #3177

Open

cmasone-attic mentioned this issue May 22, 2017

Simplify Pull() #3490

Merged

cmasone-attic closed this as completed in #3490 May 22, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sync should read "batches" of chunks at each level using `ReadManyValues` #3182

Sync should read "batches" of chunks at each level using `ReadManyValues` #3182

ghost commented Feb 11, 2017

Sync should read "batches" of chunks at each level using ReadManyValues #3182

Sync should read "batches" of chunks at each level using ReadManyValues #3182

Comments

ghost commented Feb 11, 2017

Sync should read "batches" of chunks at each level using `ReadManyValues` #3182

Sync should read "batches" of chunks at each level using `ReadManyValues` #3182