Parallel copy

[rclark]

Allows you to split a read operation into an arbitrary number of jobs. Pass a job parameter to options when using tilelive.createReadStream or tilelive.deserialize:

var readable = tilelive.createReadStream(src, { type: 'scanline', job: { total: 4, num: 1 } });

This instructs tilelive to only read tiles that would fall into job 1 of 4. A complete read would mean four calls each with a different num.

Still to-do:

• deserialize shouldn't utilize the same x % total === num - 1 approach that other streams do. That means deserializing every row before throwing out the ones that aren't part of the current job. It should skip based on row number instead. Done.
• other ideas for tests?

[coveralls]

Coverage increased (+0.15%) when pulling ef301a6 on parallel-copy into 580d44f on master.

[rclark]

In the failing tests, as the ratio of num jobs : total tiles increases the distribution of tile reads across jobs gets very bad very quickly.

Because of the approach here (dividing jobs based on tile.x values) getting an even distribution is going to be very tileset-dependent. I think the next step is to look at this distribution in some more true-to-life situations to determine if it is a reasonable approach.

[yhahn]

@rclark I'm not too concerned about even distribution across jobs. Any approach (bbox, etc) is going to be datasource dependent unless the approach involves knowing the geographic shape/density of the data beforehand.

I'll keep pondering the pyramid question tonight.

[rclark]

The horrible distribution I was seeing is because the modulus approach will never put tiles into jobs where job num > max tile.x. Weird implication: the further west your copy operation is, the less you can benefit from parallelization.

[rclark]

@yhahn I managed to write stream-pyramid.js such that:

• Tiles in zoom levels where num tiles < num jobs are all fed to job 1
• modulus-splitting occurs on tiles at the zoom level where num tiles >= num jobs. Each job gets a set of these tiles and renders out its children pyramid-style.

The logic is a mess and I am not proud of this.

[rclark]

Okay @yhahn I fell back on straight bbox-splitting with low-zoom and along-the-boundaries duplication of rendered tiles. The logic is certainly cleaner.

[yhahn]

@rclark 👍

[rclark]

Now it doesn't mod-split the deserialization stream based on stream order, but by pulling the X value out of the serialized data via regex. JSON.parse (without decoding the buffer) is noticeably costly when running hundreds of jobs.

I tried to set it up in an abstracted-enough way that it should be clear what would have to be done if you wanted to change serialization formats.

[yhahn]

@rclark 👍 want to merge + roll 5.3.0 or so?