added map_as_series

[jwittenbach]

Adds a Image.map_as_series method that uses Blocks to apply a function to each series in an Images object and then turn the data back into an Images object -- avoids needing to transform the data all the way to a Series representation, which can be quite expensive to turn back into Images due to the high level of fragmentation that can occur when the total size of the spatial dimensions greatly outnumbers the size of the temporal dimension.

[d-v-b]

This sounds great, any estimate on the performance boost over Images -> Series -> Images?

[jwittenbach]

@d-v-b from my anecdotal experience, it can be the difference between the job completely failing and being able to run to completion!

We decided to go this route rather than implementing a stand-alone thunder-movie package.

[freeman-lab]

Would be really cool to report at least one benchmark alongside this change, just pick some fairly big representative workflow and time the two methods. though obviously not so big that the older method fails.

[jwittenbach]

tests pass...merging

I'll get back with some stats on how this performs compared to Images.toseries().map(f).toimages()

[jwittenbach]

@d-v-b @freeman-lab @sofroniewn

Preliminary analysis using a 20 node cluster (19 workers): https://gist.github.com/jwittenbach/dca311743395d904c3d7

The last cell is still running...25 minutes later

[jwittenbach]

Also bumped number of nodes up to 40 (39 workers). The new map_as_series approach shows a linear speed up (2x nodes => 2x speed). The old approach still stalls out and never completes.

[sofroniewn]

Nice, that's great!

[d-v-b]

Nice, I hope we can kiss those hanging stages goodbye