New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Estimate partitions number basing on a desired partition size #2289

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
2 participants
@pomadchin
Member

pomadchin commented Jul 19, 2017

This PR also introduces a new default S3GeoTiffRDD behaviour.
The question is: should we change the default reader behaviour? Or it's still should be an optional feature.

@pomadchin pomadchin requested a review from echeipesh Jul 19, 2017

@pomadchin pomadchin force-pushed the pomadchin:feature/s3geotiffrdd-opt branch 2 times, most recently from 26b5618 to c7a012c Jul 19, 2017

Added S3 metadata reader to estimate partitions number basing on a de…
…sired partition size in bytes.

Signed-off-by: Grisha Pomadchin <gr.pomadchin@gmail.com>

@pomadchin pomadchin force-pushed the pomadchin:feature/s3geotiffrdd-opt branch from c7a012c to a44f158 Jul 19, 2017

@echeipesh

This comment has been minimized.

Contributor

echeipesh commented Jul 25, 2017

To answer the question: The default behavior on principle should "just work" as much as possible.

That is not the case currently, we pretty much require partition and window size tuning before we can expect good performance. So in such a case the defaults can be changed to provide the correct behavior.

@pomadchin

This comment has been minimized.

Member

pomadchin commented Aug 2, 2017

Fixed in #2296

@pomadchin pomadchin closed this Aug 2, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment