New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3InputFormat does not filter according to extensions when partitionCount is used #2231

Closed
lossyrob opened this Issue Jun 9, 2017 · 0 comments

Comments

Projects
None yet
1 participant
@lossyrob
Member

lossyrob commented Jun 9, 2017

This logic correctly filters out keys by extensions:

.filter { obj =>
if (extensions.isEmpty)
true
else {
val key = obj.getKey
extensions.map(key.endsWith).reduce(_ || _)
}
}

That only executes if (null == partitionCountConf). If there is a partition count, that logic is missing:

.filter(! _.getKey.endsWith("/"))
.toVector

@lossyrob lossyrob added the bug label Jun 9, 2017

@lossyrob lossyrob added this to the 1.1 milestone Jun 9, 2017

lossyrob added a commit to lossyrob/geotrellis that referenced this issue Jun 9, 2017

@lossyrob lossyrob self-assigned this Jun 9, 2017

@lossyrob lossyrob added the in progress label Jun 9, 2017

lossyrob added a commit to lossyrob/geotrellis that referenced this issue Jun 22, 2017

lossyrob added a commit that referenced this issue Jun 22, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment