Allow PartitionSource to limit the number of open files #1078
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…partitioned source
| @@ -35,7 +35,7 @@ import cascading.tuple.Fields | |||
| /** | |||
| * This is a base class for partition-based output sources | |||
| */ | |||
| abstract class PartitionSource extends SchemedSource { | |||
| abstract class PartitionSource(val openWritesThreshold: Int = -1) extends SchemedSource { | |||
There was a problem hiding this comment.
This should be an option, not defaulting to -1
…s can specify a threshold less than or equal to zero. There doesn't seem to be any error checking in Cascading that forbids this, but the end result is that it's as if they passed in a threshold of 1.
|
Thanks @ianoc! I've incorporated your changes with my latest commit. |
|
Looks like the tests fail with this at the moment |
|
Oops, my bad @ianoc. Should be fixed now... |
|
@ianoc Build/tests just completed successfully. See: https://travis-ci.org/twitter/scalding/builds/38880158 |
|
Looks good, thanks for the contribution! |
ianoc
added a commit
that referenced
this pull request
Oct 24, 2014
Allow PartitionSource to limit the number of open files
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Without this new parameter, partitioned sources always defaulted to a limit of 300 open files. This change should maintain the same default behavior while allowing a little more flexibility.