This repository has been archived by the owner on Jan 20, 2022. It is now read-only.
Refactors the final flat map a bit #539
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -73,7 +75,7 @@ class Summer[Key, Value: Semigroup, Event, S, D, RC]( | |||
| lazy val storePromise = Promise[Mergeable[Key, Value]] | |||
| lazy val store = Await.result(storePromise) | |||
|
|
|||
| lazy val sSummer: AsyncSummer[(Key, (List[InputState[S]], Value)), Map[Key, (List[InputState[S]], Value)]] = summerBuilder.getSummer[Key, (List[InputState[S]], Value)](implicitly[Semigroup[(List[InputState[S]], Value)]]) | |||
| lazy val sSummer: AsyncSummer[(Key, (Seq[InputState[S]], Value)), Map[Key, (Seq[InputState[S]], Value)]] = summerBuilder.getSummer[Key, (Seq[InputState[S]], Value)](implicitly[Semigroup[(Seq[InputState[S]], Value)]]) | |||
There was a problem hiding this comment.
should this not be CMap?
There was a problem hiding this comment.
No, we unpack the CMap on entry to the summer. So they are gone by this point.
|
bump on this, think i've done all the comments? |
|
LGTM |
ianoc
added a commit
that referenced
this pull request
Aug 29, 2014
Refactors the final flat map a bit
snoble
pushed a commit
to snoble/summingbird
that referenced
this pull request
Sep 8, 2017
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Made some lists Seq's since we don't rely on it being a list.
Fix a case where the hash code could be less than zero for the batching, would cause the space to double in size. Not a correctness bug thankfully, but good to know how settings are effecting things.
I've pulled the batching with this hash outside the aggregator, which lets the aggregator use sumOption to much better effect. Also now the aggregator will get better visibility on things. Does not seem to have impacted any performance negatively in tests.
Added a rate limiter option, which allows a slow start + min/max throughputs per second for the bolt. This is useful since we lack back pressure on storm to be able to stick a cap on the flat map throughput.