-
Notifications
You must be signed in to change notification settings - Fork 267
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you update the description of the PR to indicate how including IdentityKeyedProducer in options calculation is an issue and why removing that is the right fix for this issue.
Options.getFirst[T](options, stormDag.producerToPriorityNames(producer)) | ||
} | ||
node.members | ||
.find(!_.isInstanceOf[IdentityKeyedProducer[_, _, _]]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we comment here why this is required?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed a way I fixed the issue. Turned out way with ignoring IdentityKeyedProducer
s caused more bugs than fixed.
Codecov Report
@@ Coverage Diff @@
## develop #739 +/- ##
==========================================
+ Coverage 71.88% 72.18% +0.3%
==========================================
Files 153 153
Lines 3735 3736 +1
Branches 202 204 +2
==========================================
+ Hits 2685 2697 +12
+ Misses 1050 1039 -11
Continue to review full report at Codecov.
|
👍 |
Currently if you have
flatMap().name('flatMap').sumByKey().name('sumByKey')
and you have someSummer
options on bothflatMap
andsumByKey
for both of nodes (for partial aggregation onflatMap
and for final aggregation onsumByKey
) only options fromflatMap
will be applied.Root cause of this is in a way we get options for
Node
s - we get options for first producer in producers chain corresponding toNode
. In case ofSummerNode
members contains twoProducer
s -IdentityKeyedProducer
andSummer
.This PR fixes that by passing
Producer
instance toBuildSummer
which is used to get options for summer.