SAMZA-2703: Add input stream partition count and input stream regex monitoring for refreshing job model#1548
Conversation
…r refreshing job model
mynameborat
left a comment
There was a problem hiding this comment.
Looks good to me. Few minor comments.
samza-core/src/main/java/org/apache/samza/config/JobCoordinatorConfig.java
Outdated
Show resolved
Hide resolved
samza-core/src/main/java/org/apache/samza/coordinator/lifecycle/JobRestartSignal.java
Outdated
Show resolved
Hide resolved
...ore/src/main/java/org/apache/samza/coordinator/lifecycle/JobRestartSignalFactoryContext.java
Show resolved
Hide resolved
mynameborat
left a comment
There was a problem hiding this comment.
Feel free to commit once the travis build pass.
Looks good to me.
|
Is the |
That test doesn't use this new job coordinator flow, so it is unrelated to this change. It didn't fail on the github actions CI build, so it is a flaky test. I'm not sure about the root cause of the flakiness, but I saw that there was a validation bug in that test which would have helped to debug it, so I fixed that (#1549). I'll investigate more if it comes up again. |
Feature: For
StaticResourceJobCoordinator, trigger job restart when input stream partition count changes or input stream regex set changes.Changes:
JobRestartSignalinterface for triggering a job restart.StreamPartitionCountMonitorandStreamRegexMonitortoStaticResourceJobCoordinatorfor triggering aJobRestartSignalwhen input stream partition count changes or input stream regex set changes.Tests:
Added some code into
StreamPartitionCountMonitorwhich triggered the callback after 5 minutes and verified that the placeholderNoOpJobRestartSignalwas called.API changes (all backwards compatible):
Configure
job.coordinator.restart.signal.factoryto specify an implementation ofJobRestartSignalFactoryto use.