Skip to content

SAMZA-2703: Add input stream partition count and input stream regex monitoring for refreshing job model#1548

Merged
cameronlee314 merged 4 commits intoapache:masterfrom
cameronlee314:dynamicjc
Oct 27, 2021
Merged

SAMZA-2703: Add input stream partition count and input stream regex monitoring for refreshing job model#1548
cameronlee314 merged 4 commits intoapache:masterfrom
cameronlee314:dynamicjc

Conversation

@cameronlee314
Copy link
Contributor

Feature: For StaticResourceJobCoordinator, trigger job restart when input stream partition count changes or input stream regex set changes.

Changes:

  1. Add pluggable JobRestartSignal interface for triggering a job restart.
  2. Add StreamPartitionCountMonitor and StreamRegexMonitor to StaticResourceJobCoordinator for triggering a JobRestartSignal when input stream partition count changes or input stream regex set changes.

Tests:
Added some code into StreamPartitionCountMonitor which triggered the callback after 5 minutes and verified that the placeholder NoOpJobRestartSignal was called.

API changes (all backwards compatible):
Configure job.coordinator.restart.signal.factory to specify an implementation of JobRestartSignalFactory to use.

Copy link
Contributor

@mynameborat mynameborat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Few minor comments.

@cameronlee314 cameronlee314 changed the title Add input stream partition count and input stream regex monitoring for refreshing job model SAMZA-2703: Add input stream partition count and input stream regex monitoring for refreshing job model Oct 26, 2021
Copy link
Contributor

@mynameborat mynameborat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to commit once the travis build pass.
Looks good to me.

@mynameborat
Copy link
Contributor

Is the KafkaCheckpointIntegrationTest failure related to your change? As in, it is expecting one commit but got 2.

@cameronlee314
Copy link
Contributor Author

Is the KafkaCheckpointIntegrationTest failure related to your change? As in, it is expecting one commit but got 2.

That test doesn't use this new job coordinator flow, so it is unrelated to this change. It didn't fail on the github actions CI build, so it is a flaky test. I'm not sure about the root cause of the flakiness, but I saw that there was a validation bug in that test which would have helped to debug it, so I fixed that (#1549). I'll investigate more if it comes up again.

@cameronlee314 cameronlee314 merged commit 1348d51 into apache:master Oct 27, 2021
@cameronlee314 cameronlee314 deleted the dynamicjc branch November 17, 2021 23:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants