[BEAM-6994] SamzaRunner: further improvements for upgrading Samza #8221
Conversation
|
|
||
| final StateInfo stateInfo = getStateInfo(); | ||
|
|
||
| if (listener != null && (stateInfo.state == State.DONE || stateInfo.state == State.FAILED)) { |
There was a problem hiding this comment.
I'm trying to understand when would we hit this while state is neither Done nor FAILED. What is being protected by having this check?
There was a problem hiding this comment.
Oh, the status can be RUNNING, since it can be a timed waifForFinish().
| private final SystemReduceFn<K, InputT, ?, OutputT, BoundedWindow> reduceFn; | ||
| private final String stepName; | ||
| private final String stepId; | ||
| private final PCollection.IsBounded isBounded; |
There was a problem hiding this comment.
import PCollection.IsBounded directly?
| configBuilder.put("job.host-affinity.enabled", "true"); | ||
| } | ||
|
|
||
| switch (options.getSamzaExecutionEnvironment()) { |
There was a problem hiding this comment.
not sure whether we have logged the entire pipeline option somewhere. Otherwise it would be good to have a log entry here about which execution environment we're running
There was a problem hiding this comment.
Makes sense. Added more logs.
|
Run JavaPortabilityApi PreCommit |
|
Run Python PreCommit |
|
Run Website PreCommit |
1 similar comment
|
Run Website PreCommit |
|
|
||
| repositories { | ||
| maven { | ||
| url "https://artifactory.corp.linkedin.com:8083/artifactory/DDS/" |
There was a problem hiding this comment.
Out of curiosity: What is this used for?
There was a problem hiding this comment.
Good catch! That was for internal linkedin testing of Samza builds. Let me get rid of it. Thanks!
3 participants
More improvements for SamzaRunner:
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.