added new config for job.changelog.system #31
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
navina
approved these changes
Dec 13, 2016
nickpan47
reviewed
Dec 13, 2016
| && changelogSystem.isDefined) { | ||
| // get the system name | ||
| Some(changelogSystem.get + "." + systemStream) | ||
| } else { |
There was a problem hiding this comment.
Shouldn't it error out if systemStream only has stream name and changelogSystem is not defined?
There was a problem hiding this comment.
that was current behavior. I will add this check.
| def testIsChangelogSystemSetting { | ||
| val configMap = Map[String, String]( | ||
| FACTORY.format("store1") -> "some.factory.Class", | ||
| CHANGELOG_STREAM.format("store1") -> "system1.stream1", |
There was a problem hiding this comment.
Also, with this change, I think that we will need to update our internal config rewriter as well, to detect all systems used by the job.
There was a problem hiding this comment.
Good point, but I think this change is not required in the open source.
navina
reviewed
Dec 13, 2016
| // these values will be combined into <asystem>.<astream> | ||
| val systemStream = getOption(CHANGELOG_STREAM format name) | ||
| val changelogSystem = getOption(CHANGELOG_SYSTEM) | ||
| val systemStreamRes = if ( systemStream.isDefined |
There was a problem hiding this comment.
May be you can simplify this if-else statement as :
if (systemStream.isDefined && !systemStream.getOrElse("").contains(".")) {
if (changelogSystem.isDefined) {
Some(changelogSystem.get + "." + systemStream.get)
} else {
throw new SamzaException(...)
}
} else {
...
}
navina
reviewed
Dec 13, 2016
fredji97
pushed a commit
to fredji97/samza
that referenced
this pull request
May 11, 2018
As per subject, it's a demo of stream-to-table join using RocksDb. Author: Wei Song <wsong@linkedin.com> Reviewers: Prateek Maheshwari <pmaheshwari@apache.org> Closes apache#31 from weisong44/latest
khandelwal-ayush
pushed a commit
to khandelwal-ayush/samza
that referenced
this pull request
Jun 12, 2024
* SAMZA-2797: Call flush during stop from CoordinatorStreamWriter (apache#1692) * SAMZA-2798: Populate worker.opts in environment variable only if available (apache#1693) Description Populate worker.opts in the environment variable only if available in the configs. Changes Check if worker.opts is present and then add it to environment variable Tests Updated unit tests * Add MAX_BACKGROUND_JOBS config for RocksDB (apache#1694) * SAMZA-2784: Remove excessive commit logs (apache#1695) * SAMZA-2799: Remove worker.opts handling in shell command builder (apache#1696) --------- Co-authored-by: ajo thomas <ajo.thomas24@gmail.com> Co-authored-by: Bharath Kumarasubramanian <bharathkk@apache.org> Co-authored-by: Shekhar Sharma <72765053+shekhars-li@users.noreply.github.com> Co-authored-by: Daniel Chen <xrchen@uwaterloo.ca>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SAMZA-1060.
Allow to specify a changelog system separately, so user can only specify stream name for each store.
If user specifies both (system and stream) it overwrites the job.changelog.system setting.