SAMZA-2499: Add submission config file support#1336
Merged
Conversation
Feature: https://cwiki.apache.org/confluence/display/SAMZA/SEP-23:+Simplify+Job+Runner Changes: 1. Add config-path option in ApplicationRunnerCommandLine. 2. Update loadConfig() to read property file specified by config-path as submission config. Tests: Followed https://samza.apache.org/startup/hello-samza/latest/ to test against Hello Samza example: 1. Update wikipedia-feed.properties to include the following properties: job.config.loader.factory=org.apache.samza.config.loaders.PropertiesConfigLoaderFactory job.config.loader.properties.path=./__package/config/wikipedia-feed.properties 2. Deploy the job: deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/wikipedia-feed.properties API Changes: 1. Add --config-path option support when running run-app.sh Upgrade Instructions: None Usage Instructions: Alternative way when submitting job, deploy/samza/bin/run-app.sh \ --config app.class=A \ --config job.stream.factory=B \ --config yarn.package.path=C \ --config job.config.loader.factory=D \ --config job.config.loader.properties.xx=E \ --config job.name=F can be simplied to deploy/samza/bin/run-app.sh --config-path=/path/to/submission/properties/file/submission.properties where submission.properties contains app.class=A job.stream.factory=B yarn.package.path=C job.config.loader.factory=D job.config.loader.properties.xx=E job.name=F
cameronlee314
approved these changes
Apr 2, 2020
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature:
https://cwiki.apache.org/confluence/display/SAMZA/SEP-23:+Simplify+Job+Runner
Changes:
Tests:
Followed https://samza.apache.org/startup/hello-samza/latest/ to test against Hello Samza example:
job.config.loader.factory=org.apache.samza.config.loaders.PropertiesConfigLoaderFactory
job.config.loader.properties.path=./__package/config/wikipedia-feed.properties
deploy/samza/bin/run-app.sh --config-path=$PWD/deploy/samza/config/wikipedia-feed.properties
API Changes:
Upgrade Instructions:
None
Usage Instructions:
Alternative way when submitting job,
deploy/samza/bin/run-app.sh
--config app.class=A
--config job.stream.factory=B
--config yarn.package.path=C
--config job.config.loader.factory=D
--config job.config.loader.properties.xx=E
--config job.name=F
can be simplied to
deploy/samza/bin/run-app.sh --config-path=/path/to/submission/properties/file/submission.properties
where submission.properties contains
app.class=A
job.stream.factory=B
yarn.package.path=C
job.config.loader.factory=D
job.config.loader.properties.xx=E
job.name=F