Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[NEMO-294] Beam-Runner #163

Merged
merged 9 commits into from Dec 3, 2018
Merged

[NEMO-294] Beam-Runner #163

merged 9 commits into from Dec 3, 2018

Conversation

wonook
Copy link
Member

@wonook wonook commented Nov 27, 2018

JIRA: NEMO-294: Support Nemo Runner execution by providing PipelineOptions to the Beam program

Major changes:

  • Update launch scripts to launch shaded jars with the current project version, instead of 0.1-SNAPSHOT
  • Refactor JobLauncher so that jobs can be launched from the JobLauncher's main class as well as from a particular Beam application. This also changes so that the job ID can be received from the application itself.
  • Use Google's AutoService from Beam to register NemoRunner as one of its runners with the NemoRunnerRegistrar.
  • Rename NemoPipelineRunner into NemoRunner to follow conventions.
  • Instead of having a default executor_json file, put in a default JSON string instead, so that it doesn't require a JSON file by default.
  • Add a MinimalWordCount examples from the beam homepage (from the quickstart article)
  • Now, we can create a QuickStart with the MinimalWordCount example on the Beam site by adding a simple profile on the maven pom.xml on the MinimalWordCount example.

Minor changes to note:

  • Fix minor typos (e.g., confg --> config)
  • Fix minor indentation inconsistencies

Tests for the changes:

  • Existing tests confirm that these changes do not break the code

Other comments:

  • N/A

Closes #163

@wonook wonook self-assigned this Nov 27, 2018
Copy link
Contributor

@taegeonum taegeonum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! one minor comment.

@taegeonum
Copy link
Contributor

@johnyangk @seojangho Could you please take a look?

@taegeonum taegeonum merged commit 6e2a4db into apache:master Dec 3, 2018
@wonook wonook deleted the 294-BeamRunner branch December 11, 2018 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants