Skip to content

Comments

[BEAM-4849] Support running Beam Samza jobs in Yarn#6029

Merged
lukecwik merged 1 commit intoapache:masterfrom
xinyuiscool:BEAM-4849
Aug 10, 2018
Merged

[BEAM-4849] Support running Beam Samza jobs in Yarn#6029
lukecwik merged 1 commit intoapache:masterfrom
xinyuiscool:BEAM-4849

Conversation

@xinyuiscool
Copy link
Contributor

Add a couple of util classes and script to run beam job containers in Yarn:

  1. A factory class to read config and load into a static Samza job model;
  2. Supply the static model to the container runner.

Follow this checklist to help us incorporate your contribution quickly and easily:

  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- --- --- --- ---
Java Build Status Build Status Build Status Build Status Build Status Build Status Build Status
Python Build Status --- Build Status
Build Status
--- --- --- ---

@lukecwik
Copy link
Member

Run Java PreCommit

2 similar comments
@lukecwik
Copy link
Member

Run Java PreCommit

@lukecwik
Copy link
Member

Run Java PreCommit

@lukecwik
Copy link
Member

For some reason after attempting to get the precommit to pass it has been consistently failing on

Task :beam-examples-java:compileTestJava FAILED
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Phrase/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22: error: cannot access BigqueryMatcher
import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
                                         ^
  bad class file: /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Phrase/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
    unable to access file: java.util.zip.ZipException: invalid stored block lengths
    Please remove or make sure it appears in the correct subdirectory of the classpath.
1 error
:beam-examples-java:compileTestJava (Thread[Task worker for ':' Thread 8,5,main]) completed. Took 16.434 secs.

Any ideas?

@xinyuiscool
Copy link
Contributor Author

Pretty weird, my changes are very localized to the Samza Runner, and it shouldn't affect examples. Could it be some intermittent failure? Let me try running precommit.

@xinyuiscool
Copy link
Contributor Author

Run Java PreCommit

@lukecwik
Copy link
Member

Sorry for the delay, was out on vacation.

@lukecwik lukecwik merged commit 95f611c into apache:master Aug 10, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants