Skip to content

[BEAM-2233,BEAM-2223] Adjust TestSparkPipelineOptions registration; add per-runner profiles to Java 8 examples pom#3002

Closed
kennknowles wants to merge 2 commits intoapache:masterfrom
kennknowles:java8-examples-runner-profiles
Closed

[BEAM-2233,BEAM-2223] Adjust TestSparkPipelineOptions registration; add per-runner profiles to Java 8 examples pom#3002
kennknowles wants to merge 2 commits intoapache:masterfrom
kennknowles:java8-examples-runner-profiles

Conversation

@kennknowles
Copy link
Copy Markdown
Member

@kennknowles kennknowles commented May 9, 2017

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

  • Make sure the PR title is formatted like:
    [BEAM-<Jira issue #>] Description of pull request
  • Make sure tests pass via mvn clean verify.
  • Replace <Jira issue #> in the title with the actual Jira issue
    number, if there is one.
  • If this contribution is large, please file an Apache
    Individual Contributor License Agreement.

We have cascading failures:

  • All runners were included together in the include-runners profile in the Java 8 examples
  • TestOptions has a hard dep on hamcrest that we list as provided, putting a burden on users
  • TestSparkPipelineOptions was registered, so the Spark runner had this same hard dep and class load time
  • If you do options.as(MyUnregisteredOptions.class) this will cause TestPipeline/PipelineOptionsFactory to crash with a parse error even if none of the unregistered options were present

These should all be fixed, IMO as follows:

  • Runners should have separate profiles (done in this PR)
  • TestOptions should either not use hamcrest or should declare a correct dependency (not resolved)
  • TestSparkPipelineOptions should either not need to be registered or should be allowed to be registered (expected resolution: no need to register)
  • TestPipeline / PipelineOptionsFactory should not crash due to how options have been observed, when they are not mutated, and probably should just not crash at all

@kennknowles kennknowles force-pushed the java8-examples-runner-profiles branch from b45ab8b to 90dc726 Compare May 9, 2017 21:25
@kennknowles kennknowles changed the title [BEAM-2233,BEAM-2223] Add per-runner profiles to Java 8 examples pom [BEAM-2233,BEAM-2223] Adjust TestSparkPipelineOptions registration; add per-runner profiles to Java 8 examples pom May 9, 2017
@kennknowles
Copy link
Copy Markdown
Member Author

run spark validatesrunner

@kennknowles
Copy link
Copy Markdown
Member Author

retest this please

@kennknowles
Copy link
Copy Markdown
Member Author

I have manually run the Spark ValidatesRunner tests and eventually got them to succeed via the method of registering the test options only in src/test.

Copy link
Copy Markdown
Member

@davorbonaci davorbonaci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kennknowles kennknowles force-pushed the java8-examples-runner-profiles branch from 90dc726 to 35ec567 Compare May 9, 2017 22:10
@kennknowles
Copy link
Copy Markdown
Member Author

run spark validatesrunner

@davorbonaci
Copy link
Copy Markdown
Member

All passed. Merging.

@asfgit asfgit closed this in b9ea415 May 10, 2017
@kennknowles kennknowles deleted the java8-examples-runner-profiles branch May 26, 2017 05:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants