Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added the port erasure to SparkFunSuite's cleanup. #104

Merged

Conversation

tdanford
Copy link
Contributor

Fixes ISSUE #103

We've been getting intermittent errors, with respect to Spark being unable to bind to a port,
in the context of repeated unit tests. This apparently is a known problem, see the thread here:
http://blog.quantifind.com/posts/spark-unit-test/
and the follow-up from Matei here:
https://groups.google.com/forum/#!topic/spark-users/MeVzgoJXm8I

The upshot is that we need to clearProperty('spark.driver.port') when we shut down our sparkContext
after a sparkTest.

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/114/

@tdanford
Copy link
Contributor Author

Hold off on merging this for now ... Carl and I are still tracking something down.

@@ -27,3 +27,6 @@ Trunk (not yet released)

* Fixed issues where VCF header was not being written correctly. This prevented variant calls from being
written after conversion. This was fixed in PR#85.
* ISSUE 103: Added a call to clearProperty('spark.driver.port') in the cleanup from a sparkTest, so that
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: there should be a space between entries.

@fnothaft
Copy link
Member

Thanks @tdanford and @carlyeks . I believe you are still doing some further analysis, so I will wait for your all-clear before merging this.

@tdanford
Copy link
Contributor Author

Yeah, we're still looking-- this isn't quite fixing all our problems yet. We're going to have to run scalatest by hand, since we can't get it to stop truncating our stack traces through maven :-/

On Thu, Feb 13, 2014 at 3:58 PM, Frank Austin Nothaft
notifications@github.com wrote:

Thanks @tdanford and @carlyeks . I believe you are still doing some further analysis, so I will wait for your all-clear before merging this.

Reply to this email directly or view it on GitHub:
#104 (comment)

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/120/

We've been getting intermittent errors, with respect to Spark being unable to bind to a port,
in the context of repeated unit tests.  This apparently is a known problem, see the thread here:
  http://blog.quantifind.com/posts/spark-unit-test/
and the follow-up from Matei here:
  https://groups.google.com/forum/#!topic/spark-users/MeVzgoJXm8I

The upshot is that we need to clearProperty('spark.driver.port') when we shut down our sparkContext
after a sparkTest.
@tdanford
Copy link
Contributor Author

Okay, once the Jenkins build succeeds, this request should be ready (again) for review.

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/121/

fnothaft added a commit that referenced this pull request Feb 14, 2014
Added the port erasure to SparkFunSuite's cleanup.
@fnothaft fnothaft merged commit 0957bdf into bigdatagenomics:master Feb 14, 2014
@fnothaft
Copy link
Member

Thanks @tdanford and @carlyeks !

@tdanford tdanford deleted the tdanford/sparkfunsuite-port branch February 14, 2014 19:06
@tdanford
Copy link
Contributor Author

Thanks Frank!

This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants