New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use SparkConf object to configure SparkContext #223

Merged
merged 1 commit into from Apr 23, 2014

Conversation

Projects
None yet
4 participants
@massie
Member

massie commented Apr 22, 2014

The new Spark SparkConf object replaces the old configuration system that uses
Java system properties and environment variables. SparkConf is thread-safe and
self-contained. For now, SparkConf is constructed to honor the old system and
environment properties. With time, we should change this behavior.

One effect of this move to SparkConf is to prevent errors around binding a port
for the SparkContext in each test case.

This commit also fixes a race condition in GenotypePredicatesSuites.

Use SparkConf object to configure SparkContext
The new Spark SparkConf object replaces the old configuration system that uses
Java system properties and environment variables. SparkConf is thread-safe and
self-contained. For now, SparkConf is constructed to honor the old system and
environment properties. With time, we should change this behavior.

One effect of this move to SparkConf is to prevent errors around binding a port
for the SparkContext in each test case.

This commit also fixes a race condition in GenotypePredicatesSuites.
@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Apr 23, 2014

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/297/

AmplabJenkins commented Apr 23, 2014

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/297/

@nealsid

This comment has been minimized.

Show comment
Hide comment
@nealsid

nealsid Apr 23, 2014

sometimes you do need to pass '=' on the RHS of environment variables. Maybe you could split and always choose the LHS of the very first one as the key and the remainder as the value?

sometimes you do need to pass '=' on the RHS of environment variables. Maybe you could split and always choose the LHS of the very first one as the key and the remainder as the value?

fnothaft added a commit that referenced this pull request Apr 23, 2014

Merge pull request #223 from massie/sparkconf
Use SparkConf object to configure SparkContext

@fnothaft fnothaft merged commit cf42239 into bigdatagenomics:master Apr 23, 2014

1 check passed

default Merged build finished.
Details
@nealsid

This comment has been minimized.

Show comment
Hide comment
@nealsid

nealsid Apr 23, 2014

LGTM, thanks @massie!

nealsid commented on 12b5355 Apr 23, 2014

LGTM, thanks @massie!

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Apr 23, 2014

Member

Merged! Thanks @massie! @nealsid, this should fix the issues you reported about @AmplabJenkins earlier today.

Member

fnothaft commented Apr 23, 2014

Merged! Thanks @massie! @nealsid, this should fix the issues you reported about @AmplabJenkins earlier today.

@massie massie deleted the massie:sparkconf branch Sep 2, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment