Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EC2] [SPARK-6600] Open ports in ec2/spark_ec2.py to allow HDFS NFS gateway #5257

Closed
wants to merge 1 commit into from

Conversation

florianverhein
Copy link
Contributor

Authorizes incoming access to master on the ports required to use the hadoop hdfs nfs gateway from outside the cluster.

@SparkQA
Copy link

SparkQA commented Mar 30, 2015

Test build #29389 has started for PR 5257 at commit 72a586a.

@florianverhein
Copy link
Contributor Author

Some things to think about:

  • Do we want an option for this (e.g. as for ganglia)? I haven't done this as I think it would be confusing at the moment, since a user would assume that the option would enable the hdfs nfs gateway on the cluster. However as far as I'm aware, spark-ec2 doesn't do this yet ([SPARK-8060] Improve DataFrame Python test coverage and documentation. #6601). So I think it would be better if the option were added as part of that work.
  • Further, since the ports are opened to the authorized address, I don't see a problem in having this done by default for now.

I have tested this with a spark-ec2 cluster running the gateway (i.e. with these settings, I can mount the hfds on my local machine - which is really handy!)

@SparkQA
Copy link

SparkQA commented Mar 30, 2015

Test build #29389 has finished for PR 5257 at commit 72a586a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.
  • This patch does not change any dependencies.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29389/
Test PASSed.

@srowen
Copy link
Member

srowen commented Mar 30, 2015

I think this seems reasonable. I'll leave it open for comments for some time.

@asfgit asfgit closed this in 4122623 Apr 1, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants