-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
all-spark-notebook does not work when using spark on mesos #36
Comments
Ah, yes. The mesos slaves need to have You can either have the cluster jobs run as |
Thanks! That makes a lot of sense now. Ideally, I would like the user to be "spark" because this is what the username we used for submitting spark jobs. Is it difficult to make it configurable (i.e. jovyan to "whatevernamewewanttouse")? :) |
I agree it would be nice, but not sure how easy to enable that. We've had lots of trouble with getting users, groups, and permissions straight. Maybe the last round of fixes in that area greased the wheels for making the user name configurable too. I'll give it a look at least. |
After a quick snoop, the answer is: not easily. The Dockerfiles have |
And to clarify my last comment, that would only allow build time changes of NB_USER. Run time is a whole other big problem involving usermod and ownership fixes. |
Hey @parente, thanks for looking into it. I was hoping it is an easy sweep. I have been building images for the past few days too so I understand the pain and the complexity. Meanwhile, I think I will keep hacking the Dockerfiles! |
I documented the Mesos user requirements (create jovyan or run with --no-switch_user on slaves) in the pyspark and all-spark READMEs. I think this is the best we can do for the moment. If a more flexible option presents itself, I'll certainly revisit this. |
hey @parente, |
ok, i think i did this wrong.
|
Hi there,
I have tried this for hours and I cannot use spark on mesos with the current configurations defined in the Dockerfile. Have anyone tried using this docker image with mesos?
I hacked around those configurations inside the container and the only way to make this to work is to use root instead of a user name "jov*".
The text was updated successfully, but these errors were encountered: