-
-
Notifications
You must be signed in to change notification settings - Fork 143
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
dask-jobqueue binder #276
Comments
I am going to try to do this and see how far I can push it. |
I have some proof of concept here: Here is the binder link: For now there is a single notebook Full disclosure: I have seen some sporadic problems with the processes supervised by supervisord (mostly |
Thanks @lesteve! This is nice! I had trouble making the binder start, I needed to launch it 4 times... Don't know why. Then I have the mysqldb daemon not started, but thanks to your first cell I could start it easily. I think the idea works and the RAM may not be a limitation for some simple examples. There may be more on Pangeo binder, but not sure this will make a big difference if we don"t use separated pods for the workers. The first question that came to my mind then is : how using SlurmCluster is different from LocalCluster. That's the beauty of Dask, just change LocalCluster with SlurmCluster and the rest of the code is the same. What specific example can we set up for dask-jobqueue?
|
I had another look at this, I tweaked it a bit, and it looks like this is working better than my last attempt (not sure why ...). So maybe worth revisiting? https://mybinder.org/v2/gh/lesteve/test-binder/master?filepath=simple.ipynb For me the main point would be a quick intro into dask-jobqueue:
Comments more than welcome! |
That's great news! I'll have a look. |
Looks a lot more stable. Scaling the cluster up and down doesn't seem to break the Slurm scheduler anymore. |
The idea is to have a binder setup with a toy cluster so that people can play with dask-jobqueue a bit without having to set it up on their cluster.
Our SLURM CI setup uses a single Dockerfile, maybe this image could be used to have a binder.
Binder allows you to use a Dockerfile:
https://github.com/binder-examples/minimal-dockerfile
Questions:
If this idea works, we could think about what kind of notebooks to create (related to #253).
The text was updated successfully, but these errors were encountered: