Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support different remote_python paths on cluster nodes #4085

Merged
merged 6 commits into from
Sep 17, 2020

Conversation

abduhbm
Copy link
Contributor

@abduhbm abduhbm commented Aug 28, 2020

Trying to resolve #4063

cluster = SSHCluster(["hostA", "hostA", "hostB"], 
                     remote_python={'hostA': '/tmp/condaA/bin/python', 
                                    'hostB': '/tmp/condaB/bin/python'}, 
                     connect_options={"known_hosts": None},
                     worker_options={"nthreads": 2},
                     scheduler_options={"port": 0, "dashboard_address": ":8797"})
> w = cluster.workers
> w[0].remote_python
/tmp/condaA/bin/python
> w[1].remote_python
/tmp/condaB/bin/python

@jacobtomlinson please let me know if more work is needed here. Thanks,

@abduhbm
Copy link
Contributor Author

abduhbm commented Sep 14, 2020

@jacobtomlinson I would appreciate your feedback on this.

Copy link
Member

@jacobtomlinson jacobtomlinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this @abduhbm.

My main comment here is other options take either a single option or a list of options. So for connect_options it can either be a dictionary of settings which are applied to all connections, or a list of dictionaries.

The list length must match the order and length of the addresses list.

I would suggest this should follow the same pattern.

@abduhbm
Copy link
Contributor Author

abduhbm commented Sep 15, 2020

@jacobtomlinson thanks for the feedback.
remote_python now accepts str or list of str.

Copy link
Member

@jacobtomlinson jacobtomlinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great thanks

@mrocklin
Copy link
Member

My apologies for the delay. @jacobtomlinson thank you for the review.

I'm merging this in. Thanks @abduhbm !

@mrocklin mrocklin merged commit 755e105 into dask:master Sep 17, 2020
@abduhbm abduhbm deleted the different-remote-python branch September 17, 2020 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SSHCluster expects conda environment to be at the same path on all systems
3 participants