AzureMLCluster #67
AzureMLCluster #67
Conversation
…ot available to worker
Show "Complete" status for canceled runs.
Add comment to explain why we need complete()
Todrabas devel
Update azureml.py
Todrabas devel
Handling API change in asyncio between Python 3.6 and 3.7
Todrabas devel
Fix some bugs
Merging the changes. Fix some bugs
Merging Fred's changes to master
Make env object required
Make env object required
add note about WSL
Fixing SSH port forwarding
|
I think we are nearly there with this. I'd really like to push on getting it merged next week. I'm pretty busy this week but should be able to do some final testing next week. Given the size and age of this PR I think as long as we can successfully run through the instructions then any future changes can be made in extra PRs. |
|
Thanks Jacob! Please, keep me posted. Happy to jump on a call to quickly
address all the issues if any.
…On Thu, Jun 18, 2020 at 3:18 AM Jacob Tomlinson ***@***.***> wrote:
I think we are nearly there with this. I'd really like to push on getting
it merged next week. I'm pretty busy this week but should be able to do
some final testing next week.
Given the size and age of this PR I think as long as we can successfully
run through the instructions then any future changes can be made in extra
PRs.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#67 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAYZOHGJ25FNZMZSSILM3WDRXHSYLANCNFSM4K3SFWMA>
.
|
update to ssh instead of vnet in index.rst
Fixing bug wrt localhost
There was a problem hiding this comment.
Thanks for the changes.
I've run through the instructions again and everything seemed ok. I made a couple of minor comments but nothing huge.
I think there is still work to be done here. A few things that spring to mind:
- The Jupyter environment should have the Dask extension installed
- The Jupyter environment should be pre-configured to find the Dask cluster when running
client = Client(). We do it like this in the Helm Chart - We shouldn't be printing output as actions are happening by default. Perhaps this should be a kwarg to toggle on for those that find it useful.
- I'd love to explore pulling all of the boilerplate code from the instructions into the
AzureMLClusterclass. We should create theWorkspace,ComputeandEnvironmentobjects insideAzureMLClusterfor people, unless they want to pass their own via kwargs. I appreciate this will probably explode the number of kwargsAzureMLClusterneeds to take, but this will be necessary for widespread adoption.
With the few changes I've suggested inline I think we should merge this now and I'll raise issues for the other points I've made above.
| Setup | ||
| ~~~~~ | ||
|
|
||
| Next, create the ``Workspace`` object given your AzureML ``Workspace`` parameters. Check |
There was a problem hiding this comment.
Could we add some instructions here on how to find this info.
For example I grabbed the config.json file from the Azure Portal.
doc/source/index.rst
Outdated
| """"""""""""""" | ||
| If you do not have a virtual network yet there are two ways to create one. | ||
|
|
||
| 1. Using `https://azure.portal.com <https://azure.portal.com>`_: |
There was a problem hiding this comment.
| 1. Using `https://azure.portal.com <https://azure.portal.com>`_: | |
| 1. Using `https://portal.azure.com <https://portal.azure.com>`_: |
doc/source/index.rst
Outdated
| .. code-block:: python | ||
|
|
||
| amlcluster = AzureMLCluster( | ||
| workspace=ws, |
There was a problem hiding this comment.
| workspace=ws, | |
| workspace=ws |
doc/source/index.rst
Outdated
| .. code-block:: python | ||
| env = ws.environments[env_name] |
There was a problem hiding this comment.
| .. code-block:: python | |
| env = ws.environments[env_name] | |
| .. code-block:: python | |
| env = ws.environments[env_name] |
| , admin_ssh_key=admin_ssh_key_priv ### path, not contents of the key | ||
| ) | ||
|
|
||
| Once the cluster has started, the Dask Cluster widget will print out two links: |
There was a problem hiding this comment.
We should also give instructions on how to make this cluster stop. If I restart my notebook kernel or del the cluster object what happens, is anything cleaned for me? Do I need to kill the run and delete the compute resource in AzureML myself?
address jacob's comments, get rid of en-us in links
Further refinements to links
|
@jacobtomlinson the comments should be addressed now |
|
Awesome work, thanks @lostmygithubaccount! I think I'm going to hit merge and look to get a release out later today. I just want to say thanks again for all the effort here. This has been a mammoth PR and has taken us quite a while to get it merged. |
Add deployment type
Add
AzureMLClusteras one of the providers. The class derives fromdask.distributed.deploy.Clusterand makes creating, scaling and working with Dask on Azure ML.