Adding dask notebook to integration test (#267)

* daskjob - auto detect if remote * Add back needed dask requirements * Add mlrun_dask notebook to integration test * Updating mlrun_dask notebook docs (no need to define function.use_remote anymore)
mlrun · May 18, 2020 · c015be3 · c015be3
1 parent ab5d9c5
commit c015be3
Show file tree

Hide file tree

Showing 4 changed files with 21 additions and 4 deletions.
diff --git a/dockerfiles/api-requirements.txt b/dockerfiles/api-requirements.txt
@@ -1,4 +1,8 @@
 croniter==0.3.31
 fastapi==0.54.1
 uvicorn==0.11.3
-pydantic==1.5.1
+pydantic==1.5.1
+dask==2.12.0
+dask_kubernetes==0.10.0
+kubernetes-asyncio==10.0.0
+kubernetes==10.0.0
diff --git a/examples/mlrun_dask.ipynb b/examples/mlrun_dask.ipynb
@@ -107,8 +107,7 @@
     "* **.nthreads** - number of worker threads\n",
     "* **.kfp_image** - optional, container image to use by KFP Pipeline runner (default to mlrun/dask)\n",
     "\n",
-    "If you want to access the dask dashboard from remote you need to use `NodePort` service type (set **.service_type** to 'NodePort'), and the external IP need to be specified in mlrun configuration (`mlconf.remote_host`), this will be set automatically if you are running on an Iguazio cluster.\n",
-    "If you want to use the `NodePort` for remote access to the scheduler you need to also specify `function.use_remote=True`."
+    "If you want to access the dask dashboard or scheduler from remote you need to use `NodePort` service type (set **.service_type** to 'NodePort'), and the external IP need to be specified in mlrun configuration (`mlconf.remote_host`), this will be set automatically if you are running on an Iguazio cluster."
    ]
   },
   {

diff --git a/mlrun/runtimes/daskjob.py b/mlrun/runtimes/daskjob.py
@@ -93,7 +93,7 @@ def __init__(self, spec=None,
                  metadata=None):
         super().__init__(spec, metadata)
         self._cluster = None
-        self.use_remote = False
+        self.use_remote = not get_k8s_helper(init=False).is_running_inside_kubernetes_cluster()
         self.spec.build.base_image = self.spec.build.base_image or 'daskdev/dask:latest'
 
     @property

diff --git a/tests/integration/test-notebooks.yml b/tests/integration/test-notebooks.yml
@@ -5,6 +5,9 @@ env:
   # path in which artifacts will be stored - must start with "/User". e.g. /User/test-artifacts
   MLRUN_ARTIFACT_PATH:
 
+  # IP of the app node in which dask scheduler is running. e.g. 3.20.113.215
+  MLRUN_REMOTE_HOST:
+
   # user used to pass the mlrun-api ingress auth - e.g. iguazio
   MLRUN_httpdb__user:
 
@@ -47,3 +50,14 @@ notebook_tests:
     V3IO_ACCESS_KEY: ${V3IO_ACCESS_KEY}
 
     JUPYTER_NOTEBOOK_FILE_NAME: 'mlrun_jobs.ipynb'
+- notebook_name: mlrun_dask.ipynb
+  env:
+    MLRUN_DBPATH: ${MLRUN_DBPATH}
+    MLRUN_ARTIFACT_PATH: ${MLRUN_ARTIFACT_PATH}
+    MLRUN_REMOTE_HOST: ${MLRUN_REMOTE_HOST}
+    MLRUN_httpdb__user: ${MLRUN_httpdb__user}
+    MLRUN_httpdb__password: ${MLRUN_httpdb__password}
+    V3IO_USERNAME: ${V3IO_USERNAME}
+    V3IO_ACCESS_KEY: ${V3IO_ACCESS_KEY}
+
+    JUPYTER_NOTEBOOK_FILE_NAME: 'mlrun_dask.ipynb'