Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

do not merge: move dependencies to databricks utils #1980

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

do not merge: move dependencies to databricks utils #1980

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fdb4643
fix ExplanationDashboard interpretability failure
imatiach-msft May 30, 2023
41ced8d
move pip installs to databricks utilities
imatiach-msft Jun 7, 2023
dca6b66
Update DatabricksUtilities.scala
imatiach-msft Jun 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 6 additions & 2 deletions core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/DatabricksUtilities.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ object DatabricksUtilities {
val Region = "eastus"
val PoolName = "synapseml-build-10.4"
val GpuPoolName = "synapseml-build-10.4-gpu"
val AdbRuntime = "10.4.x-scala2.12"
val AdbRuntime = "12.2.x-scala2.12"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think scala2.12 begs for its own val?

val AdbGpuRuntime = "10.4.x-gpu-ml-scala2.12"
val NumWorkers = 5
val AutoTerminationMinutes = 15
Expand All @@ -57,7 +57,11 @@ object DatabricksUtilities {
Map("pypi" -> Map("package" -> "onnxmltools==1.7.0")),
Map("pypi" -> Map("package" -> "lightgbm")),
Map("pypi" -> Map("package" -> "mlflow")),
Map("pypi" -> Map("package" -> "openai"))
Map("pypi" -> Map("package" -> "openai")),
Map("pypi" -> Map("package" -> "raiwidgets")),
Map("pypi" -> Map("package" -> "itsdangerous==2.0.1")),
Map("pypi" -> Map("package" -> "hyperopt")),
Map("pypi" -> Map("package" -> "scikit-learn"))
).toJson.compactPrint

// TODO: install synapse.ml.dl wheel package here
Expand Down
10 changes: 5 additions & 5 deletions notebooks/features/hyperparameter_tuning/HyperOpt-SynapseML.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,7 @@
"outputs": [],
"source": [
"# %pip install hyperopt\n",
"import os\n",
"\n",
"os.system(\"pip install hyperopt\")"
"# %pip install scikit-learn"
]
},
{
Expand All @@ -54,8 +52,7 @@
"outputs": [],
"source": [
"# version >= 1.28.0 supports reading logModelAllowlistFile from url\n",
"# %pip install mlflow==1.29.0\n",
"os.system(\"pip install mlflow==1.29.0\")"
"# %pip install mlflow"
]
},
{
Expand All @@ -71,12 +68,15 @@
"\n",
"if running_on_synapse_internal():\n",
" experiment_name = \"hyperopt-synapseml\"\n",
" os.system(\"pip install mlflow\")\n",
"elif running_on_synapse():\n",
" experiment_name = \"hyperopt-synapseml\"\n",
" # from notebookutils.visualization import display # use this display on interactive notebook\n",
" from synapse.ml.core.platform import (\n",
" materializing_display as display,\n",
" ) # display for pipeline testing\n",
"\n",
" os.system(\"pip install mlflow\")\n",
"else:\n",
" experiment_name = \"/Shared/hyperopt-synapseml\""
]
Expand Down
21 changes: 21 additions & 0 deletions notebooks/features/isolation_forest/IsolationForest - Multivariate Anomaly Detection.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,27 @@
" - If you are running it on Synapse, you'll need to [create an AML workspace and set up linked Service](https://microsoft.github.io/SynapseML/docs/next/mlflow/installation/). \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First we install required dependencies and import the packages and define some UDFs we will need later.\n",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: too many ands

"When running Synapse Analytics, please follow instructions here [Package management - Azure Synapse Analytics | Microsoft Docs](https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-azure-portal-add-libraries) to install [\"raiwidgets\"](https://pypi.org/project/raiwidgets/) and [\"interpret-community\"](https://pypi.org/project/interpret-community/) packages."
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: replace here with in

]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# %pip install raiwidgets\n",
"# %pip install itsdangerous==2.0.1\n",
"# %pip install interpret-community"
]
},
{
"cell_type": "markdown",
"metadata": {
Expand Down
52 changes: 21 additions & 31 deletions notebooks/features/responsible_ai/Interpretability - Explanation Dashboard.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,27 @@
"\n",
"In this example, similar to the \"Interpretability - Tabular SHAP explainer\" notebook, we use Kernel SHAP to explain a tabular classification model built from the Adults Census dataset and then visualize the explanation in the ExplanationDashboard from https://github.com/microsoft/responsible-ai-widgets.\n",
"\n",
"First we import the packages and define some UDFs we will need later."
"First we install required dependencies and import the packages and define some UDFs we will need later."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When running Synapse Analytics, please follow instructions here [Package management - Azure Synapse Analytics | Microsoft Docs](https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-azure-portal-add-libraries) to install [\"raiwidgets\"](https://pypi.org/project/raiwidgets/) and [\"interpret-community\"](https://pypi.org/project/interpret-community/) packages."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# %pip install raiwidgets\n",
"# %pip install itsdangerous==2.0.1\n",
"# %pip install interpret-community"
]
},
{
Expand Down Expand Up @@ -312,36 +332,6 @@
" converted_importance_values.append(converted_list)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When running Synapse Analytics, please follow instructions here [Package management - Azure Synapse Analytics | Microsoft Docs](https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-azure-portal-add-libraries) to install [\"raiwidgets\"](https://pypi.org/project/raiwidgets/) and [\"interpret-community\"](https://pypi.org/project/interpret-community/) packages."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"!pip install --upgrade raiwidgets\n",
"!pip install itsdangerous==2.0.1"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"!pip install --upgrade interpret-community"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down