Skip to content

Latest commit

 

History

History
 
 

python-sdk

page_type languages products description
sample
python
azurecli
azure-machine-learning
Top-level directory for official Azure Machine Learning Python SDK sample code and notebooks.

Azure Machine Learning Python SDK examples

cleanup code style: black license: MIT

Welcome to the Azure Machine Learning examples repository!

Prerequisites

  1. An Azure subscription. If you don't have an Azure subscription, create a free account before you begin.
  2. A terminal and Python >=3.6,<3.9.

Set up

Clone this repository and install required packages:

git clone https://github.com/Azure/azureml-examples --depth 1
cd azureml-examples/python-sdk
pip install --upgrade -r requirements.txt

To create or setup a workspace with the assets used in these examples, run the setup script.

If you do not have an Azure ML workspace, run python setup-workspace.py --subscription-id $ID, where $ID is your Azure subscription id. A resource group, Azure ML workspace, and other necessary resources will be created in the subscription.

If you have an Azure ML Workspace, install the Azure ML CLI and run az ml folder attach -w $WS -g $RG, where $WS and $RG are the workspace and resource group names.

Run python setup-workspace.py -h to see other arguments.

Getting started

To get started, see the introductory tutorial which uses Azure ML to:

  • run a "hello world" job on cloud compute, demonstrating the basics
  • run a series of PyTorch training jobs on cloud compute, demonstrating mlflow tracking & using cloud data

These concepts are sufficient to understand all examples in this repository, which are listed below.

Examples

Tutorials (tutorials)

path status notebooks description
automl-with-azureml automl-nlp-text-classification-multiclass
automl-nlp-text-classification-multilabel
automl-nlp-ner
auto-ml-classification-bank-marketing-all-features
auto-ml-classification-credit-card-fraud
auto-ml-classification-text-dnn
auto-ml-continuous-retraining
auto-ml-forecasting-backtest-many-models
auto-ml-forecasting-backtest-single-model
auto-ml-forecasting-bike-share
auto-ml-forecasting-data-preparation
auto-ml-forecasting-demand-forecasting-many-models
auto-ml-forecasting-demand-tcn
auto-ml-forecasting-energy-demand
auto-ml-forecasting-function
auto-ml-forecasting-github-dau
auto-ml-forecasting-hierarchical-timeseries
auto-ml-forecasting-many-models
auto-ml-forecasting-orange-juice-sales
auto-ml-forecasting-pipelines
auto-ml-forecasting-univariate-recipe-experiment-settings
auto-ml-forecasting-univariate-recipe-run-experiment
auto-ml-image-classification-multiclass-batch-scoring
auto-ml-image-classification-multiclass
auto-ml-image-classification-multilabel
auto-ml-image-instance-segmentation
auto-ml-image-object-detection
auto-ml-classification-credit-card-fraud-local
binary-classification-metric-and-confidence-interval
auto-ml-regression-explanation-featurization
auto-ml-regression
automl-nlp-text-classification-multiclass.ipynb
automl-nlp-text-classification-multilabel.ipynb
automl-nlp-ner.ipynb
auto-ml-classification-bank-marketing-all-features.ipynb
auto-ml-classification-credit-card-fraud.ipynb
auto-ml-classification-text-dnn.ipynb
auto-ml-continuous-retraining.ipynb
auto-ml-forecasting-backtest-many-models.ipynb
auto-ml-forecasting-backtest-single-model.ipynb
auto-ml-forecasting-bike-share.ipynb
auto-ml-forecasting-data-preparation.ipynb
auto-ml-forecasting-demand-forecasting-many-models.ipynb
auto-ml-forecasting-demand-tcn.ipynb
auto-ml-forecasting-energy-demand.ipynb
auto-ml-forecasting-function.ipynb
auto-ml-forecasting-github-dau.ipynb
auto-ml-forecasting-hierarchical-timeseries.ipynb
auto-ml-forecasting-many-models.ipynb
auto-ml-forecasting-orange-juice-sales.ipynb
auto-ml-forecasting-pipelines.ipynb
auto-ml-forecasting-univariate-recipe-experiment-settings.ipynb
auto-ml-forecasting-univariate-recipe-run-experiment.ipynb
auto-ml-image-classification-multiclass-batch-scoring.ipynb
auto-ml-image-classification-multiclass.ipynb
auto-ml-image-classification-multilabel.ipynb
auto-ml-image-instance-segmentation.ipynb
auto-ml-image-object-detection.ipynb
auto-ml-classification-credit-card-fraud-local.ipynb
binary-classification-metric-and-confidence-interval.ipynb
auto-ml-regression-explanation-featurization.ipynb
auto-ml-regression.ipynb
Tutorials showing how to build high quality machine learning models using Azure Automated Machine Learning.
automl-with-databricks automl-databricks-local-01
automl-databricks-local-with-deployment
automl-databricks-local-01.ipynb
automl-databricks-local-with-deployment.ipynb
no description
dataset-uploads dataset-uploads upload_dataframe_register_as_dataset.ipynb
upload_directory_create_file_dataset.ipynb
no description
deploy-local deploy-local 1.deploy-local.ipynb
2.deploy-local-cli.ipynb
no description
train-on-low-priority-aml-compute train-on-low-priority-aml-compute train-on-low-priority-aml-compute.ipynb Learn how to train models using low-priority AML compute. Low-priority compute costs significantly less than dedicated compute, but jobs on low priority compute can be preempted.
using-pipelines using-pipelines dataset-and-pipelineparameter.ipynb
image-classification.ipynb
publish-and-run-using-rest-endpoint.ipynb
style-transfer-parallel-run.ipynb
no description
using-rapids using-rapids 1.train-and-hpo.ipynb
2.train-multi-gpu.ipynb
Learn how to accelerate PyData tools (Numpy, Pandas, Scikit-Learn, etc.) on NVIDIA GPUs with RAPIDS and Azure ML.

Notebooks (notebooks)

path status description

Train (workflows/train)

path status description
deepspeed/cifar/job.py train-deepspeed-cifar-job train CIFAR-10 using DeepSpeed and PyTorch
deepspeed/transformers/job.py train-deepspeed-transformers-job train Huggingface transformer using DeepSpeed
fastai/mnist-mlproject/job.py train-fastai-mnist-mlproject-job train fastai resnet18 model on mnist data via mlflow mlproject
fastai/mnist/job.py train-fastai-mnist-job train fastai resnet18 model on mnist data
fastai/pets/job.py train-fastai-pets-job train fastai resnet34 model on pets data
lightgbm/iris/job.py train-lightgbm-iris-job train a lightgbm model on iris data
pytorch/cifar-distributed/job.py train-pytorch-cifar-distributed-job train CNN model on CIFAR-10 dataset with distributed PyTorch
pytorch/mnist-mlproject/job.py train-pytorch-mnist-mlproject-job train a pytorch CNN model on mnist data via mlflow mlproject
pytorch/mnist/job.py train-pytorch-mnist-job train a pytorch CNN model on mnist data
scikit-learn/diabetes-mlproject/job.py train-scikit-learn-diabetes-mlproject-job train sklearn ridge model on diabetes data via mlflow mlproject
scikit-learn/diabetes/job.py train-scikit-learn-diabetes-job train sklearn ridge model on diabetes data
tensorflow/mnist-distributed-horovod/job.py train-tensorflow-mnist-distributed-horovod-job train tensorflow CNN model on mnist data distributed via horovod
tensorflow/mnist-distributed/job.py train-tensorflow-mnist-distributed-job train tensorflow CNN model on mnist data distributed via tensorflow
tensorflow/mnist/job.py train-tensorflow-mnist-job train tensorflow NN model on mnist data
transformers/glue/1-aml-finetune-job.py train-transformers-glue-1-aml-finetune-job Submit GLUE finetuning with Huggingface transformers library on Azure ML
transformers/glue/2-aml-comparison-of-sku-job.py train-transformers-glue-2-aml-comparison-of-sku-job Experiment comparing training performance of GLUE finetuning task with differing hardware.
transformers/glue/3-aml-hyperdrive-job.py train-transformers-glue-3-aml-hyperdrive-job Automatic hyperparameter optimization with Azure ML HyperDrive library.
xgboost/iris/job.py train-xgboost-iris-job train xgboost model on iris data

Deploy (workflows/deploy)

path status description
pytorch/mnist/job.py deploy-pytorch-mnist-job deploy pytorch cnn model trained on mnist data to aks
scikit-learn/diabetes/job.py deploy-scikit-learn-diabetes-job deploy sklearn ridge model trained on diabetes data to AKS

Experimental tutorials (experimental)

path status notebooks description why experimental?
automl-model-testing classification-TSI
configuration
forecasting-TSI
regression-TSI
classification-TSI.ipynb
configuration.ipynb
forecasting-TSI.ipynb
regression-TSI.ipynb
no description unknown
deploy-triton deploy-triton 1.bidaf-ncd-local.ipynb Learn how to efficiently deploy to GPUs with the Triton inference server and Azure ML. in preview
using-pytorch-lightning using-pytorch-lightning 1.train-single-node.ipynb
2.log-with-tensorboard.ipynb
3.log-with-mlflow.ipynb
Learn how to train and log metrics with PyTorch Lightning and Azure ML. issues with multinode pytorch lightning

Contents

A lightweight template repository for automating the ML lifecycle can be found here. The contents of this repository are described below.

Note: It is not recommended to fork this repository and use it as a template directly. This repository is structured to host a large number of examples and CI for automation and testing.

directory description
experimental self-contained directories of experimental tutorials
tutorials self-contained directories of tutorials
workflows self-contained directories of job to be run, organized by scenario then tool then project

Contributing

We welcome contributions and suggestions! Please see the contributing guidelines for details.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. Please see the code of conduct for details.

Reference