Welcome to the PREVIEW of Azure Machine Learning Automated ML Model Testing. The AutoML model testing feature provides support for automated model testing.
Automated Machine Learning, also referred to as Automated ML or AutoML, is the process of automating the time consuming, iterative tasks of machine learning model development. It allows data scientists, analysts, and developers to quickly and easily build ML models with high scale, efficiency, and productivity all while sustaining model quality.
The AutoML model testing feature allows the user to test the models which are generated by AutoML training runs. Testing a model takes as input a test dataset and outputs predictions and metrics for those predictions.
There are currently two supported scenarios:
- Test the best model which is created by the main AutoML run. The model testing run will be started automatically at the end of the main run after all models have been created and the best model has been identified.
- Test any model after the main AutoML run has completed ("on-demand" model testing). These test runs are started manually by the user either using the SDK or the UI.
Model testing is supported for the following task types:
- Classification
- Regression
- Forecasting
- An Azure subscription. If you don't have an Azure subscription, create a free account before you begin.
- A valid Azure Machine Learning workspace. See here for more details and how to create one.
- AzureML SDK version 1.29 or later for script and notebook environments. See here for SDK installation details.
There are three ways to start a test run from a script or a notebook. See the included notebooks for examples and further details.
-
Providing an existing test dataset when creating the
AutoMLConfig.automl_config = AutoMLConfig(task='forecasting', ... # Provide an existing test dataset test_data=test_dataset, ... forecasting_parameters=forecasting_parameters)
Supported input test dataset types: Dataset.
Sample notebooks:
-
Specifying a train/test percentage split when creating the
AutoMLConfig. Note, this scenario is not supported for forecasting tasks.automl_config = AutoMLConfig(task = 'regression', ... # Specify train/test split training_data=training_data, test_size=0.2)
To use a train/test split instead of providing test data directly, use the
test_sizeparameter when creating theAutoMLConfig. This parameter must be a floating point value between 0.0 and 1.0 exclusive and specifies the percentage of the training dataset that should be used for the test dataset. For regression based tasks, random sampling is used. For classification tasks, stratified sampling is used. Forecasting does not currently support specifying a test dataset using a train/test split.The
test_dataandtest_sizeAutoMLConfigparameters are mutually exclusive and can not be specified at the same time.Sample notebooks:
-
Use
ModelProxyto test a model after the main AutoML run has completed (aka. "on-demand" model testing).from azureml.train.automl.model_proxy import ModelProxy model_proxy = ModelProxy(best_run) predictions, metrics = model_proxy.test(test_data)
Supported input test dataset types: Dataset.
Sample notebooks:
When a remote test run is requested by passing in a value for test_data or
test_size to AutoMLConfig, only the best training run will have a test run
associated with it.
best_run, fitted_model = remote_run.get_output()
test_run = next(best_run.get_children(type='automl.model_test'))
test_run.wait_for_completion(show_output=False, wait_post_processing=True)
# Get test metrics
test_run_metrics = test_run.get_metrics()
for name, value in test_run_metrics.items():
print(f"{name}: {value}")
# Get test predictions as a Dataset
test_run_details = test_run.get_details()
dataset_id = test_run_details['outputDatasets'][0]['identifier']['savedId']
test_run_predictions = Dataset.get_by_id(workspace, dataset_id)
predictions_df = test_run_predictions.to_pandas_dataframe()
# Alternatively, the test predictions can
# be retrieved via the run outputs.
test_run.download_file("predictions/predictions.csv")
predictions_df = pd.read_csv("predictions.csv")ModelProxy already returns the predictions and metrics and
does not require further processing to retrieve the outputs.
There are 2 ways to start a model test run via UI
1. Start a model test run while creating a new AutoML run.
2. Start an on-demand model test run for a completed model.
-
Browse to https://ml.azure.com/. Click on
Create newbutton -> selectNew Automated ML run. Follow the steps to create a new AutoML run.
-
On the [Optional] Validate step -> Test dataset -> Select Provide a test dataset option.
Dataset options:
- Select a test dataset from the dropdown.
- Or create a new dataset by clicking
Create datasetbutton.
Dataset schema:
- The schema of the test dataset should match the training dataset (The Target column is optional).
- The test dataset should not be the same as training dataset or the validation dataset.
-
If you prefer test dataset to be a split percentage of training dataset, in the
Test datasetdropdown -> Select Test split (choose a percentage of the training data) option.
-
Click the
Finishbutton. After the run gets completed, navigate to theModelstab of the run and click on thebest model.
-
On the
model detailspage, navigate to theTest resultstab. It lists the model test run(s). Click on a run in the list to view the test run details.
-
On the
model test run detailpage, click on the 'Output datasets' link to explore the predictions dataset.
-
On the
dataset detailspage, click onExploreto preview the predictions csv. Note: Work is in progress to easily download the predictions dataset file.
-
Alternatively the prediction file can be also viewed/downloaded from
outputs + logstab, expandOutputsto locatepredicted_true file.
-
Browse to https://ml.azure.com/. On the left navigation click on
Automated ML-> select an existing run to open therun detailspage.
-
Navigate to the
Modelstab of the run and click on a completedmodellink for which amodel testrun has to be created.
-
On the
model detailspage, click on theTest modelbutton to open theTest modelpanel.
-
On the
Test modelpanel, selectcompute clusterand atest datasetand click theTestbutton. The schema of the test dataset should match the training dataset (The Target column is optional).
-
On successful creation of
model test run, the details pages will be displayed with a success message. Click on theTest Resultstab to see the progress of the run. Click on the link to the test run to open its details page.
-
On the
model test run detailpage, after the run completes, click on the 'Output datasets' link to explore the predictions dataset.
-
On the
dataset detailspage, click onExploreto preview the predictions csv. Note: Work is in progress to easily download the predictions dataset file.
-
Alternatively the prediction file can be also viewed/downloaded from
outputs + logstab, expandOutputsto locatepredicted_true file.
-
The predictions.csv file which is generated by the model test run will be stored in the default datastore which gets created when the machine learning workspace is created (
workspaceblobstore). This datastore is visible to all users with the same subscription. Do not use this feature if any of the information used for or created by the test run needs to remain private.Workspaces and resource groups can be secured with "Access Control (IAM)". See the "Access Control (IAM)" section of the respective resource in the Azure portal.
-
Local AutoML runs do not support test runs. To use this feature, please use remote runs.
-
ADB/Spark runs do not support test runs.
-
Pandas DataFrames are not supported as input test datasets. Supported input test dataset types: Dataset.
-
Model testing feature is disabled when forecasting DNN models are used (non-DNN based forecasting is supported). However, the feature works with text DNNs, because AutoML uses text DNNs as a featurizer.
-
Forecasting runs do not support train/test split (
test_sizeparameter toAutoMLConfig). -
The on-demand model testing feature will not work for runs created with an SDK older than 1.29. To use this feature, the run must have been created with SDK version >= 1.29.
-
test_data/test_sizeis not supported at the same time ascv_split_column_names -
test_data/test_sizeis not supported at the same time ascv_splits_indices -
Test runs are not supported for AutoML runs which only have the following models in the
allowed_modelslist: [AveragedPerceptronClassifier, FastLinearRegressor, OnlineGradientDescentRegressor]. -
Model testing is not supported for image classification tasks.
We welcome contributions and suggestions! See CONTRIBUTING.md for more details.
All forms of feedback are welcome through this repo's issues: https://github.com/Azure/automl-testdataset-preview/issues
Please see the contributing guidelines for further details.
This project has adopted the Microsoft Open Source Code of Conduct. Please see the code of conduct for details.
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.
