diff --git a/docs/machine-learning/automate-training-with-cli.md b/docs/machine-learning/automate-training-with-cli.md index 853cf63c34110..07b14abe6d228 100644 --- a/docs/machine-learning/automate-training-with-cli.md +++ b/docs/machine-learning/automate-training-with-cli.md @@ -1,7 +1,7 @@ --- title: Automate model training with the ML.NET CLI description: Discover how to use the ML.NET CLI tool to automatically train the best model from the command-line. -ms.date: 06/03/2020 +ms.date: 11/10/2022 ms.custom: how-to, mlnet-tooling #Customer intent: As a developer, I want to use ML.NET CLI to automatically train the "best model" from the command-prompt. I also want to understand the output provided by the tool (metrics and output assets) --- @@ -71,7 +71,7 @@ The following image displays the classification metrics list for the top five mo Accuracy is a popular metric for classification problems, however accuracy isn't always the best metric to select the best model from as explained in the following references. There are cases where you need to evaluate the quality of your model with additional metrics. -To explore and understand the metrics that are output by the CLI, see [Evaluation metrics for classification](resources/metrics.md#evaluation-metrics-for-multi-class-classification). +To explore and understand the metrics that are output by the CLI, see [Evaluation metrics for classification](resources/metrics.md#evaluation-metrics-for-multi-class-classification-and-text-classification). ### Metrics for Regression and Recommendation models diff --git a/docs/machine-learning/automate-training-with-model-builder.md b/docs/machine-learning/automate-training-with-model-builder.md index e6a5c85554c1c..5171fc7870852 100644 --- a/docs/machine-learning/automate-training-with-model-builder.md +++ b/docs/machine-learning/automate-training-with-model-builder.md @@ -1,7 +1,7 @@ --- title: What is Model Builder and how does it work? description: How to use the ML.NET Model Builder to automatically train a machine learning model -ms.date: 10/12/2021 +ms.date: 11/10/2022 ms.custom: overview, mlnet-tooling #Customer intent: As a developer, I want to use Model Builder to automatically train a model using a visual interface. --- @@ -13,10 +13,7 @@ Model Builder uses automated machine learning (AutoML) to explore different mach You don't need machine learning expertise to use Model Builder. All you need is some data, and a problem to solve. Model Builder generates the code to add the model to your .NET application. -![Model Builder Scenarios](./media/model-builder-scenarios.png#lightbox) - -> [!NOTE] -> Model Builder is currently in Preview. +:::image type="content" source="media/model-builder-scenarios-2-0.png" alt-text="Model Builder scenario screen" lightbox="media/model-builder-scenarios-2-0.png"::: ## Creating a Model Builder Project @@ -45,14 +42,15 @@ A scenario is a description of the type of prediction you want to make using you Each scenario maps to a different Machine Learning Task which include: -- Binary classification -- Multiclass classification -- Regression -- Clustering -- Anomaly detection -- Ranking -- Recommendation -- Forecasting +| Task | Scenario | +| --- | --- | +| Binary classification | Data classification | +| Multiclass classification | Data classification | +| Image classification | Image classification | +| Text classification | Text classification | +| Regression | Value prediction | +| Recommendation | Recommendation | +| Forecasting | Forecasting | For example, the scenario of classifying sentiments as positive or negative would fall under the binary classification task. @@ -62,37 +60,196 @@ For more information about the different ML Tasks supported by ML.NET see [Machi In Model Builder, you need to select a scenario. The type of scenario depends on what type of prediction you are trying to make. -#### Data classification - -Classification is used to categorize data into categories. +#### Tabular -![Diagram showing examples of binary classification including fraud detection, risk mitigation, and application screening](media/binary-classification-examples.png) +##### Data classification -![Examples of multiclass classification including document and product classification, support ticket routing, and customer issue prioritization](media/multiclass-classification-examples.png) +Classification is used to categorize data into categories. -#### Value prediction + + +:::row::: + :::column::: + **Sample Input** + :::column-end::: + :::column::: + **Sample Output** + :::column-end::: +:::row-end::: +:::row::: + :::column::: + | SepalLength | SepalWidth | Petal Length | Petal Width | Species | + | --- | --- | --- | --- | --- | + | 5.1 | 3.5 | 1.4 | 0.2 | setosa | + :::column-end::: + :::column::: + | Predicted species | + | --- | + | setosa | + :::column-end::: +:::row-end::: + +##### Value prediction Value prediction, which falls under the regression task, is used to predict numbers. ![Diagram showing regression examples such as price prediction, sales forecasting, and predictive maintenance](media/regression-examples.png) -#### Image classification +:::row::: + :::column::: + **Sample Input** + :::column-end::: + :::column::: + **Sample Output** + :::column-end::: +:::row-end::: +:::row::: + :::column::: + | vendor_id | rate_code | passenger_count | trip_time_in_secs | trip_distance | payment_type | fare_amount | + | --- | --- | --- | --- | --- | --- | --- | + | CMT | 1 | 1 | 1271 | 3.8 | CRD | 17.5 | + :::column-end::: + :::column::: + | Predicted Fare | + | --- | + | 4.5 | + :::column-end::: +:::row-end::: + +#### Recommendation + +The recommendation scenario predicts a list of suggested items for a particular user, based on how similar their likes and dislikes are to other users'. + +You can use the recommendation scenario when you have a set of users and a set of "products", such as items to purchase, movies, books, or TV shows, along with a set of users' "ratings" of those products. + +:::row::: + :::column::: + **Sample Input** + :::column-end::: + :::column::: + **Sample Output** + :::column-end::: +:::row-end::: +:::row::: + :::column::: + | UserId | ProductId | Rating | + | --- | --- | --- | + | 1 | 2 | 4.2 | + :::column-end::: + :::column::: + | Predicted rating | + | --- | + | 4.5 | + :::column-end::: +:::row-end::: + +##### Forecasting + +The forecasting scenario uses historical data with a time-series or seasonal component to it. + +You can use the forecasting scenario to forecast demand or sale for a product. + +:::row::: + :::column::: + **Sample Input** + :::column-end::: + :::column::: + **Sample Output** + :::column-end::: +:::row-end::: +:::row::: + :::column::: + | Date | SaleQty | + | --- | --- | + | 1/1/1970 | 1000 | + :::column-end::: + :::column::: + | 3 Day Forecast | + | --- | + | [1000,1001,1002] | + :::column-end::: +:::row-end::: + +#### Computer Vision + +##### Image classification Image classification is used to identify images of different categories. For example, different types of terrain or animals or manufacturing defects. You can use the image classification scenario if you have a set of images, and you want to classify the images into different categories. -#### Object detection +:::row::: + :::column::: + **Sample Input** + :::column-end::: + :::column::: + **Sample Output** + :::column-end::: +:::row-end::: +:::row::: + :::column::: + :::image type="content" source="media/automate-training-with-model-builder/dog-classification.png" alt-text="Profile view of standing pug"::: + :::column-end::: + :::column::: + | Predicted Label | + | --- | + | Dog | + :::column-end::: +:::row-end::: + +##### Object detection Object detection is used to locate and categorize entities within images. For example, locating and identifying cars and people in an image. You can use object detection when images contain multiple objects of different types. -#### Recommendation - -The recommendation scenario predicts a list of suggested items for a particular user, based on how similar their likes and dislikes are to other users'. - -You can use the recommendation scenario when you have a set of users and a set of "products", such as items to purchase, movies, books, or TV shows, along with a set of users' "ratings" of those products. +:::row::: + :::column::: + **Sample Input** + :::column-end::: + :::column::: + **Sample Output** + :::column-end::: +:::row-end::: +:::row::: + :::column::: + :::image type="content" source="media/automate-training-with-model-builder/dog-classification.png" alt-text="Profile view of standing pug"::: + :::column-end::: + :::column::: + :::image type="content" source="media/automate-training-with-model-builder/dog-object-detection-min.png" alt-text="Profile view of standing pug with bounding box and dog label"::: + :::column-end::: +:::row-end::: + +#### Natural Language Processing + +##### Text classification + +Text classification categorizes raw text input. + +You can use the text classification scenario if you have a set of documents or comments, and you want to classify them into different categories. + +:::row::: + :::column::: + **Example Input** + :::column-end::: + :::column::: + **Example Output** + :::column-end::: +:::row-end::: +:::row::: + :::column::: + | Review | + | --- | + | I really like this steak!| + :::column-end::: + :::column::: + | Sentiment | + | --- | + | Positive | + :::column-end::: +:::row-end::: ## Environment @@ -100,11 +257,15 @@ You can train your machine learning model locally on your machine or in the clou When you train locally, you work within the constraints of your computer resources (CPU, memory, and disk). When you train in the cloud, you can scale up your resources to meet the demands of your scenario, especially for large datasets. -Local CPU training is supported for all scenarios except Object Detection. - -Local GPU training is supported for Image Classification. - -Azure training is supported for Image Classification and Object Detection. +| Scenario | Local CPU | Local GPU | Azure | +|-----------------------|------------|------------|--------| +| Data classification | ✔️ | ❌ | ❌ | +| Value prediction | ✔️ | ❌ | ❌ | +| Recommendation | ✔️ | ❌ | ❌ | +| Forecasting | ✔️ | ❌ | ❌ | +| Image classification | ✔️ | ✔️ | ✔️ | +| Object detection | ❌ | ❌ | ✔️ | +| Text classification | ✔️ | ✔️ | ❌ | ## Data diff --git a/docs/machine-learning/automated-machine-learning-mlnet.md b/docs/machine-learning/automated-machine-learning-mlnet.md new file mode 100644 index 0000000000000..7c18ab2a54b20 --- /dev/null +++ b/docs/machine-learning/automated-machine-learning-mlnet.md @@ -0,0 +1,71 @@ +--- +title: What is Automated Machine Learning (AutoML)? +description: Learn what automated machine learning is and its different components in ML.NETs +ms.date: 11/10/2022 +ms.topic: overview +ms.custom: mvc +--- + +# What is Automated Machine Learning (AutoML)? + +Automated machine learning (AutoML) automates the process of applying machine learning to data. Given a dataset, you can run AutoML to iterate over different data transformations, machine learning algorithms, and hyperparameters to select the best model. + +> [!NOTE] +> This topic refers to the ML.NET AutoML API, which is currently in preview. Material may be subject to change. + +## How does AutoML work? + +In general, the workflow to train machine learning models is as follows: + +- Define a problem +- Collect data +- Preprocess data +- Train a model +- Evaluate the model + +:::image type="content" source="media/ml-automl-workflow.png" alt-text="Traditional ML and AutoML training workflow" lightbox="media/ml-automl-workflow.png"::: + +Preprocessing, training, and evaluation are an experimental and iterative process that requires multiple trials until you achieve satisfactory results. Because these tasks tend to be repetitive, AutoML can help automate these steps. In addition to automation, optimization techniques are used during the training and evaluation process to find and select algorithms and hyperparameters. + +## When should I use AutoML? + +Whether you're just getting started with machine learning or you're an experienced user, AutoML provides solutions for automating the model development process. + +- **Beginners** - If you're new to machine learning, AutoML simplifies the model development process by providing a set of defaults that reduces the number of decisions you have to make when training your model. In doing so, you can focus on your data and the problem you're trying to solve and let AutoML do the rest. +- **Experienced users** - If you have some experience with machine learning, you can customize, configure, and extend the defaults provided by AutoML based on your needs while still leveraging its automation capabilities. + +## AutoML in ML.NET + +- **Featurizer** - Convenience API to automate data preprocessing. +- **Trial** - A single hyperparamters optimization run. +- **Experiment** - A collection of AutoML trials. ML.NET provides a high-level API for creating experiments which sets defaults for the individual Sweepable Pipeline, Search Space, and Tuner components. +- **Search Space** - The range of available options to choose hyperparameters from. +- **Tuner** - The algorithms used to optimize hyperparameters. ML.NET supports the following tuners: + - **Cost Frugal Tuner** - Implementation of [Frugal Optimization for Cost-related Hyperparameters](https://arxiv.org/abs/2005.01571) which takes training cost into consideration + - **Eci Cost Frugal Tuner** - Implementation of Cost Frugal Tuner for hierarchical search spaces. Default tuner used by AutoML. + - **SMAC** - Tuner that uses random forests to apply Bayesian optimization. + - **Grid Search** - Tuner that works best for small search spaces. + - **Random Search** +- **Sweepable Estimator** - An ML.NET estimator that contains a search space. +- **Sweepable Pipeline** - An ML.NET pipeline that contains one or more Sweepable Estimators. +- **Trial Runner** - AutoML component that uses sweepable pipelines and trial settings to generate trial results from model training and evaluation. + +It's recommended for beginners to start with the defaults provided by the high-level experiment API. For more experienced users looking for customization options, use the sweepable estimator, sweepable pipeline, search space, trial runner and tuner components. + +For more information on getting started with the AutoML API, see the [How to use the ML.NET Automated Machine Learning (AutoML) API](how-to-guides/how-to-use-the-automl-api.md) guide. + +## Supported tasks + +AutoML provides preconfigured defaults for the following tasks: + +- Binary classification +- Multiclass classification +- Regression + +For other tasks, you can build your own trial runner to enable those scenarios. For more information, see the [How to use the ML.NET Automated Machine Learning (AutoML) API](how-to-guides/how-to-use-the-automl-api.md) guide. + +## Next steps + +- [How to use the ML.NET Automated Machine Learning (AutoML) API](how-to-guides/how-to-use-the-automl-api.md) +- [Tutorial: Classify the severity of restaurant health violations with Model Builder](tutorials/health-violation-classification-model-builder.md) +- [Tutorial: Analyze sentiment using the ML.NET CLI](tutorials/sentiment-analysis-cli.md) diff --git a/docs/machine-learning/deep-learning-overview.md b/docs/machine-learning/deep-learning-overview.md new file mode 100644 index 0000000000000..8c3a866371b16 --- /dev/null +++ b/docs/machine-learning/deep-learning-overview.md @@ -0,0 +1,189 @@ +--- +title: Deep Learning overview +description: Learn what deep learning is and components in ML.NET +ms.date: 11/10/2022 +ms.topic: overview +ms.custom: mvc +--- + +# What is deep learning? + +Deep learning is an umbrella term for machine learning techniques that make use of "deep" neural networks. Today, deep learning is one of the most visible areas of machine learning because of its success in areas like Computer Vision, Natural Language Processing, and when applied to reinforcement learning, scenarios like game playing, decision making and simulation. + +A crucial element to the success of deep learning has been the availability of data, compute, software frameworks, and runtimes that facilitate the creation of neural network models and their execution for inference. Examples of such frameworks include Tensorflow, (Py)Torch and ONNX. + +ML.NET provides access to some of these frameworks. As a result, ML.NET users can take advantage of deep learning models without having to start from scratch. + +## Deep Learning vs Machine Learning? + +Deep learning relies on neural network algorithms. This is in contrast with traditional or classical machine learning techniques which use a wider variety of algorithms such as generalized linear models, decision trees or Support Vector Machines (SVM). The most immediate, practical implication of this difference is that deep learning methods may be better suited for some kind of data. In some cases, classical machine learning techniques such as gradient-boosted trees (XGBoost, LightGBM and CatBoost) seem to have an edge for tabular data. For less structured data like text and images, neural networks tend to perform better. The best approach is always to experiment with your particular data source and use case to determine for yourself which techniques work best for your problem. + +For classical machine learning tasks, ML.NET simplifies this experimentation process through Automated Machine Learning (AutoML). For more information on AutoML, see the article [what is Automated Machine Learning (AutoML)?](automated-machine-learning-mlnet.md). + +## Neural Network architectures + +One of the main differentiating characteristics of deep learning is the use of artificial neural network algorithms. At a high-level, you can think of neural networks as a configuration of "processing units" where the output of each unit constitutes the input of another. Each of these units can take one or many inputs, and essentially carries out a weighted sum of its inputs, applies an offset (or "bias") and then a non-linear transformation function (called "activation"). Different arrangements of these components have been used to describe decision boundaries in classification, regression functions and other structures central to machine learning tasks. + +:::image type="content" source="media/single-layer-neural-net.png" alt-text="Diagram representing single layer in neural network" lightbox="media/single-layer-neural-net.png"::: + +The past decade has seen an increase in cases, applications and techniques of deep learning. This increase is driven in part by an increasing variety of operations that can be incorporated into neural networks, a richer set of arrangements that these operations can be configured in and improved computational support for these improvements. In general, neural network architectures can be grouped into the following categories: + +* Feed-forward Neural Network +* Convolutional Neural Network +* Recurrent Neural Network +* Generative Adversarial Network +* Transformers + +For more details, see the [artificial neural networks guide](/azure/machine-learning/concept-deep-learning-vs-machine-learning). + +## What can I use deep learning for? + +Deep learning architectures, have shown good performance in tasks involving "unstructured data" such as images, audio, and free-form text. As a result, deep learning has been used to solve problems like: + +* Image classification +* Audio classification +* Translation +* Text generation +* Text classification + +## Deep learning in ML.NET + +Training a deep learning model from scratch requires setting several parameters, a large amount of labeled training data, and a vast amount of compute resources (hundreds of GPU hours). ML.NET enables you to shortcut this process by using pretrained models and knowledge transfer techniques such as transfer learning and fine-tuning. + +ML.NET also enables you to import models trained in other frameworks and consume them within your .NET applications. + +Depending on the scenario, you can use local GPU as well as Azure GPU compute resources to train and consume deep learning models. + +### Train custom models + +ML.NET provides APIs to train custom deep learning models and use them to make predictions inside your .NET application. + +These APIs are powered by [TorchSharp](https://github.com/dotnet/TorchSharp) and [TensorFlow.NET](https://github.com/SciSharp/TensorFlow.NET). + +#### Image classification + +In ML.NET you can use the set of APIs to train custom image classification models. + +An image classification training pipeline in ML.NET might look like the following: + +```csharp +//Append ImageClassification trainer to your pipeline containing any preprocessing transforms +pipeline + .Append(mlContext.MulticlassClassification.Trainers.ImageClassification(featureColumnName: "Image") + .Append(mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabel"); + +// Train your model +var model = pipeline.Fit(trainingData); + +// Use your model for inferencing +var predictedData = model.Transform(newData).GetColumn("PredictedLabel"); +``` + +To get started training custom image classification models in ML.NET, see the [Train an image classification model in Azure using Model Builder](tutorials/image-classification-model-builder.md) + +#### Object detection + +ML.NET enables you to train custom object detection models in Model Builder using Azure Machine Learning. All you need is a labelled dataset containing bounding box information and the categories the objects in the bounding boxes belong to. + +The result of the training process is an ONNX model which can then be used with the transform for to make predictions. + +At this time, there is no local support for object detection in ML.NET. + +To train custom object detection models with ML.NET, see the [Detect stop signs in images with Model Builder tutorial](tutorials/object-detection-model-builder.md) + +#### Text classification + +Classifying free-form text, whether that's customer reviews or business memos is an important part of many processes. + +In ML.NET, you can use the set of APIs to train custom text classification models. The technique used to train custom text classification models in ML.NET is known as fine-tuning. Fine-tuning enables you to take a pre-trained model and retrain the layers specific to your domain or problem using your own data. This gives you the benefit of having a model that’s more tailored to solve your problem without having to go through the process of training the entire model from scratch. The pretrained model used by the Text Classification API is a TorchSharp implementation of [NAS-BERT](https://dl.acm.org/doi/abs/10.1145/3447548.3467262). + +A text classification training pipeline in ML.NET might look like the following: + +```csharp +// Define training pipeline using TextClassification trainer +var pipeline = + mlContext.Transforms.Conversion.MapValueToKey("Label","Sentiment") + .Append(mlContext.MulticlassClassification.Trainers.TextClassification(sentence1ColumnName: "Text")) + .Append(mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabel")); + +// Train your model +var model = pipeline.Fit(trainingData); + +// Use your model to make predictions +var predictedData = model.Transform(newData).GetColumn("PredictedLabel"); +``` + +To get started training text classification models with ML.NET, see the [Analyze sentiment of website comments in a web application using ML.NET Model Builder tutorial](tutorials/sentiment-analysis-model-builder.md). + +#### Sentence Similarity + +Tasks such as semantic search rely on the determination of how similar two sentences or passages of text are to each other. + +ML.NET provides the set of APIs which use the same underlying model and fine-tuning techniques as the . However, instead of producing a category as output, it produces a numerical value representing how similar two passages are. + +A training and inference pipeline for sentence similarity in ML.NET might look like the following: + +```csharp +// Define your pipeline +var pipeline = mlContext.Regression.Trainers.SentenceSimilarity(sentence1ColumnName: "Sentence", sentence2ColumnName: "Sentence2"); + +// Train your model +var model = pipeline.Fit(trainingData); + +// Use the model to make predictions and extract their similarity values +var score = model.Transform(newData).GetColumn("Score"); +``` + +To get started with sentence similarity, see the samples in the [dotnet/machinelearning-samples repo](https://aka.ms/mlnet-2-samples). + +### Consume pretrained models + +ML.NET provides APIs to consume models in other formats like TensorFlow and ONNX and use them to make predictions inside your .NET application. + +These APIs are powered by [TensorFlow.NET](https://github.com/SciSharp/TensorFlow.NET) and the [ONNX Runtime](https://onnxruntime.ai/). + +#### TensorFlow + +[TensorFlow](https://www.tensorflow.org/) is a deep learning framework with a rich ecosystem and a variety of pre-trained models available in the [Tensorflow Hub](https://www.tensorflow.org/hub). + +With ML.NET, you can take these pretrained TensorFlow models and use them for inferencing inside your .NET applications. + +An inference pipeline using a pretrained TensorFlow model might look like the following: + +```csharp +// Load TensorFlow model +TensorFlowModel tensorFlowModel = mlContext.Model.LoadTensorFlowModel(_modelPath); + +//Append ScoreTensorFlowModel transform to your pipeline containing any preprocessing transforms +pipeline.Append(tensorFlowModel.ScoreTensorFlowModel(outputColumnName: "Prediction/Softmax", inputColumnName:"Features")) + +// Create ML.NET model from pipeline +ITransformer model = pipeline.Fit(dataView); + +var predictions = model.Transform(dataView).GetColumn("Prediction/Softmax"); +``` + +To get started consuming pretrained TensorFlow models with ML.NET, see the [movie reviews sentiment analysis using a pre-trained TensorFlow model in ML.NET tutorial](tutorials/text-classification-tf.md) + +#### ONNX + +The [Open Neural Network Exchange(ONNX)](https://onnx.ai/) is an open-source format designed to enable interoperability between machine learning and deep learning frameworks. This means you can train a model in one of the many popular machine learning frameworks like PyTorch, convert it into ONNX format and consume the ONNX model in a different framework like ML.NET. + +The [ONNX model repository](https://github.com/onnx/models) hosts several pre-trained ONNX models you can use for inferencing in a wide variety of tasks. + +With ML.NET, you can take these pretrained ONNX models and use them for inferencing inside your .NET applications. + +An inference pipeline using a pretrained ONNX model might look like the following: + +```csharp +// Append ApplyOnnxModel transform to pipeline containing any preprocessing transforms +pipeline.Append((modelFile: modelLocation, outputColumnNames: new[] { TinyYoloModelSettings.ModelOutput }, inputColumnNames: new[] { TinyYoloModelSettings.ModelInput }) + +// Create ML.NET model from pipeline +var model = pipeline.Fit(data); + +// Use the model to make predictions +var predictions = pipeline.Fit(data).GetColumn(TinyYoloModelSettings.ModelOutput); +``` + +To get started consuming pretrained ONNX models with ML.NET, see the [object detection using ONNX in ML.NET tutorial](tutorials/object-detection-onnx.md) diff --git a/docs/machine-learning/how-does-mldotnet-work.md b/docs/machine-learning/how-does-mldotnet-work.md index f12624665d7c8..b0d03280893be 100644 --- a/docs/machine-learning/how-does-mldotnet-work.md +++ b/docs/machine-learning/how-does-mldotnet-work.md @@ -1,7 +1,7 @@ --- title: What is ML.NET and how does it work? description: ML.NET gives you the ability to add machine learning to .NET applications, in either online or offline scenarios. With this capability, you can make automatic predictions using the data available to your application without having to be connected to a network to use ML.NET. This article explains the basics of machine learning in ML.NET. -ms.date: 08/12/2021 +ms.date: 11/10/2022 ms.topic: overview ms.custom: mvc #Customer intent: As a developer, I want to learn how ML.NET works so that I can leverage machine learning in my applications. @@ -67,6 +67,22 @@ Examples of the type of predictions that you can make with ML.NET: Categorize pathologies in medical images :::column-end::: :::row-end::: +:::row::: + :::column::: + **Text classification** + :::column-end::: + :::column::: + Categorize documents based on their content. + :::column-end::: +:::row-end::: +:::row::: + :::column::: + **Sentence similarity** + :::column-end::: + :::column::: + Measure how similar two sentences are. + :::column-end::: +:::row-end::: ## Hello ML.NET World diff --git a/docs/machine-learning/how-to-guides/how-to-use-the-automl-api.md b/docs/machine-learning/how-to-guides/how-to-use-the-automl-api.md index 45dd11ffe95de..0db64d647a720 100644 --- a/docs/machine-learning/how-to-guides/how-to-use-the-automl-api.md +++ b/docs/machine-learning/how-to-guides/how-to-use-the-automl-api.md @@ -1,239 +1,687 @@ --- -title: How to use the ML.NET automated ML API -description: The ML.NET automated ML API automates the model building process and generates a model ready for deployment. Learn the options that you can use to configure automated machine learning tasks. -ms.date: 10/05/2021 +title: How to use the ML.NET Automated ML (AutoML) API +description: The ML.NET Automated ML (AutoML) API automates the model building process and generates a model ready for deployment. Learn the options that you can use to configure automated machine learning tasks. +ms.date: 11/10/2022 ms.custom: mvc,how-to ms.topic: how-to --- -# How to use the ML.NET automated machine learning API +# How to use the ML.NET Automated Machine Learning (AutoML) API -Automated machine learning (AutoML) automates the process of applying machine learning to data. Given a dataset, you can run an AutoML **experiment** to iterate over different data featurizations, machine learning algorithms, and hyperparameters to select the best model. +In this article, you learn how to use the ML.NET Automated ML (AutoML API). + +Samples for the AutoML API can be found in the [dotnet/machinelearning-samples](https://aka.ms/mlnet-2-samples) repo. + +## Installation + +To use the AutoML API, install the [`Microsoft.ML.AutoML`](https://www.nuget.org/packages/Microsoft.ML.AutoML) NuGet package in the .NET project you want to reference it in. > [!NOTE] -> This topic refers to the automated machine learning API for ML.NET, which is currently in preview. Material may be subject to change. +> This guide uses version 0.20.0 and later of the `Microsoft.ML.AutoML` NuGet package. Although samples and code from earlier versions still work, it is highly recommended you use the APIs introduced in this version for new projects. + +For more information on installing NuGet packages, see the following guides: + +- [Install and use a NuGet package in Visual Studio](/nuget/quickstart/install-and-use-a-package-in-visual-studio) +- [Install and use a package in Visual Studio for Mac](/nuget/quickstart/install-and-use-a-package-in-visual-studio-mac) +- [Install and use a package (dotnet CLI)](/nuget/quickstart/install-and-use-a-package-using-the-dotnet-cli) + +## Quick Start + +AutoML provides several defaults for quickly training machine learning models. In this section you will learn how to: + +- Load your data +- Define your pipeline +- Configure your experiment +- Run your experiment +- Use the best model to make predictions + +### Define your problem + +Given a dataset stored in a comma-separated file called *taxi-fare-train.csv* that looks like the following: -## Load data +| vendor_id | rate_code | passenger_count | trip_time_in_secs | trip_distance|payment_type | fare_amount | +|---|---|---|---|---|---|---| +CMT|1|1|1271|3.8|CRD|17.5 +CMT|1|1|474|1.5|CRD|8 +CMT|1|1|637|1.4|CRD|8.5 -Automated machine learning supports loading a dataset into an [IDataView](xref:Microsoft.ML.IDataView). Data can be in the form of tab-separated value (TSV) files and comma separated value (CSV) files. +### Load your data -Example: +Start by inizializing your . `MLContext` is a starting point for all ML.NET operations. Initializing mlContext creates a new ML.NET environment that can be shared across the model creation workflow objects. It's similar, conceptually, to `DBContext` in Entity Framework. + +Then, to load your data, use the method. ```csharp -using Microsoft.ML; -using Microsoft.ML.AutoML; - // ... - MLContext mlContext = new MLContext(); - IDataView trainDataView = mlContext.Data.LoadFromTextFile("my-data-file.csv", hasHeader: true); +// Initialize MLContext +MLContext ctx = new MLContext(); + +// Define data path +var dataPath = Path.GetFullPath(@"..\..\..\..\Data\taxi-fare-train.csv"); + +// Infer column information +ColumnInferenceResults columnInference = + ctx.Auto().InferColumns(dataPath, labelColumnName: "fare_amount", groupColumns: false); ``` -## Select the machine learning task type + loads a few rows from the dataset. It then inspects the data and tries to guess or infer the data type for each of the columns based on their content. -Before creating an experiment, determine the kind of machine learning problem you want to solve. Automated machine learning supports the following ML tasks: +The default behavior is to group columns of the same type into feature vectors or arrays containing the elements for each of the individual columns. Setting `groupColumns` to `false` overrides that default behavior and only performs column inference without grouping columns. By keeping columns separate it allows you to apply different data transformations when preprocessing the data at the individual column level rather than the column grouping. -* Binary Classification -* Multiclass Classification -* Regression -* Recommendation -* Ranking +The result of is a object which contains the options needed to create a as well as column information. -## Create experiment settings +For the sample dataset in *taxi-fare-train.csv*, column information might look like the following: -Create experiment settings for the determined ML task type: +- **LabelColumnName**: fare_amount +- **CategoricalColumnNames**: vendor_id, payment_type +- **NumericColumnNames**: rate_code, passenger_count, trip_time_in_secs, trip_distance -* Binary Classification +Once you have your column information, use the defined by the to create a to load your data into an . - ```csharp - var experimentSettings = new BinaryExperimentSettings(); - ``` +```csharp +// Create text loader +TextLoader loader = ctx.Data.CreateTextLoader(columnInference.TextLoaderOptions); -* Multiclass Classification +// Load data into IDataView +IDataView data = loader.Load(dataPath); +``` - ```csharp - var experimentSettings = new MulticlassExperimentSettings(); - ``` +It's often good practice to split your data into train and validation sets. Use to create an 80% training and 20% validation split of your dataset. -* Regression +```csharp +TrainTestData trainValidationData = ctx.Data.TrainTestSplit(data, testFraction: 0.2); +``` - ```csharp - var experimentSettings = new RegressionExperimentSettings(); - ``` +### Define your pipeline -* Recommendation +Your pipeline defines the data processing steps and machine learning pipeline to use for training your model. - ```csharp - var experimentSettings = new RecommendationExperimentSettings(); - ``` +```csharp +SweepablePipeline pipeline = + ctx.Auto().Featurizer(data, columnInformation: columnInference.ColumnInformation) + .Append(ctx.Auto().Regression(labelColumnName: columnInference.ColumnInformation.LabelColumnName)); +``` -* Ranking +A is a collection of . A is an ML.NET with a . - ```csharp - var experimentSettings = new RankingExperimentSettings(); - ``` +The is a convenience API which builds a sweepable pipeline of data processing sweepable estimators based on the column information you provide. Instead of building a pipeline from scratch, automates the data preprocessing step. For more information on supported transforms by ML.NET, see the [data transformations guide](../resources/transforms.md). -## Configure experiment settings +The output is a single column containing a numerical feature vector representing the transformed data for each of the columns. This feature vector is then used as input for the algorithms used to train a machine learning model. -Experiments are highly configurable. See the [AutoML API docs](/dotnet/api/microsoft.ml.automl?view=ml-dotnet-preview&preserve-view=false) for a full list of configuration settings. +If you want finer control over your data preprocessing, you can create a pipeline with each of the individual preprocessing steps. For more information, see the [prepare data for building a model guide](prepare-data-ml-net.md). -Some examples include: +> [!TIP] +> Use with to maximize the utility of AutoML. -1. Specify the maximum time that the experiment is allowed to run. +For training, AutoML provides a sweepable pipeline with default trainers and search space configurations for the following machine learning tasks: - ```csharp - experimentSettings.MaxExperimentTimeInSeconds = 3600; - ``` +- +- +- -1. Use a cancellation token to cancel the experiment before it is scheduled to finish. +For the taxi fare prediction problem, since the goal is to predict a numerical value, use `Regression`. For more information on choosing a task, see the [machine learning tasks in ML.NET guide](/dotnet/machine-learning/resources/tasks.md). - ```csharp - experimentSettings.CancellationToken = cts.Token; +### Configure your experiment - // Cancel experiment after the user presses any key - CancelExperimentAfterAnyKeyPress(cts); - ``` +First, create an AutoML experiment. An is a collection of . -1. Specify a different optimizing metric. +```csharp +AutoMLExperiment experiment = ctx.Auto().CreateExperiment(); +``` - ```csharp - var experimentSettings = new RegressionExperimentSettings(); - experimentSettings.OptimizingMetric = RegressionMetric.MeanSquaredError; - ``` +Once your experiment is created, use the extension methods it provides to configure different settings. + +```csharp +experiment + .SetPipeline(pipeline) + .SetRegressionMetric(RegressionMetric.RSquared, labelColumn: columnInference.ColumnInformation.LabelColumnName) + .SetTrainingTimeInSeconds(60) + .SetDataset(trainValidationData); +``` -1. The `CacheDirectory` setting is a pointer to a directory where all models trained during the AutoML task will be saved. If `CacheDirectory` is set to null, models will be kept in memory instead of written to disk. +In this example, you: - ```csharp - experimentSettings.CacheDirectory = null; - ``` +- Set the sweepable pipeline to run during the experiment using . +- Choose `RSquared` as the metric to optimize during training using . For more information on evaluation metrics, see the [evaluate your ML.NET model with metrics](../resources/metrics.md) guide. +- Set 60 seconds as the amount of time you want to train for using . A good heuristic to determine how long to train for is the size of your data. Typically, larger datasets require longer training time. For more information, see [training time guidance](../automate-training-with-model-builder.md#how-long-should-i-train-for). +- Provide the training and validation datasets to use using . -1. Instruct automated ML not to use certain trainers. +Once your experiment is defined, you'll want some way to track its progress. The quickest way to track progress is by modifying the event from . - A default list of trainers to optimize are explored per task. This list can be modified for each experiment. For instance, trainers that run slowly on your dataset can be removed from the list. To optimize on one specific trainer call `experimentSettings.Trainers.Clear()`, then add the trainer that you want to use. +```csharp +// Log experiment trials +ctx.Log += (_, e) => { + if (e.Source.Equals("AutoMLExperiment")) + { + Console.WriteLine(e.RawMessage); + } +}; +``` - ```csharp - var experimentSettings = new RegressionExperimentSettings(); - experimentSettings.Trainers.Remove(RegressionTrainer.LbfgsPoissonRegression); - experimentSettings.Trainers.Remove(RegressionTrainer.OnlineGradientDescent); - ``` +### Run your experiment + +Now that you've defined your experiment, use the method to start your experiment. -The list of supported trainers per ML task can be found at the corresponding link below: +```csharp +TrialResult experimentResults = await experiment.RunAsync(); +``` -* [Supported Binary Classification Algorithms](xref:Microsoft.ML.AutoML.BinaryClassificationTrainer) -* [Supported Multiclass Classification Algorithms](xref:Microsoft.ML.AutoML.MulticlassClassificationTrainer) -* [Supported Regression Algorithms](xref:Microsoft.ML.AutoML.RegressionTrainer) -* [Supported Recommendation Algorithms](xref:Microsoft.ML.AutoML.RecommendationTrainer) -* [Supported Ranking Algorithms](xref:Microsoft.ML.AutoML.RankingTrainer) +Once the time to train expires, the result is a for the best model found during training. -## Optimizing metric +At this point, you can save your model or use it for making predictions. For more information on how use an ML.NET model, see the following guides: -The optimizing metric, as shown in the example above, determines the metric to be optimized during model training. The optimizing metric you can select is determined by the task type you choose. Below is a list of available metrics. +- [Save and load a trained model](/dotnet/machine-learning/how-to-guides/save-load-machine-learning-models-ml-net.md) +- [Make predictions with a trained model](/dotnet/machine-learning/how-to-guides/machine-learning-model-predictions-ml-net) -|[Binary Classification](xref:Microsoft.ML.AutoML.BinaryClassificationMetric) | [Multiclass Classification](xref:Microsoft.ML.AutoML.MulticlassClassificationMetric) | [Regression & Recommendation](xref:Microsoft.ML.AutoML.RegressionMetric) | [Ranking](xref:Microsoft.ML.AutoML.RankingMetric) | -|---------|---------|---------|---------| -|Accuracy | LogLoss | RSquared | Discounted Cumulative Gains | -|AreaUnderPrecisionRecallCurve| LogLossReduction| MeanAbsoluteError | Normalized Discounted Cumulative Gains | -|AreaUnderRocCurve | MacroAccuracy | MeanSquaredError | | -|F1Score | MicroAccuracy | RootMeanSquaredError | | -|NegativePrecision | TopKAccuracy | | | -|NegativeRecall | | | | -|PositivePrecision | | | | -|PositiveRecall | | | | +## Modify column inference results -## Data pre-processing and featurization +Because only loads a subset of your data, it's possible that edge cases contained outside of the samples used to infer columns are not caught and the wrong data types are set for your columns. In those cases, you can update the properties of to account for those cases where the column inference results are not correct. -> [!NOTE] -> The feature column only supported types of , , and . +For example, in the taxi fare dataset, the data in the `rate_code` column is a number. However, that numerical value represents a category. By default, calling will place `rate_code` in the `NumericColumnNames` property instead of `CategoricalColumnNames`. Because these properties are .NET collections, you can use standard operations to add and remove items from them. -Data pre-processing happens by default and the following steps are performed automatically for you: +You can do the following to update the for `rate_code`. -1. Drop features with no useful information +```csharp +columnInference.ColumnInformation.NumericColumnNames.Remove("rate_code"); +columnInference.ColumnInformation.CategoricalColumnNames.Add("rate_code"); +``` - Drop features with no useful information from training and validation sets. These include features with all values missing, same value across all rows or with extremely high cardinality (e.g., hashes, IDs or GUIDs). +## Exclude trainers -1. Missing value indication and imputation +By default, AutoML tries multiple trainers as part of the training process to see which one works best for your data. However, throughout the training process you might discover there are some trainers that use up too many compute resources or don't provide good evaluation metrics. You have the option to exclude trainers from the training process. Note that which trainers are used depends on the task. For a list of supported trainers in ML.NET, see the [Machine learning tasks in ML.NET guide](../resources/tasks.md). - Fill missing value cells with the default value for the datatype. Append indicator features with the same number of slots as the input column. The value in the appended indicator features is `1` if the value in the input column is missing and `0` otherwise. +For example, in the taxi fare regression scenario, to exclude the LightGBM algorithm, set the `useLgbm` parameter to `false`. -1. Generate additional features +```csharp +ctx.Auto().Regression(labelColumnName: columnInference.ColumnInformation.LabelColumnName, useLgbm:false) +``` - For text features: Bag-of-word features using unigrams and tri-character-grams. +The process for excluding trainers in other tasks like binary and multiclass classification works the same way. - For categorical features: One-hot encoding for low cardinality features, and one-hot-hash encoding for high cardinality categorical features. +## Customize a sweepable estimator -1. Transformations and encodings +When you want to more granular customization of estimator options included as part of your sweepable pipeline, you need to: - Text features with very few unique values transformed into categorical features. Depending on cardinality of categorical features, perform one-hot encoding or one-hot hash encoding. +1. Initialize a search space +1. Use the search space to define a custom factory +1. Create a sweepable estimator +1. Add your sweepable estimator to your sweepable pipeline -## Exit criteria +AutoML provides a set of preconfigured search spaces for trainers in the following machine learning tasks: -Define the criteria to complete your task: +- +- +- -1. Exit after a length of time - Using `MaxExperimentTimeInSeconds` in your experiment settings you can define how long in seconds that a task should continue to run. +In this example, the search space used is for the . Initialize it by using . -1. Exit on a cancellation token - You can use a cancellation token that lets you cancel the task before it is scheduled to finish. +```csharp +var sdcaSearchSpace = new SearchSpace(); +``` - ```csharp - var cts = new CancellationTokenSource(); - var experimentSettings = new RegressionExperimentSettings(); - experimentSettings.MaxExperimentTimeInSeconds = 3600; - experimentSettings.CancellationToken = cts.Token; - ``` +Then, use the search space to define a custom factory method to create the . In this example, the values of `L1Regularization` and `L2Regularization` are both being set to something other than the default. For `L1Regularization`, the value set is determined by the tuner during each trial. The `L2Regularization` is fixed for each trial to the hard-coded value. During each trial, the custom factory's output is an with the configured hyperparameters. + +```csharp +// Use the search space to define a custom factory to create an SdcaRegressionTrainer +var sdcaFactory = (MLContext ctx, SdcaOption param) => +{ + var sdcaOption = new SdcaRegressionTrainer.Options(); + sdcaOption.L1Regularization = param.L1Regularization; + sdcaOption.L2Regularization = 0.02f; -## Create an experiment + sdcaOption.LabelColumnName = columnInference.ColumnInformation.LabelColumnName; + + return ctx.Regression.Trainers.Sdca(sdcaOption); +}; +``` + +A sweepable estimator is the combination of an estimator and a search space. Now that you've defined a search space and used it to create a custom factory method for generating trainers, use the method to create a new sweepable estimator. + +```csharp +// Define Sdca sweepable estimator (SdcaRegressionTrainer + SdcaOption search space) +var sdcaSweepableEstimator = ctx.Auto().CreateSweepableEstimator(sdcaFactory, sdcaSearchSpace); +``` -Once you have configured the experiment settings, you are ready to create the experiment. +To use your sweepable estimator in your experiment, add it to your sweepable pipeline. ```csharp -RegressionExperiment experiment = mlContext.Auto().CreateRegressionExperiment(experimentSettings); +SweepablePipeline pipeline = + ctx.Auto().Featurizer(data, columnInformation: columnInference.ColumnInformation) + .Append(sdcaSweepableEstimator); ``` -## Run the experiment +Because sweepable pipelines are a collection of sweepable estimators, you can configure and customize as many of these sweepable estimators as you need. + +## Customize your search space + +There are scenarios where you want to go beyond customizing the sweepable estimators used in your experiment and want control the search space range. You can do so by accessing the search space properties using keys. In this case, the `L1Regularization` parameter is a `float`. Therefore, to customize the search range, use . + +```csharp +sdcaSearchSpace["L1Regularization"] = new UniformSingleOption(min: 0.01f, max: 2.0f, logBase: false, defaultValue: 0.01f); +``` + +Depending on the data type of the hyperparameter you want to set, you can choose from the following options: + +- Numbers + - + - + - +- Booleans and strings + - + +Search spaces can contain nested search spaces as well. + +```csharp +var searchSpace = new SearchSpace(); +searchSpace["SingleOption"] = new UniformSingleOption(min:-10f, max:10f, defaultValue=0f) +var nestedSearchSpace = new SearchSpace(); +nestedSearchSpace["IntOption"] = new UniformIntOption(min:-10, max:10, defaultValue=0); +searchSpace["Nest"] = nestedSearchSpace; +``` + +Another option for customizing search ranges is by extending them. For example, only provides the `L1Regularization` and `L2Regularization` parameters. However, has more parameters you can set such as `BiasLearningRate`. + +To extend the search space, create a new class such as `SdcaExtendedOption` which inherits from . + +```csharp +public class SdcaExtendedOption : SdcaOption +{ + [Range(0.10f, 1f, 0.01f)] + public float BiasLearningRate {get;set;} +} +``` + +To specify the search space range, use which is equivalent to . + +Then, anywhere you use your search space, reference the `SdcaExtendedOption` instead of . -Running the experiment triggers data pre-processing, learning algorithm selection, and hyperparameter tuning. AutoML will continue to generate combinations of featurization, learning algorithms, and hyperparameters until the `MaxExperimentTimeInSeconds` is reached or the experiment is terminated. +For example, when you initialize your search space, you can do so as follows: ```csharp -ExperimentResult experimentResult = experiment - .Execute(trainingDataView, LabelColumnName, progressHandler: progressHandler); +var sdcaSearchSpace = new SearchSpace(); ``` -Explore other overloads for `Execute()` if you want to pass in validation data, column information indicating the column purpose, or prefeaturizers. +## Create your own trial runner -## Training modes +By default, AutoML supports binary classification, multiclass classification, and regression. However, ML.NET supports many more scenarios such as: -### Training dataset +- Recommendation +- Forecasting +- Ranking +- Image classification +- Text classification +- Sentence similarity -AutoML provides an overloaded experiment execute method which allows you to provide training data. Internally, automated ML divides the data into train-validate splits. +For scenarios that don't have preconfigured search spaces and sweepable estimators you can create your own and use a trial runner to enable AutoML for that scenario. + +For example, given restaurant review data that looks like the following: + +:::row::: + :::column::: + Wow... Loved this place. + :::column-end::: + :::column::: + 1 + :::column-end::: +:::row-end::: +:::row::: + :::column::: + Crust is not good. + :::column-end::: + :::column::: + 0 + :::column-end::: +:::row-end::: + +You want to use the trainer to analyze sentiment where 0 is negative and 1 is positive. However, there is no `ctx.Auto().TextClassification()` configuration. + +In order to use AutoML with the text classification trainer you'll have to: + +1. Create your own search space. + + ```csharp + // Define TextClassification search space + public class TCOption + { + [Range(64, 128, 32)] + public int BatchSize { get; set; } + } + ``` + + In this case, AutoML will search for different configurations of the `BatchSize` hyperparameter. + +1. Create a sweepable estimator and add it to your pipeline. + + ```csharp + // Initialize serach space + var tcSearchSpace = new SearchSpace(); + + // Create factory for Text Classification trainer + var tcFactory = (MLContext ctx, TCOption param) => + { + return ctx.MulticlassClassification.Trainers.TextClassification( + sentence1ColumnName: textColumnName, + batchSize:param.BatchSize); + }; + + // Create text classification sweepable estimator + var tcEstimator = + ctx.Auto().CreateSweepableEstimator(tcFactory, tcSearchSpace); + + // Define text classification pipeline + var pipeline = + ctx.Transforms.Conversion.MapValueToKey(columnInference.ColumnInformation.LabelColumnName) + .Append(tcEstimator); + ``` + + In this example, the `TCOption` search space and a custom factory are used to create a sweepable estimator. + +1. Create a custom trial runner + + To create a custom trial runner, implement : + + ```csharp + public class TCRunner : ITrialRunner + { + private readonly MLContext _context; + private readonly TrainTestData _data; + private readonly IDataView _trainDataset; + private readonly IDataView _evaluateDataset; + private readonly SweepablePipeline _pipeline; + private readonly string _labelColumnName; + private readonly MulticlassClassificationMetric _metric; + + public TCRunner( + MLContext context, + TrainTestData data, + SweepablePipeline pipeline, + string labelColumnName = "Label", + MulticlassClassificationMetric metric = MulticlassClassificationMetric.MicroAccuracy) + { + _context = context; + _data = data; + _trainDataset = data.TrainSet; + _evaluateDataset = data.TestSet; + _labelColumnName = labelColumnName; + _pipeline = pipeline; + _metric = metric; + } + + public void Dispose() + { + return; + } + + // Run trial asynchronously + public Task RunAsync(TrialSettings settings, CancellationToken ct) + { + try + { + return Task.Run(() => Run(settings)); + } + catch (Exception ex) when (ct.IsCancellationRequested) + { + throw new OperationCanceledException(ex.Message, ex.InnerException); + } + catch (Exception) + { + throw; + } + } + + // Helper function to define trial run logic + private TrialResult Run(TrialSettings settings) + { + try + { + // Initialize stop watch to measure time + var stopWatch = new Stopwatch(); + stopWatch.Start(); + + // Get pipeline parameters + var parameter = settings.Parameter["_pipeline_"]; + + // Use parameters to build pipeline + var pipeline = _pipeline.BuildFromOption(_context, parameter); + + // Train model + var model = pipeline.Fit(_trainDataset); + + // Evaluate the model + var predictions = model.Transform(_evaluateDataset); + + // Get metrics + var evaluationMetrics = _context.MulticlassClassification.Evaluate(predictions, labelColumnName: _labelColumnName); + var chosenMetric = GetMetric(evaluationMetrics); + + return new TrialResult() + { + Metric = chosenMetric, + Model = model, + TrialSettings = settings, + DurationInMilliseconds = stopWatch.ElapsedMilliseconds + }; + } + catch (Exception) + { + return new TrialResult() + { + Metric = double.MinValue, + Model = null, + TrialSettings = settings, + DurationInMilliseconds = 0, + }; + } + } + + // Helper function to choose metric used by experiment + private double GetMetric(MulticlassClassificationMetrics metric) + { + return _metric switch + { + MulticlassClassificationMetric.MacroAccuracy => metric.MacroAccuracy, + MulticlassClassificationMetric.MicroAccuracy => metric.MicroAccuracy, + MulticlassClassificationMetric.LogLoss => metric.LogLoss, + MulticlassClassificationMetric.LogLossReduction => metric.LogLossReduction, + MulticlassClassificationMetric.TopKAccuracy => metric.TopKAccuracy, + _ => throw new NotImplementedException(), + }; + } + } + ``` + + The `TCRunner` implementation in this example: + - Extracts the hyperparameters chosen for that trial + - Uses the hyperparameters to create an ML.NET pipeline + - Uses the ML.NET pipeline to train a model + - Evaluates the model + - Returns a object with the information for that trial + +1. Initialize your custom trial runner + + ```csharp + var tcRunner = new TCRunner(context: ctx, data: trainValidationData, pipeline: pipeline); + ``` + +1. Create and configure your experiment. Use the extension method to add your custom trial runner to your experiment. + + ```csharp + AutoMLExperiment experiment = ctx.Auto().CreateExperiment(); + + // Configure AutoML experiment + experiment + .SetPipeline(pipeline) + .SetMulticlassClassificationMetric(MulticlassClassificationMetric.MicroAccuracy, labelColumn: columnInference.ColumnInformation.LabelColumnName) + .SetTrainingTimeInSeconds(120) + .SetDataset(trainValidationData) + .SetTrialRunner(tcRunner); + ``` + +1. Run your experiment + + ```csharp + var tcCts = new CancellationTokenSource(); + TrialResult textClassificationExperimentResults = await experiment.RunAsync(tcCts.Token); + ``` + +## Choose a different tuner + +AutoML supports a variety of tuning algorithms to iterate through the search space in search of the optimal hyperparameters. By default, it uses the Eci Cost Frugal tuner. Using experiment extension methods, you can choose another tuner that best fits your scenario. + +Use the following methods to set your tuner: + +- **SMAC** - +- **Grid Search** - +- **Random Search** - +- **Cost Frugal** - +- **Eci Cost Frugal** - + +For example, to use the grid search tuner, your code might look like the following: ```csharp -experiment.Execute(trainDataView); +experiment.SetGridSearchTuner(); ``` -### Custom validation dataset +## Configure experiment monitoring -Use custom validation dataset if random split is not acceptable, as is usually the case with time series data. You can specify your own validation dataset. The model will be evaluated against the validation dataset specified instead of one or more random datasets. +The quickest way to monitor the progress of an experiment is to define the event from . However, the event outputs a raw dump of the logs generated by AutoML during each trial. Because of the large amount of unformatted information, it's difficult. + +For a more controlled monitoring experience, implement a class with the interface. ```csharp -experiment.Execute(trainDataView, validationDataView); +public class AutoMLMonitor : IMonitor +{ + private readonly SweepablePipeline _pipeline; + + public AutoMLMonitor(SweepablePipeline pipeline) + { + _pipeline = pipeline; + } + + public IEnumerable GetCompletedTrials() => _completedTrials; + + public void ReportBestTrial(TrialResult result) + { + return; + } + + public void ReportCompletedTrial(TrialResult result) + { + var trialId = result.TrialSettings.TrialId; + var timeToTrain = result.DurationInMilliseconds; + var pipeline = _pipeline.ToString(result.TrialSettings.Parameter); + Console.WriteLine($"Trial {trialId} finished training in {timeToTrain}ms with pipeline {pipeline}"); + } + + public void ReportFailTrial(TrialSettings settings, Exception exception = null) + { + if (exception.Message.Contains("Operation was canceled.")) + { + Console.WriteLine($"{settings.TrialId} cancelled. Time budget exceeded."); + } + Console.WriteLine($"{settings.TrialId} failed with exception {exception.Message}"); + } + + public void ReportRunningTrial(TrialSettings setting) + { + return; + } +} ``` -## Explore model metrics +The interface has four lifecycle events: + +- +- +- +- + +> [!TIP] +> Although it's not required, include your in your monitor so you can inspect the pipeline that was generated for a trial using the property of the . + +In this example, only the and are implemented. + +Once you've implemented your monitor, set it as part of your experiment configuration using . -After each iteration of an ML experiment, metrics relating to that task are stored. +```csharp +var monitor = new AutoMLMonitor(pipeline); +experiment.SetMonitor(monitor); +``` -For example, we can access validation metrics from the best run: +Then, run your experiment: ```csharp -RegressionMetrics metrics = experimentResult.BestRun.ValidationMetrics; -Console.WriteLine($"R-Squared: {metrics.RSquared:0.##}"); -Console.WriteLine($"Root Mean Squared Error: {metrics.RootMeanSquaredError:0.##}"); +var cts = new CancellationTokenSource(); +TrialResult experimentResults = await experiment.RunAsync(cts.Token); +``` + +When you run the experiment with this implementation, the output should look similar to the following: + +```text +Trial 0 finished training in 5835ms with pipeline ReplaceMissingValues=>OneHotEncoding=>Concatenate=>FastForestRegression +Trial 1 finished training in 15080ms with pipeline ReplaceMissingValues=>OneHotEncoding=>Concatenate=>SdcaRegression +Trial 2 finished training in 3941ms with pipeline ReplaceMissingValues=>OneHotHashEncoding=>Concatenate=>FastTreeRegression ``` -The following are all the available metrics per ML task: +## Persist trials -* [Binary classification metrics](xref:Microsoft.ML.AutoML.BinaryClassificationMetric) -* [Multiclass classification metrics](xref:Microsoft.ML.AutoML.MulticlassClassificationMetric) -* [Regression & recommendation metrics](xref:Microsoft.ML.AutoML.RegressionMetric) -* [Ranking](xref:Microsoft.ML.AutoML.RankingMetric) +By default, AutoML only stores the for the best model. However, if you wanted to persist each of the trials, you can do so from within your monitor. + +Inside your monitor: + +1. Define a property for your completed trials and a method for accessing them. + + ```csharp + private readonly List _completedTrials; + + public IEnumerable GetCompletedTrials() => _completedTrials; + ``` + +1. Initialize it in your constructor + + ```csharp + public AutoMLMonitor(SweepablePipeline pipeline) + { + //... + _completedTrials = new List(); + //... + } + ``` -## See also +1. Append each trial result inside your lifecycle method. -For full code samples and more visit the [dotnet/machinelearning-samples](https://github.com/dotnet/machinelearning-samples/tree/main#automate-mlnet-models-generation-preview-state) GitHub repository. + ```csharp + public void ReportCompletedTrial(TrialResult result) + { + //... + _completedTrials.Add(result); + } + ``` + +1. When training completes, you can access all the completed trials by calling `GetCompletedTrials` + + ```csharp + var completedTrials = monitor.GetCompletedTrials(); + ``` + +At this point you can perform additional processing on the collection of completed trials like choosing another model other than the one selected by AutoML, logging trial results to a database, or rebuilding the pipeline from any of the completed trials. + +## Cancel experiments + +When you run experiments asynchronously, make sure to cleanly terminate the process. To do so, use a . + +> [!WARNING] +> Cancelling an experiment will not save any of the intermediary outputs. Set a checkpoint to save intermediary outputs. + +```csharp +var cts = new CancellationTokenSource(); +TrialResult experimentResults = await experiment.RunAsync(cts.Token); +``` + +## Set checkpoints + +Checkpoints provide a way for you to save intermediary outputs from the training process in the event of an early termination or error. To set a checkpoint, use the extension method and provide a directory to store the intermediary outputs. + +```csharp +var checkpointPath = Path.Join(Directory.GetCurrentDirectory(), "automl"); +experiment.SetCheckpoint(checkpointPath); +``` diff --git a/docs/machine-learning/how-to-guides/install-gpu-model-builder.md b/docs/machine-learning/how-to-guides/install-gpu-model-builder.md index 275d5761e5220..3cd1e5096be9c 100644 --- a/docs/machine-learning/how-to-guides/install-gpu-model-builder.md +++ b/docs/machine-learning/how-to-guides/install-gpu-model-builder.md @@ -1,7 +1,7 @@ --- title: How to install GPU support in Model Builder description: Learn how to install GPU support in Model Builder -ms.date: 09/02/2021 +ms.date: 11/10/2022 author: luisquintanilla ms.author: luquinta ms.topic: how-to @@ -12,14 +12,21 @@ ms.topic: how-to Learn how to install the GPU drivers to use your GPU with Model Builder. -## Prerequisites +## Hardware requirements -- Model Builder Visual Studio extension. The extension is built into Visual Studio as of version 16.6.1. -- [Model Builder Visual Studio GPU support extension](https://marketplace.visualstudio.com/items?itemName=MLNET.ModelBuilderGPU2022). - At least one CUDA compatible GPU. For a list of compatible GPUs, see [NVIDIA's guide](https://developer.nvidia.com/cuda-gpus). +- At least 6GB of dedicated GPU memory. + +## Prerequisites + +- [Model Builder Visual Studio extension](install-model-builder.md). The extension is built into Visual Studio as of version 16.6.1. - NVIDIA developer account. If you don't have one, [create a free account](https://developer.nvidia.com/developer-program). - Make sure the appropriate [driver](https://www.nvidia.com/drivers) is installed for the GPU. +**Image classification only** + +- [Model Builder Visual Studio GPU support extension](https://marketplace.visualstudio.com/items?itemName=MLNET.ModelBuilderGPU2022) + ## Install dependencies 1. Install [CUDA v10.1](https://developer.nvidia.com/cuda-10.1-download-archive-update2). Make sure you install CUDA v10.1, not any other newer version. @@ -27,9 +34,15 @@ Learn how to install the GPU drivers to use your GPU with Model Builder. ## Troubleshooting -**How do I know what GPU I have?** +**What if I don't have a GPU installed locally?** -Follow instructions provided: +Deep learning scenarios tend to run faster on GPUs. + +Some scenarios like image classification support training on Azure GPU VMs. + +However, if local GPUs or Azure are not an option for you, these scenarios also run on CPU. Note however that training times are significantly longer. + +**How do I know what GPU I have?** 1. Right-click on desktop 1. If you see "NVIDIA Control Panel" or "NVIDIA Display" in the pop-up window, you have an NVIDIA GPU @@ -47,3 +60,9 @@ Follow instructions provided: 1. Open a PowerShell or command line window 1. Type in `nvcc --version` + +**cuda is not available, please confirm you have a cuda-supported gpu** + +1. Open the [GeForce Experience](https://www.nvidia.com/geforce/geforce-experience/) app. +1. The application should show installed and available driver updates. If you have trouble seeing updates, you can get the latest drivers from [https://www.nvidia.com/geforce/drivers/](https://www.nvidia.com/geforce/drivers/). +1. Install the latest drivers. diff --git a/docs/machine-learning/media/automate-training-with-model-builder/dog-classification.png b/docs/machine-learning/media/automate-training-with-model-builder/dog-classification.png new file mode 100644 index 0000000000000..bdbf290e024f4 Binary files /dev/null and b/docs/machine-learning/media/automate-training-with-model-builder/dog-classification.png differ diff --git a/docs/machine-learning/media/automate-training-with-model-builder/dog-object-detection-min.png b/docs/machine-learning/media/automate-training-with-model-builder/dog-object-detection-min.png new file mode 100644 index 0000000000000..c7207c71be379 Binary files /dev/null and b/docs/machine-learning/media/automate-training-with-model-builder/dog-object-detection-min.png differ diff --git a/docs/machine-learning/media/automate-training-with-model-builder/dog-object-detection.png b/docs/machine-learning/media/automate-training-with-model-builder/dog-object-detection.png new file mode 100644 index 0000000000000..dd21fde3dba81 Binary files /dev/null and b/docs/machine-learning/media/automate-training-with-model-builder/dog-object-detection.png differ diff --git a/docs/machine-learning/media/ml-automl-workflow.png b/docs/machine-learning/media/ml-automl-workflow.png new file mode 100644 index 0000000000000..b0dba67f69b69 Binary files /dev/null and b/docs/machine-learning/media/ml-automl-workflow.png differ diff --git a/docs/machine-learning/media/model-builder-scenarios-2-0.png b/docs/machine-learning/media/model-builder-scenarios-2-0.png new file mode 100644 index 0000000000000..00dc20fc31f9d Binary files /dev/null and b/docs/machine-learning/media/model-builder-scenarios-2-0.png differ diff --git a/docs/machine-learning/media/single-layer-neural-net.png b/docs/machine-learning/media/single-layer-neural-net.png new file mode 100644 index 0000000000000..ee207dad2e414 Binary files /dev/null and b/docs/machine-learning/media/single-layer-neural-net.png differ diff --git a/docs/machine-learning/resources/metrics.md b/docs/machine-learning/resources/metrics.md index 47dc6703cffb8..3281192bf4a9b 100644 --- a/docs/machine-learning/resources/metrics.md +++ b/docs/machine-learning/resources/metrics.md @@ -1,7 +1,7 @@ --- title: ML.NET metrics description: Understand the metrics that are used to evaluate the performance of an ML.NET model -ms.date: 12/17/2019 +ms.date: 11/10/2022 --- # Evaluate your ML.NET model with metrics @@ -26,7 +26,7 @@ For further details on binary classification metrics read the following articles - [Binary Classification Metrics class](xref:Microsoft.ML.Data.BinaryClassificationMetrics) - [The Relationship Between Precision-Recall and ROC Curves](http://pages.cs.wisc.edu/~jdavis/davisgoadrichcamera2.pdf) -## Evaluation metrics for Multi-class Classification +## Evaluation metrics for Multi-class Classification and text classification | Metrics | Description | Look for | |-----------|-----------------------|-----------| diff --git a/docs/machine-learning/toc.yml b/docs/machine-learning/toc.yml index af9d83d387c4a..425b66fc3bd76 100644 --- a/docs/machine-learning/toc.yml +++ b/docs/machine-learning/toc.yml @@ -13,21 +13,23 @@ items: - name: The ML.NET API href: how-does-mldotnet-work.md + - name: What is Automated Machine Learning (AutoML)? + href: automated-machine-learning-mlnet.md - name: Tutorials items: - name: Model Builder & CLI items: - - name: Predict prices (Model Builder & regression) + - name: Predict prices (Model Builder value prediction) href: tutorials/predict-prices-with-model-builder.md - name: Analyze sentiment using the ML.NET CLI href: tutorials/sentiment-analysis-cli.md - - name: Analyze sentiment in Razor Pages (Model Builder & binary classification) + - name: Analyze sentiment in Razor Pages (Model Builder text classification) href: tutorials/sentiment-analysis-model-builder.md - - name: Categorize health violations (Model Builder & multiclass classification) + - name: Categorize health violations (Model Builder data classification) href: tutorials/health-violation-classification-model-builder.md - - name: Categorize land use from satellite images (Model Builder & image classification) + - name: Categorize land use from satellite images (Model Builder image classification) href: tutorials/image-classification-model-builder.md - - name: Detect traffic signs (Model Builder & object detection) + - name: Detect traffic signs (Model Builder object detection) href: tutorials/object-detection-model-builder.md - name: Train a recommendation model using Model Builder href: tutorials/movie-recommendation-model-builder.md @@ -67,10 +69,12 @@ items: - name: ML.NET tasks href: resources/tasks.md - - name: Algorithms - href: how-to-choose-an-ml-net-algorithm.md - name: Data transforms href: resources/transforms.md + - name: Algorithms + href: how-to-choose-an-ml-net-algorithm.md + - name: Deep learning overview + href: deep-learning-overview.md - name: Model evaluation metrics href: resources/metrics.md - name: Improve model accuracy @@ -89,8 +93,6 @@ href: how-to-guides/install-gpu-model-builder.md - name: Install the CLI href: how-to-guides/install-ml-net-cli.md - - name: Use the automated ML API - href: how-to-guides/how-to-use-the-automl-api.md - name: API items: - name: Install extra dependencies @@ -103,6 +105,8 @@ items: - name: Train and evaluate a model href: how-to-guides/train-machine-learning-model-ml-net.md + - name: Use the automated ML API + href: how-to-guides/how-to-use-the-automl-api.md - name: Train a model using cross-validation href: how-to-guides/train-machine-learning-model-cross-validation-ml-net.md - name: Inspect intermediate pipeline data values diff --git a/docs/machine-learning/tutorials/health-violation-classification-model-builder.md b/docs/machine-learning/tutorials/health-violation-classification-model-builder.md index 8cfda621593fc..bf1843ab80e8a 100644 --- a/docs/machine-learning/tutorials/health-violation-classification-model-builder.md +++ b/docs/machine-learning/tutorials/health-violation-classification-model-builder.md @@ -3,7 +3,7 @@ title: 'Tutorial: Classify health violations with Model Builder' description: This tutorial illustrates how to build a multiclass classification model using ML.NET Model Builder to classify restaurant health violation severity in San Francisco. author: luisquintanilla ms.author: luquinta -ms.date: 09/20/2021 +ms.date: 11/10/2022 ms.topic: tutorial ms.custom: mvc,mlnet-tooling #Customer intent: As a non-developer, I want to use Model Builder to automatically generate a model to classify violation severity using Model Builder. @@ -25,9 +25,6 @@ In this tutorial, you learn how to: > - Evaluate the model > - Use the model for predictions -> [!NOTE] -> Model Builder is currently in Preview. - ## Prerequisites For a list of prerequisites and installation instructions, visit the [Model Builder installation guide](../how-to-guides/install-model-builder.md). @@ -69,7 +66,7 @@ When first adding Model Builder to the solution it will prompt you to create an ## Choose a scenario -![Model Builder wizard in Visual Studio](../media/model-builder-scenarios.png) +:::image type="content" source="../media/model-builder-scenarios-2-0.png" alt-text="Model Builder Scenario Screen" lightbox="../media/model-builder-scenarios-2-0.png"::: To train your model, select from the list of available machine learning scenarios provided by Model Builder. In this case, the scenario is *Data classification*. @@ -179,4 +176,4 @@ To learn more about topics mentioned in this tutorial, visit the following resou - [Model Builder Scenarios](../automate-training-with-model-builder.md#scenario) - [Multiclass Classification](../resources/glossary.md#multiclass-classification) -- [Multiclass Classification Model Metrics](../resources/metrics.md#evaluation-metrics-for-multi-class-classification) +- [Multiclass Classification Model Metrics](../resources/metrics.md#evaluation-metrics-for-multi-class-classification-and-text-classification) diff --git a/docs/machine-learning/tutorials/image-classification-model-builder.md b/docs/machine-learning/tutorials/image-classification-model-builder.md index bfa6a6ef5225f..ef2e9c9712607 100644 --- a/docs/machine-learning/tutorials/image-classification-model-builder.md +++ b/docs/machine-learning/tutorials/image-classification-model-builder.md @@ -28,9 +28,6 @@ In this tutorial, you: > - Evaluate the model > - Consume the model -> [!NOTE] -> Model Builder is currently in Preview. - ## Prerequisites - For a list of pre-requisites and installation instructions, visit the [Model Builder installation guide](../how-to-guides/install-model-builder.md). @@ -69,7 +66,7 @@ When first adding Model Builder to the solution it will prompt you to create an ## Choose a scenario -![Model Builder Scenarios](../media/model-builder-scenarios.png) +:::image type="content" source="../media/model-builder-scenarios-2-0.png" alt-text="Model Builder Scenario Screen" lightbox="../media/model-builder-scenarios-2-0.png"::: To train your model, you need to select from the list of available machine learning scenarios provided by Model Builder. diff --git a/docs/machine-learning/tutorials/media/sentiment-analysis-model-builder/web-app.png b/docs/machine-learning/tutorials/media/sentiment-analysis-model-builder/web-app.png index a3d6a75866073..9a558bb67834a 100644 Binary files a/docs/machine-learning/tutorials/media/sentiment-analysis-model-builder/web-app.png and b/docs/machine-learning/tutorials/media/sentiment-analysis-model-builder/web-app.png differ diff --git a/docs/machine-learning/tutorials/movie-recommendation-model-builder.md b/docs/machine-learning/tutorials/movie-recommendation-model-builder.md index cf596b3ef6af1..f7d52ab4461eb 100644 --- a/docs/machine-learning/tutorials/movie-recommendation-model-builder.md +++ b/docs/machine-learning/tutorials/movie-recommendation-model-builder.md @@ -25,9 +25,6 @@ In this tutorial, you: > - Evaluate the model > - Consume the model -> [!NOTE] -> Model Builder is currently in Preview. - ## Prerequisites For a list of pre-requisites and installation instructions, visit the [Model Builder installation guide](../how-to-guides/install-model-builder.md). @@ -62,11 +59,11 @@ When first adding Model Builder to the solution it will prompt you to create an ## Choose a scenario -![Model Builder Scenarios](../media/model-builder-scenarios.png) +:::image type="content" source="../media/model-builder-scenarios-2-0.png" alt-text="Model Builder Scenario Screen" lightbox="../media/model-builder-scenarios-2-0.png"::: To train your model, you need to select from the list of available machine learning scenarios provided by Model Builder. -For this sample, the task is image classification. In the scenario step of the Model Builder tool, select the **Recommendation** scenario. +For this sample, the task is recommendation. In the scenario step of the Model Builder tool, select the **Recommendation** scenario. ## Select an environment diff --git a/docs/machine-learning/tutorials/object-detection-model-builder.md b/docs/machine-learning/tutorials/object-detection-model-builder.md index d2fa28e4f4827..9e604039acc39 100644 --- a/docs/machine-learning/tutorials/object-detection-model-builder.md +++ b/docs/machine-learning/tutorials/object-detection-model-builder.md @@ -26,9 +26,6 @@ In this tutorial, you learn how to: > - Evaluate the model > - Use the model for predictions -> [!NOTE] -> Model Builder is currently in Preview. - ## Prerequisites For a list of prerequisites and installation instructions, visit the [Model Builder installation guide](../how-to-guides/install-model-builder.md). diff --git a/docs/machine-learning/tutorials/predict-prices-with-model-builder.md b/docs/machine-learning/tutorials/predict-prices-with-model-builder.md index 4ed862e325d85..49b3172486b59 100644 --- a/docs/machine-learning/tutorials/predict-prices-with-model-builder.md +++ b/docs/machine-learning/tutorials/predict-prices-with-model-builder.md @@ -26,9 +26,6 @@ In this tutorial, you learn how to: > - Evaluate the model > - Use the model for predictions -> [!NOTE] -> Model Builder is currently in Preview. - ## Pre-requisites For a list of pre-requisites and installation instructions, visit the [Model Builder installation guide](../how-to-guides/install-model-builder.md). @@ -76,7 +73,7 @@ When first adding Model Builder to the solution it will prompt you to create an ## Choose a scenario -![Model Builder wizard in Visual Studio](../media/model-builder-scenarios.png) +:::image type="content" source="../media/model-builder-scenarios-2-0.png" alt-text="Model Builder Scenario Screen" lightbox="../media/model-builder-scenarios-2-0.png"::: To train your model, you need to select from the list of available machine learning scenarios provided by Model Builder. In this case, the scenario is `Value prediction`. diff --git a/docs/machine-learning/tutorials/sentiment-analysis-model-builder.md b/docs/machine-learning/tutorials/sentiment-analysis-model-builder.md index 908e6c4a588bd..121a2f004dff1 100644 --- a/docs/machine-learning/tutorials/sentiment-analysis-model-builder.md +++ b/docs/machine-learning/tutorials/sentiment-analysis-model-builder.md @@ -1,7 +1,7 @@ --- -title: 'Tutorial: Analyze sentiment - binary classification' -description: This tutorial shows you how to create a Razor Pages application that classifies sentiment from website comments and takes the appropriate action. The binary sentiment classifier uses Model Builder in Visual Studio. -ms.date: 09/20/2021 +title: 'Tutorial: Analyze sentiment - Text Classification' +description: This tutorial shows you how to create a Razor Pages application that classifies sentiment from website comments and takes the appropriate action. The classification uses the Text Classification scenario in the Model Builder Visual Studio extension. +ms.date: 11/10/2022 author: luisquintanilla ms.author: luquinta ms.topic: tutorial @@ -27,9 +27,6 @@ In this tutorial, you learn how to: > - Evaluate the model > - Use the model for predictions -> [!NOTE] -> Model Builder is currently in Preview. - You can find the source code for this tutorial at the [dotnet/machinelearning-samples](https://github.com/dotnet/machinelearning-samples) repository. ## Pre-requisites @@ -38,15 +35,12 @@ For a list of pre-requisites and installation instructions, visit the [Model Bui ## Create a Razor Pages application -1. Create an **ASP.NET Core Razor Pages Application**. +Create an **ASP.NET Core Razor Pages Application**. - 1. Open Visual Studio and select **File > New > Project** from the menu bar. - 1. In the New Project dialog, select the **Visual C#** node followed by the **Web** node. - 1. Then select the **ASP.NET Core Web Application** project template. - 1. In the **Name** text box, type "SentimentRazor". - 1. Make sure **Place solution and project in the same directory** is **unchecked** (VS 2019), or **Create directory for solution** is **checked** (VS 2017). - 1. Select the **OK** button. - 1. Choose **Web Application** in the window that displays the different types of ASP.NET Core Projects, and then select the **OK** button. +1. In Visual Studio open the **Create a new project** dialog. +1. In the "Create a new project" dialog, select the **ASP.NET Core Web App** project template. +1. In the **Name** text box, type "SentimentRazor" and select the **Next** button. +1. In the Additional information dialog, leave all the defaults as is and select the **Create** button. ## Prepare and understand the data @@ -60,63 +54,70 @@ Each row in the *wikipedia-detox-250-line-data.tsv* dataset represents a differe 1 | == OK! == IM GOING TO VANDALIZE WILD ONES WIKI THEN!!! 0 | I hope this helps. +## Create a Model Builder config file + +When first adding a machine learning model to the solution it will prompt you to create an `mbconfig` file. The `mbconfig` file keeps track of everything you do in Model Builder to allow you to reopen the session. + +1. In **Solution Explorer**, right-click the *SentimentRazor* project, and select **Add** > **Machine Learning Model...**. +1. In the dialog, name the Model Builder project **SentimentAnalysis.mbconfig**, and select **Add**. + ## Choose a scenario -![Model Builder wizard in Visual Studio](../media/model-builder-scenarios.png) +:::image type="content" source="../media/model-builder-scenarios-2-0.png" alt-text="Model Builder Scenario Screen" lightbox="../media/model-builder-scenarios-2-0.png"::: To train your model, you need to select from the list of available machine learning scenarios provided by Model Builder. -1. In **Solution Explorer**, right-click the *SentimentRazor* project, and select **Add** > **Machine Learning Model...**. -1. For this sample, the scenario is sentiment analysis. In the *scenario* step of the Model Builder tool, select the **Sentiment Analysis** scenario. +For this sample, the task is text classification. In the *Scenario* step of the Model Builder extension, select the **Text classification** scenario. + +## Select an environment + +Model Builder can train on different environments depending on the selected scenario. + +Select **Local (GPU)** as your environment and click the **Next step** button. + +> [!NOTE] +> This scenario uses deep learning techniques which work best in GPU environments. If you don't have a GPU, choose the Local (CPU) environment but note that the expected time to train will be significantly longer. For more information on using GPUs with Model Builder, see the [GPU support in Model Builder guide](../how-to-guides/install-gpu-model-builder.md). ## Load the data Model Builder accepts data from two sources, a SQL Server database or a local file in `csv` or `tsv` format. -1. In the data step of the Model Builder tool, select **File** from the data source dropdown. +1. In the data step of the Model Builder tool, select **File** from the data source options. 1. Select the button next to the **Select a file** text box and use File Explorer to browse and select the *wikipedia-detox-250-line-data.tsv* file. -1. Choose **Sentiment** in the **Column to predict (Label)** dropdown. -1. Leave the default values for the **Input Columns (Features)** dropdown. -1. Select the **Train** link to move to the next step in the Model Builder tool. +1. Choose **Sentiment** from the **Column to predict (Label)** dropdown. +1. Choose **SentimentText** from the **Text Column** dropdown. +1. Select the **Next step** button to move to the next step in Model Builder. ## Train the model -The machine learning task used to train the sentiment analysis model in this tutorial is binary classification. During the model training process, Model Builder trains separate models using different binary classification algorithms and settings to find the best performing model for your dataset. - -The time required for the model to train is proportionate to the amount of data. Model Builder automatically selects a default value for **Time to train (seconds)** based on the size of your data source. +The machine learning task used to train the sentiment analysis model in this tutorial is text classification. During the model training process, Model Builder trains a text classification model for your dataset using the [NAS-BERT](https://arxiv.org/abs/2105.14444) neural network architecture. -1. Although Model Builder sets the value of **Time to train (seconds)** to 10 seconds, increase it to 30 seconds. Training for a longer period of time allows Model Builder to explore a larger number of algorithms and combination of parameters in search of the best model. 1. Select **Start Training**. +1. Once training is complete, the results from the training process are displayed in the *Training results* section of the *Train* screen. + In addition to providing training results, three code-behind files are created under the *SentimentAnalysis.mbconfig* file. - Throughout the training process, progress data is displayed in the `Progress` section of the train step. + - *SentimentAnalysis.consumption.cs* - This file contains the `ModelInput` and `ModelOutput` schemas as well as the `Predict` function generated for consuming the model. + - *SentimentAnalysis.training.cs* - This file contains the training pipeline (data transforms, trainer, trainer hyperparameters) chosen by Model Builder to train the model. You can use this pipeline for re-training your model. + - **SentimentAnalysis.zip* - This is a serialized zip file which represents your trained ML.NET model. - - Status displays the completion status of the training process. - - Best accuracy displays the accuracy of the best performing model found by Model Builder so far. Higher accuracy means the model predicted more correctly on test data. - - Best algorithm displays the name of the best performing algorithm performed found by Model Builder so far. - - Last algorithm displays the name of the algorithm most recently used by Model Builder to train the model. - -1. Once training is complete, select the **evaluate** link to move to the next step. +1. Select the **Next step** button to move to the next step. ## Evaluate the model -The result of the training step will be one model that has the best performance. In the evaluate step of the Model Builder tool, the output section will contain the algorithm used by the best-performing model in the **Best Model** entry along with metrics in **Best Model Accuracy**. Additionally, a summary table containing the top five models and their metrics is shown. - -If you're not satisfied with your accuracy metrics, some easy ways to try to improve model accuracy are to increase the amount of time to train the model or use more data. Otherwise, select the **code** link to move to the final step in the Model Builder tool. - -## Add the code to make predictions +The result of the training step will be one model that has the best performance. In the evaluate step of the Model Builder tool, the output section will contain the trainer used by the best-performing model in the as well as evaluation metrics. -Two projects will be created as a result of the training process. +If you're not satisfied with your evaluation metrics, some easy ways to try to improve model performance are to use more data. -### Reference the trained model +Otherwise, select the **Next step** button to move to the *Consume* step in Model Builder. -1. In the *code* step of the Model Builder tool, select **Add Projects** to add the autogenerated projects to the solution. +## Add consumption project templates (Optional) - The following projects should appear in the **Solution Explorer**: +In the *Consume* step, Model Builder provides project templates that you can use to consume the model. This step is optional and you can choose the method that best fits your needs for using the model. - - *SentimentRazorML.ConsoleApp*: A .NET Core Console application that contains the model training and prediction code. - - *SentimentRazorML.Model*: A .NET Standard class library containing the data models that define the schema of input and output model data as well as the saved version of the best performing model during training. +- Console application +- Web API - For this tutorial, only the *SentimentRazorML.Model* project is used because predictions will be made in the *SentimentRazor* web application rather than in the console. Although the *SentimentRazorML.ConsoleApp* won't be used for scoring, it can be used to retrain the model using new data at a later time. Retraining is outside the scope of this tutorial though. +## Add the code to make predictions ### Configure the PredictionEngine pool @@ -131,71 +132,46 @@ To make a single prediction, you have to create a for your application in the *Program.cs* file: ```csharp - private readonly string _modelPath; - ``` - -1. The model file is stored in the build directory alongside the assembly files of your application. To make it easier to access, create a helper method called `GetAbsolutePath` after the `Configure` method - - ```csharp - public static string GetAbsolutePath(string relativePath) - { - FileInfo _dataRoot = new FileInfo(typeof(Program).Assembly.Location); - string assemblyFolderPath = _dataRoot.Directory.FullName; - - string fullPath = Path.Combine(assemblyFolderPath, relativePath); - return fullPath; - } - ``` - -1. Use the `GetAbsolutePath` method in the `Startup` class constructor to set the `_modelPath`. - - ```csharp - _modelPath = GetAbsolutePath("MLModel.zip"); - ``` - -1. Configure the `PredictionEnginePool` for your application in the `ConfigureServices` method: - - ```csharp - services.AddPredictionEnginePool() - .FromFile(_modelPath); + builder.Services.AddPredictionEnginePool() + .FromFile("SentimentAnalysis.zip"); ``` ### Create sentiment analysis handler -Predictions will be made inside the main page of the application. Therefore, a method that takes the user input and uses the `PredictionEnginePool` to return a prediction needs to be added. +Predictions will be made inside the main page of the application. Therefore, a method that takes the user input and uses the to return a prediction needs to be added. 1. Open the *Index.cshtml.cs* file located in the *Pages* directory and add the following using statements: ```csharp using Microsoft.Extensions.ML; - using SentimentRazorML.Model; + using static SentimentRazor.SentimentAnalysis; ``` - In order to use the `PredictionEnginePool` configured in the `Startup` class, you have to inject it into the constructor of the model where you want to use it. + In order to use the configured in the *Program.cs* file, you have to inject it into the constructor of the model where you want to use it. -1. Add a variable to reference the `PredictionEnginePool` inside the `IndexModel` class. +1. Add a variable to reference the inside the `IndexModel` class inside the *Pages/Index.cshtml.cs* file. ```csharp private readonly PredictionEnginePool _predictionEnginePool; ``` -1. Create a constructor in the `IndexModel` class and inject the `PredictionEnginePool` service into it. +1. Modify the constructor in the `IndexModel` class and inject the service into it. ```csharp - public IndexModel(PredictionEnginePool predictionEnginePool) + public IndexModel(ILogger logger, PredictionEnginePool predictionEnginePool) { + _logger = logger; _predictionEnginePool = predictionEnginePool; } ``` @@ -223,7 +199,7 @@ Predictions will be made inside the main page of the application. Therefore, a m var input = new ModelInput { SentimentText = text }; ``` - 1. Use the `PredictionEnginePool` to predict sentiment. + 1. Use the to predict sentiment. ```csharp var prediction = _predictionEnginePool.Predict(input); @@ -232,7 +208,7 @@ Predictions will be made inside the main page of the application. Therefore, a m 1. Convert the predicted `bool` value into toxic or not toxic with the following code. ```csharp - var sentiment = Convert.ToBoolean(prediction.Prediction) ? "Toxic" : "Not Toxic"; + var sentiment = Convert.ToBoolean(prediction.PredictedLabel) ? "Toxic" : "Not Toxic"; ``` 1. Finally, return the sentiment back to the web page. @@ -247,39 +223,143 @@ The results returned by the `OnGetAnalyzeSentiment` will be dynamically displaye 1. Open the *Index.cshtml* file in the *Pages* directory and replace its contents with the following code: - [!code-cshtml [IndexPage](~/machinelearning-samples/samples/modelbuilder/BinaryClassification_Sentiment_Razor/SentimentRazor/Pages/Index.cshtml)] + ```cshtml + @page + @model IndexModel + @{ + ViewData["Title"] = "Home page"; + } + +
+

Live Sentiment

+ +

+ +
+

Your sentiment is...

+

😡 😐 😍

+ +
+
+
▲
+ +
+
+
+
+ ``` 1. Next, add css styling code to the end of the *site.css* page in the *wwwroot\css* directory: - [!code-css [CssStyling](~/machinelearning-samples/samples/modelbuilder/BinaryClassification_Sentiment_Razor/SentimentRazor/wwwroot/css/site.css#L61-L105)] + ```css + /* Style for sentiment display */ + + .sentiment { + background-color: #eee; + position: relative; + display: inline-block; + padding: 1rem; + padding-bottom: 0; + border-radius: 1rem; + } + + .sentiment h4 { + font-size: 16px; + text-align: center; + margin: 0; + padding: 0; + } + + .sentiment p { + font-size: 50px; + } + + .sentiment .marker { + position: relative; + left: 22px; + width: calc(100% - 68px); + } + + .sentiment .marker > div { + transition: 0.3s ease-in-out; + position: absolute; + margin-left: -30px; + text-align: center; + } + + .sentiment .marker > div > div { + font-size: 50px; + line-height: 20px; + color: green; + } + + .sentiment .marker > div label { + font-size: 30px; + color: gray; + } + ``` 1. After that, add code to send inputs from the web page to the `OnGetAnalyzeSentiment` handler. 1. In the *site.js* file located in the *wwwroot\js* directory, create a function called `getSentiment` to make a GET HTTP request with the user input to the `OnGetAnalyzeSentiment` handler. - [!code-javascript [GetSentimentMethod](~/machinelearning-samples/samples/modelbuilder/BinaryClassification_Sentiment_Razor/SentimentRazor/wwwroot/js/site.js#L5-L10)] + ```javascript + function getSentiment(userInput) { + return fetch(`Index?handler=AnalyzeSentiment&text=${userInput}`) + .then((response) => { + return response.text(); + }) + } + ``` 1. Below that, add another function called `updateMarker` to dynamically update the position of the marker on the web page as sentiment is predicted. - [!code-javascript [UpdateMarkerMethod](~/machinelearning-samples/samples/modelbuilder/BinaryClassification_Sentiment_Razor/SentimentRazor/wwwroot/js/site.js#L12-L15)] + ```javascript + function updateMarker(markerPosition, sentiment) { + $("#markerPosition").attr("style", `left:${markerPosition}%`); + $("#markerValue").text(sentiment); + } + ``` 1. Create an event handler function called `updateSentiment` to get the input from the user, send it to the `OnGetAnalyzeSentiment` function using the `getSentiment` function and update the marker with the `updateMarker` function. - [!code-javascript [UpdateSentimentMethod](~/machinelearning-samples/samples/modelbuilder/BinaryClassification_Sentiment_Razor/SentimentRazor/wwwroot/js/site.js#L17-L34)] + ```javascript + function updateSentiment() { + + var userInput = $("#Message").val(); + + getSentiment(userInput) + .then((sentiment) => { + switch (sentiment) { + case "Not Toxic": + updateMarker(100.0, sentiment); + break; + case "Toxic": + updateMarker(0.0, sentiment); + break; + default: + updateMarker(45.0, "Neutral"); + } + }); + } + ``` 1. Finally, register the event handler and bind it to the `textarea` element with the `id=Message` attribute. - [!code-javascript [UpdateSentimentEvtHandler](~/machinelearning-samples/samples/modelbuilder/BinaryClassification_Sentiment_Razor/SentimentRazor/wwwroot/js/site.js#L36)] + ```javascript + $("#Message").on('change input paste', updateSentiment) + ``` ## Run the application Now that your application is set up, run the application, which should launch in your browser. -When the application launches, enter *Model Builder is cool!* into the text area. The predicted sentiment displayed should be *Not Toxic*. +When the application launches, enter *This model doesn't have enough data!* into the text area. The predicted sentiment displayed should be *Toxic*. ![Running window with the predicted sentiment window](./media/sentiment-analysis-model-builder/web-app.png) -If you need to reference the Model Builder generated projects at a later time inside of another solution, you can find them inside the `C:\Users\%USERNAME%\AppData\Local\Temp\MLVSTools` directory. +> [!NOTE] +> creates multiple instances of . Because of the size of the model, the first time you use it to make a prediction, it can take a couple of seconds. Subsequent predictions should be instantaneous. ## Next steps @@ -299,5 +379,4 @@ In this tutorial, you learned how to: To learn more about topics mentioned in this tutorial, visit the following resources: - [Model Builder Scenarios](../automate-training-with-model-builder.md#scenario) -- [Binary Classification](../resources/glossary.md#binary-classification) -- [Binary Classification Model Metrics](../resources/metrics.md#evaluation-metrics-for-binary-classification) +- [Text Classification Model Metrics](../resources/metrics.md#evaluation-metrics-for-multi-class-classification-and-text-classification)