Skip to content

Commit

Permalink
docs: update anomaly detector docs (#2103)
Browse files Browse the repository at this point in the history 
* update anomaly detector docs

* Apply suggestions from code review

Co-authored-by: Mark Hamilton <mhamilton723@gmail.com>

* Update docs/Explore Algorithms/AI Services/Overview.ipynb


* modify AI services Overview

---------
  • Loading branch information
@JessicaXYWang
JessicaXYWang committed Oct 30, 2023
1 parent 377df2f commit 1af71ed
Show file tree
Hide file tree
Showing 2 changed files with 65 additions and 28 deletions.
14 changes: 11 additions & 3 deletions docs/Explore Algorithms/AI Services/Multivariate Anomaly Detection.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,18 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Recipe: Cognitive Services - Multivariate Anomaly Detection \n",
"This recipe shows how you can use SynapseML and Azure Cognitive Services on Apache Spark for multivariate anomaly detection. Multivariate anomaly detection allows for the detection of anomalies among many variables or timeseries, taking into account all the inter-correlations and dependencies between the different variables. In this scenario, we use SynapseML to train a model for multivariate anomaly detection using the Azure Cognitive Services, and we then use to the model to infer multivariate anomalies within a dataset containing synthetic measurements from three IoT sensors. \n",
"# Recipe: AI Services - Multivariate Anomaly Detection \n",
"This recipe shows how you can use SynapseML and Azure AI services on Apache Spark to detect anomalies in multivariate time-series data. Multivariate anomaly detection takes correlations and dependencies between the different variables into account when discovering anomalies. In this scenario, we use SynapseML to train a model for multivariate anomaly detection using the Azure AI Services, and we then use to the model to detect anomalies in a dataset of synthetic measurements from three IoT sensors. \n",
"\n",
"To learn more about the Anomaly Detector Cognitive Service, refer to [this documentation page](https://docs.microsoft.com/azure/cognitive-services/anomaly-detector/). "
"To learn more about the Azure AI Anomaly Detector, refer to [this documentation page](https://docs.microsoft.com/azure/ai-services/anomaly-detector/). "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Important\n",
"Starting on the 20th of September, 2023 you won’t be able to create new Anomaly Detector resources. The Anomaly Detector service is being retired on the 1st of October, 2026."
]
},
{
Expand Down
79 changes: 54 additions & 25 deletions docs/Explore Algorithms/AI Services/Overview.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Cognitive Services"
"# Azure AI Services"
]
},
{
Expand All @@ -19,12 +19,20 @@
"<image width=\"200\" alt-text=\"icon\" src=\"https://mmlspark.blob.core.windows.net/graphics/Readme/cog_services_on_spark_2.svg\" />"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Important\n",
"Starting on the 20th of September, 2023 you won’t be able to create new Anomaly Detector resources. The Anomaly Detector service is being retired on the 1st of October, 2026."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"[Azure Cognitive Services](https://azure.microsoft.com/services/cognitive-services/) are a suite of APIs, SDKs, and services available to help developers build intelligent applications without having direct AI or data science skills or knowledge by enabling developers to easily add cognitive features into their applications. The goal of Azure Cognitive Services is to help developers create applications that can see, hear, speak, understand, and even begin to reason. The catalog of services within Azure Cognitive Services can be categorized into five main pillars - Vision, Speech, Language, Web Search, and Decision.\n",
"[Azure AI services](https://azure.microsoft.com/services/ai-services/) is a suite of APIs, SDKs, and services that developers can use to add intelligent features to their applications. AI services empower developers even when they don't have direct AI or data science skills or knowledge. Azure AI services help developers create applications that can see, hear, speak, understand, and even begin to reason. The catalog of services within Azure AI services can be categorized into five main pillars: Vision, Speech, Language, Web search, and Decision.\n",
"\n",
"## Usage\n",
"\n",
Expand Down Expand Up @@ -103,7 +111,7 @@
"source": [
"## Prerequisites\n",
"\n",
"1. Follow the steps in [Getting started](https://docs.microsoft.com/azure/cognitive-services/big-data/getting-started) to set up your Azure Databricks and Cognitive Services environment. This tutorial shows you how to install SynapseML and how to create your Spark cluster in Databricks.\n",
"1. Follow the steps in [Getting started](https://docs.microsoft.com/azure/cognitive-services/big-data/getting-started) to set up your Azure Databricks and Azure AI services environment. This tutorial shows you how to install SynapseML and how to create your Spark cluster in Databricks.\n",
"1. After you create a new notebook in Azure Databricks, copy the **Shared code** below and paste into a new cell in your notebook.\n",
"1. Choose a service sample, below, and copy paste it into a second new cell in your notebook.\n",
"1. Replace any of the service subscription key placeholders with your own key.\n",
Expand All @@ -115,9 +123,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Shared code\n",
"## Prepare your system\n",
"\n",
"To get started, we'll need to add this code to the project:"
"To begin, import required libraries and initialize your Spark session."
]
},
{
Expand All @@ -134,6 +142,13 @@
"from pyspark.sql.functions import col"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Import Azure AI services libraries and replace the keys in the following code snippet with your Azure AI services key."
]
},
{
"cell_type": "code",
"execution_count": null,
Expand All @@ -143,10 +158,10 @@
"from synapse.ml.cognitive import *\n",
"from synapse.ml.core.platform import *\n",
"\n",
"# A general Cognitive Services key for Text Analytics, Computer Vision and Form Recognizer (or use separate keys that belong to each service)\n",
"# A general AI services key for Text Analytics, Computer Vision and Form Recognizer (or use separate keys that belong to each service)\n",
"service_key = find_secret(\n",
" \"cognitive-api-key\"\n",
") # Replace the call to find_secret with your key as a python string. e.g. service_key=\"27snaiw...\"\n",
") # Replace it with your ai service key, check prerequisites for more details\n",
"service_loc = \"eastus\"\n",
"\n",
"# A Bing Search v7 subscription key\n",
Expand All @@ -157,7 +172,7 @@
"# An Anomaly Detector subscription key\n",
"anomaly_key = find_secret(\n",
" \"anomaly-api-key\"\n",
") # Replace the call to find_secret with your key as a python string.\n",
") # Replace the call to find_secret with your key as a python string. If you don't have an anomaly detection resource created before Sep 20th 2023, you won't be able to create one.\n",
"anomaly_loc = \"westus2\"\n",
"\n",
"# A Translator subscription key\n",
Expand All @@ -177,9 +192,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Text Analytics sample\n",
"## Perform sentiment analysis on text\n",
"\n",
"The [Text Analytics](https://azure.microsoft.com/services/cognitive-services/text-analytics/) service provides several algorithms for extracting intelligent insights from text. For example, we can find the sentiment of given input text. The service will return a score between 0.0 and 1.0 where low scores indicate negative sentiment and high score indicates positive sentiment. This sample uses three simple sentences and returns the sentiment for each."
"The [Text Analytics](https://azure.microsoft.com/services/ai-services/text-analytics/) service provides several algorithms for extracting intelligent insights from text. For example, we can find the sentiment of given input text. The service will return a score between 0.0 and 1.0 where low scores indicate negative sentiment and high score indicates positive sentiment. This sample uses three simple sentences and returns the sentiment for each."
]
},
{
Expand All @@ -193,7 +208,7 @@
" [\n",
" (\"I am so happy today, its sunny!\", \"en-US\"),\n",
" (\"I am frustrated by this rush hour traffic\", \"en-US\"),\n",
" (\"The cognitive services on spark aint bad\", \"en-US\"),\n",
" (\"The AI services on spark aint bad\", \"en-US\"),\n",
" ],\n",
" [\"text\", \"language\"],\n",
")\n",
Expand Down Expand Up @@ -222,9 +237,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Text Analytics for Health Sample\n",
"## Perform text analytics for health data\n",
"\n",
"The [Text Analytics for Health Service](https://docs.microsoft.com/azure/cognitive-services/language-service/text-analytics-for-health/overview?tabs=ner) extracts and labels relevant medical information from unstructured texts such as doctor's notes, discharge summaries, clinical documents, and electronic health records."
"The [Text Analytics for Health Service](https://docs.microsoft.com/azure/ai-services/language-service/text-analytics-for-health/overview?tabs=ner) extracts and labels relevant medical information from unstructured text such as doctor's notes, discharge summaries, clinical documents, and electronic health records.\n",
"\n",
"The following code sample analyzes and transforms text from doctors notes into structured data."
]
},
{
Expand Down Expand Up @@ -258,8 +275,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Translator sample\n",
"[Translator](https://azure.microsoft.com/services/cognitive-services/translator/) is a cloud-based machine translation service and is part of the Azure Cognitive Services family of cognitive APIs used to build intelligent apps. Translator is easy to integrate in your applications, websites, tools, and solutions. It allows you to add multi-language user experiences in 90 languages and dialects and can be used for text translation with any operating system. In this sample, we do a simple text translation by providing the sentences you want to translate and target languages you want to translate to."
"## Translate text into a different language\n",
"[Translator](https://azure.microsoft.com/services/ai-services/translator/) is a cloud-based machine translation service and is part of the Azure AI services family of cognitive APIs used to build intelligent apps. Translator is easy to integrate in your applications, websites, tools, and solutions. It allows you to add multi-language user experiences in 90 languages and dialects and can be used to translate text without hosting your own algorithm.\n",
"\n",
"The following code sample does a simple text translation by providing the sentences you want to translate and target languages you want to translate them to."
]
},
{
Expand Down Expand Up @@ -302,8 +321,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Form Recognizer sample\n",
"[Form Recognizer](https://azure.microsoft.com/services/form-recognizer/) is a part of Azure Applied AI Services that lets you build automated data processing software using machine learning technology. Identify and extract text, key/value pairs, selection marks, tables, and structure from your documents. The service outputs structured data that includes the relationships in the original file, bounding boxes, confidence and more. In this sample, we analyze a business card image and extract its information into structured data."
"## Extract information from a document into structured data\n",
"[Form Recognizer](https://azure.microsoft.com/services/form-recognizer/) is a part of Azure Applied AI Services that lets you build automated data processing software using machine learning technology. With Form Recognizer, you can identify and extract text, key/value pairs, selection marks, tables, and structure from your documents. The service outputs structured data that includes the relationships in the original file, bounding boxes, confidence and more.\n",
"\n",
"The following code sample analyzes a business card image and extracts its information into structured data."
]
},
{
Expand Down Expand Up @@ -352,7 +373,9 @@
"source": [
"## Computer Vision sample\n",
"\n",
"[Computer Vision](https://azure.microsoft.com/services/cognitive-services/computer-vision/) analyzes images to identify structure such as faces, objects, and natural-language descriptions. In this sample, we tag a list of images. Tags are one-word descriptions of things in the image like recognizable objects, people, scenery, and actions."
"[Azure AI Vision](https://azure.microsoft.com/products/ai-services/ai-vision/) analyzes images to identify structure such as faces, objects, and natural-language descriptions.\n",
"\n",
"The following code sample analyzes images and labels them with tags. Tags are one-word descriptions of things in the image, such as recognizable objects, people, scenery, and actions."
]
},
{
Expand Down Expand Up @@ -398,7 +421,9 @@
"source": [
"## Bing Image Search sample\n",
"\n",
"[Bing Image Search](https://azure.microsoft.com/services/cognitive-services/bing-image-search-api/) searches the web to retrieve images related to a user's natural language query. In this sample, we use a text query that looks for images with quotes. It returns a list of image URLs that contain photos related to our query."
"[Bing Image Search](https://www.microsoft.com/bing/apis/bing-image-search-api) searches the web to retrieve images related to a user's natural language query.\n",
"\n",
"The following code sample uses a text query that looks for images with quotes. The output of the code is a list of image URLs that contain photos related to the query."
]
},
{
Expand Down Expand Up @@ -442,8 +467,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Speech-to-Text sample\n",
"The [Speech-to-text](https://azure.microsoft.com/services/cognitive-services/speech-services/) service converts streams or files of spoken audio to text. In this sample, we transcribe one audio file."
"## Transform speech to text\n",
"The [Speech-to-text](https://azure.microsoft.com/products/ai-services/ai-speech/) service converts streams or files of spoken audio to text. The following code sample transcribes one audio file to text."
]
},
{
Expand Down Expand Up @@ -478,7 +503,9 @@
"metadata": {},
"source": [
"## Text-to-Speech sample\n",
"[Text to speech](https://azure.microsoft.com/services/cognitive-services/text-to-speech/#overview) is a service that allows one to build apps and services that speak naturally, choosing from more than 270 neural voices across 119 languages and variants."
"[Text to speech](https://azure.microsoft.com/products/ai-services/text-to-speech/) is a service that allows you to build applications that speak naturally. You can choose from more than 270 neural voices across 119 languages.\n",
"\n",
"The following code sample transforms text into an audio file that contains the content of the text."
]
},
{
Expand Down Expand Up @@ -524,9 +551,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Anomaly Detector sample\n",
"## Detect anomalies in time series data\n",
"\n",
"If you don't have an anomaly detection resource created before Sep 20th 2023, you won't be able to create one. You may want to skip this part.\n",
"\n",
"[Anomaly Detector](https://azure.microsoft.com/services/cognitive-services/anomaly-detector/) is great for detecting irregularities in your time series data. In this sample, we use the service to find anomalies in the entire time series."
"[Anomaly Detector](https://azure.microsoft.com/services/cognitive-services/anomaly-detector/) is great for detecting irregularities in your time series data. The following code sample uses the Anomaly Detector service to find anomalies in a time series."
]
},
{
Expand Down Expand Up @@ -579,7 +608,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Arbitrary web APIs\n",
"## Get information from arbitrary web APIs\n",
"\n",
"With HTTP on Spark, any web service can be used in your big data pipeline. In this example, we use the [World Bank API](http://api.worldbank.org/v2/country/) to get information about various countries around the world."
]
Expand Down

0 comments on commit 1af71ed

Please sign in to comment.