| 
| 
| 
|
|:-------------------------:|:-------------------------:|:-------------------------:|:-------------------------:|
-| **Vowpal Wabbit on Spark** | **The Cognitive Services on Spark**| **LightGBM on Spark** | **Spark Serving** |
+| [**Vowpal Wabbit on Spark**](https://microsoft.github.io/SynapseML/docs/features/vw/about/) | [**The Cognitive Services for Big Data**](https://microsoft.github.io/SynapseML/docs/features/cognitive_services/CognitiveServices%20-%20Overview/)| [**LightGBM on Spark**](https://microsoft.github.io/SynapseML/docs/features/lightgbm/about/) | [**Spark Serving**](https://microsoft.github.io/SynapseML/docs/features/spark_serving/about/) |
| Fast, Sparse, and Effective Text Analytics | Leverage the Microsoft Cognitive Services at Unprecedented Scales in your existing SparkML pipelines | Train Gradient Boosted Machines with LightGBM | Serve any Spark Computation as a Web Service with Sub-Millisecond Latency |
-|
|
| 
| 
|
+| |
| 
| |
|:-------------------------:|:-------------------------:|:-------------------------:|:-------------------------:|
-| **HTTP on Spark** | **CNTK on Spark** | **Model Interpretation on Spark**| **Spark Binding Autogeneration** |
-| An Integration Between Spark and the HTTP Protocol, enabling Distributed Microservice Orchestration|Distributed Deep Learning with the Microsoft Cognitive Toolkit | Distributed, Model Agnostic, Interpretations for Classifiers | Automatically Generate Spark bindings for PySpark and SparklyR|
+| [**HTTP on Spark**](https://microsoft.github.io/SynapseML/docs/features/cognitive_services/CognitiveServices%20-%20Overview/#arbitrary-web-apis) | [**ONNX on Spark**](https://microsoft.github.io/SynapseML/docs/features/onnx/about/) | [**Responsible AI**](https://microsoft.github.io/SynapseML/docs/features/responsible_ai/Model%20Interpretation%20on%20Spark/)| [**Spark Binding Autogeneration**](https://microsoft.github.io/SynapseML/docs/reference/developer-readme/#packagepython) |
+| An Integration Between Spark and the HTTP Protocol, enabling Distributed Microservice Orchestration| Distributed and Hardware Accelerated Model Inference on Spark | Understand Opaque-box Models and Measure Dataset Biases | Automatically Generate Spark bindings for PySpark and SparklyR|
-| 
|
| 
|
-|:--:|:--:|:--:|
-| **Isolation Forest on Spark** | [**CyberML**](https://github.com/Microsoft/SynapseML/blob/master/notebooks/CyberML%20-%20Anomalous%20Access%20Detection.ipynb) | **Conditional KNN** |
+| | | |
+|:---:|:---:|:---:|
+| [**Isolation Forest on Spark**](https://microsoft.github.io/SynapseML/docs/documentation/estimators/estimators_core/#isolationforest) | [**CyberML**](https://github.com/Microsoft/SynapseML/blob/master/notebooks/CyberML%20-%20Anomalous%20Access%20Detection.ipynb) | [**Conditional KNN**](https://microsoft.github.io/SynapseML/docs/features/other/ConditionalKNN%20-%20Exploring%20Art%20Across%20Cultures/) |
| Distributed Nonlinear Outlier Detection | Machine Learning Tools for Cyber Security | Scalable KNN Models with Conditional Queries |
+## Documentation and Examples
-
-## Examples
-
-- Create a deep image classifier with transfer learning ([example 9])
-- Fit a LightGBM classification or regression model on a biochemical dataset
- ([example 3]), to learn more check out the [LightGBM documentation
- page](docs/lightgbm.md).
-- Deploy a deep network as a distributed web service with [SynapseML
- Serving](docs/mmlspark-serving.md)
-- Use web services in Spark with [HTTP on Apache Spark](docs/http.md)
-- Use Bi-directional LSTMs from Keras for medical entity extraction
- ([example 8])
-- Create a text analytics system on Amazon book reviews ([example 4])
-- Perform distributed hyperparameter tuning to identify Breast Cancer
- ([example 5])
-- Easily ingest images from HDFS into Spark `DataFrame` ([example 6])
-- Use OpenCV on Spark to manipulate images ([example 7])
-- Train classification and regression models easily via implicit featurization
- of data ([example 1])
-- Train and evaluate a flight delay prediction system ([example 2])
-- Finding anomalous data access patterns using the Access Anomalies package of CyberML ([example 11])
-- Model interpretation ([example 12], [example 13], [example 14])
-
-See our [notebooks](notebooks/) for all examples.
-
-[example 1]: notebooks/Classification%20-%20Adult%20Census.ipynb "Adult Census Income Training"
-
-[example 2]: notebooks/Regression%20-%20Flight%20Delays.ipynb "Regression Example with Flight Delay Dataset"
-
-[example 3]: notebooks/LightGBM%20-%20Overview.ipynb "Quantile Regression with LightGBM"
-
-[example 4]: notebooks/TextAnalytics%20-%20Amazon%20Book%20Reviews.ipynb "Amazon Book Reviews - TextFeaturizer"
-
-[example 5]: notebooks/HyperParameterTuning%20-%20Fighting%20Breast%20Cancer.ipynb "Hyperparameter Tuning with SynapseML"
-
-[example 6]: notebooks/DeepLearning%20-%20CIFAR10%20Convolutional%20Network.ipynb "CIFAR10 CNTK CNN Evaluation"
-
-[example 7]: notebooks/OpenCV%20-%20Pipeline%20Image%20Transformations.ipynb "Pipeline Image Transformations"
-
-[example 8]: notebooks/DeepLearning%20-%20BiLSTM%20Medical%20Entity%20Extraction.ipynb "Medical Entity Extraction"
-
-[example 9]: notebooks/DeepLearning%20-%20Flower%20Image%20Classification.ipynb "Deep Flower Classification"
-
-[example 10]: notebooks/gpu/DeepLearning%20-%20Distributed%20CNTK%20training.ipynb "CIFAR10 CNTK CNN Training"
-
-[example 11]: notebooks/CyberML%20-%20Anomalous%20Access%20Detection.ipynb "Access Anomalies documenation, training and evaluation example"
-
-[example 12]: notebooks/Interpretability%20-%20Tabular%20SHAP%20explainer.ipynb "Interpretability - Tabular SHAP Explainer"
-
-[example 13]: notebooks/Interpretability%20-%20Image%20Explainers.ipynb "Interpretability - Image Explainers"
-
-[example 14]: notebooks/Interpretability%20-%20Text%20Explainers.ipynb "Interpretability - Text Explainers"
-
-## A short example
-
-Below is an excerpt from a simple example of using a pre-trained CNN to
-classify images in the CIFAR-10 dataset. View the whole source code in notebook [example 9].
-
-```python
-...
-import synapse.ml
-# Initialize CNTKModel and define input and output columns
-cntkModel = synapse.ml.cntk.CNTKModel() \
- .setInputCol("images").setOutputCol("output") \
- .setModelLocation(modelFile)
-# Train on dataset with internal spark pipeline
-scoredImages = cntkModel.transform(imagesWithLabels)
-...
-```
-
-See [other sample notebooks](notebooks/) as well as the SynapseML
-documentation for [Scala](http://mmlspark.azureedge.net/docs/scala/) and
-[PySpark](http://mmlspark.azureedge.net/docs/pyspark/).
+For quickstarts, documentation, demos, and examples please see our [website](https://aka.ms/spark).
## Setup and installation
@@ -149,7 +82,7 @@ the above example, or from python:
```python
import pyspark
spark = pyspark.sql.SparkSession.builder.appName("MyApp") \
- .config("spark.jars.packages", "com.microsoft.azure:synapseml:0.9.1") \
+ .config("spark.jars.packages", "com.microsoft.azure:synapseml_2.12:0.9.4") \
.config("spark.jars.repositories", "https://mmlspark.azureedge.net/maven") \
.getOrCreate()
import synapse.ml
@@ -162,7 +95,7 @@ your `build.sbt`:
```scala
resolvers += "SynapseML" at "https://mmlspark.azureedge.net/maven"
-libraryDependencies += "com.microsoft.azure" % "synapseml" % "0.9.1"
+libraryDependencies += "com.microsoft.azure" % "synapseml_2.12" % "0.9.4"
```
@@ -172,9 +105,9 @@ SynapseML can be conveniently installed on existing Spark clusters via the
`--packages` option, examples:
```bash
-spark-shell --packages com.microsoft.azure:synapseml:0.9.1
-pyspark --packages com.microsoft.azure:synapseml:0.9.1
-spark-submit --packages com.microsoft.azure:synapseml:0.9.1 MyApp.jar
+spark-shell --packages com.microsoft.azure:synapseml_2.12:0.9.4 --conf spark.jars.repositories=https://mmlspark.azureedge.net/maven
+pyspark --packages com.microsoft.azure:synapseml_2.12:0.9.4 --conf spark.jars.repositories=https://mmlspark.azureedge.net/maven
+spark-submit --packages com.microsoft.azure:synapseml_2.12:0.9.4 MyApp.jar --conf spark.jars.repositories=https://mmlspark.azureedge.net/maven
```
This can be used in other Spark contexts too. For example, you can use SynapseML
@@ -189,7 +122,7 @@ cloud](http://community.cloud.databricks.com), create a new [library from Maven
coordinates](https://docs.databricks.com/user-guide/libraries.html#libraries-from-maven-pypi-or-spark-packages)
in your workspace.
-For the coordinates use: `com.microsoft.azure:synapseml:0.9.1`
+For the coordinates use: `com.microsoft.azure:synapseml_2.12:0.9.4`
with the resolver: `https://mmlspark.azureedge.net/maven`. Ensure this library is
attached to your target cluster(s).
@@ -197,7 +130,7 @@ Finally, ensure that your Spark cluster has at least Spark 3.12 and Scala 2.12.
You can use SynapseML in both your Scala and PySpark notebooks. To get started with our example notebooks import the following databricks archive:
-`https://mmlspark.blob.core.windows.net/dbcs/SynapseMLExamplesv0.9.1.dbc`
+`https://mmlspark.blob.core.windows.net/dbcs/SynapseMLExamplesv0.9.4.dbc`
### Apache Livy and HDInsight
@@ -210,7 +143,7 @@ Excluding certain packages from the library may be necessary due to current issu
{
"name": "synapseml",
"conf": {
- "spark.jars.packages": "com.microsoft.azure:synapseml:0.9.1",
+ "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:0.9.4",
"spark.jars.repositories": "https://mmlspark.azureedge.net/maven",
"spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12"
}
@@ -224,7 +157,7 @@ In Azure Synapse, "spark.yarn.user.classpath.first" should be set to "true" to o
{
"name": "synapseml",
"conf": {
- "spark.jars.packages": "com.microsoft.azure:synapseml:0.9.1",
+ "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:0.9.4",
"spark.jars.repositories": "https://mmlspark.azureedge.net/maven",
"spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12",
"spark.yarn.user.classpath.first": "true"
@@ -277,7 +210,7 @@ and some necessary custom wrappers may be missing.
- [Conditional Image Retrieval](https://arxiv.org/abs/2007.07177)
-- [SynapseML: Unifying Machine Learning Ecosystems at Massive Scales](https://arxiv.org/abs/1810.08744)
+- [MMLSpark: Unifying Machine Learning Ecosystems at Massive Scales](https://arxiv.org/abs/1810.08744)
- [Flexible and Scalable Deep Learning with SynapseML](https://arxiv.org/abs/1804.04031)
@@ -293,7 +226,7 @@ and some necessary custom wrappers may be missing.
- Explore [our collaboration with Apache Spark] on image analysis.
-[website]: https://mmlspark.blob.core.windows.net/website/index.html "aka.ms/spark"
+[website]: https://microsoft.github.io/SynapseML/ "aka.ms/spark"
[the Spark+AI Summit 2018]: https://databricks.com/sparkaisummit/north-america/spark-summit-2018-keynotes#Intelligent-cloud "Developing for the Intelligent Cloud and Intelligent Edge"
@@ -301,16 +234,12 @@ and some necessary custom wrappers may be missing.
[the Spark+AI European Summit 2018]: https://youtu.be/N3ozCZXeOeU?t=472
-[our paper]: https://arxiv.org/abs/1804.04031 "Flexible and Scalable Deep Learning with SynapseML"
-
[help endangered species]: https://www.microsoft.com/en-us/ai/ai-lab-stories?activetab=pivot1:primaryr3 "Identifying snow leopards with AI"
[our collaboration with The MET and MIT]: https://www.microsoft.com/en-us/ai/ai-lab-stories?activetab=pivot1:primaryr4 "Generative art at the MET"
[our collaboration with Apache Spark]: https://blogs.technet.microsoft.com/machinelearning/2018/03/05/image-data-support-in-apache-spark/ "Image Data Support in Apache Spark"
-[SynapseML in Azure Machine Learning]: https://docs.microsoft.com/en-us/azure/machine-learning/preview/how-to-use-mmlspark "How to Use Microsoft Machine Learning Library for Apache Spark"
-
[SynapseML at the Spark Summit]: https://databricks.com/session/mmlspark-lessons-from-building-a-sparkml-compatible-machine-learning-library-for-apache-spark "MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library for Apache Spark"
## Contributing & feedback
@@ -343,8 +272,6 @@ Issue](https://help.github.com/articles/creating-an-issue/).
- [Microsoft Cognitive Toolkit](https://github.com/Microsoft/CNTK)
-- [Azure Machine Learning
- preview features](https://docs.microsoft.com/en-us/azure/machine-learning/preview)
_ApacheĀ®, Apache Spark, and SparkĀ® are either registered trademarks or
trademarks of the Apache Software Foundation in the United States and/or other
diff --git a/build.sbt b/build.sbt
index b6b1f40fdf..7c2d995ed4 100644
--- a/build.sbt
+++ b/build.sbt
@@ -259,6 +259,7 @@ val settings = Seq(
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false),
autoAPIMappings := true,
pomPostProcess := pomPostFunc,
+ sbtPlugin := false
)
ThisBuild / publishMavenStyle := true
@@ -361,10 +362,16 @@ testWebsiteDocs := {
)
}
-sonatypeProjectHosting := Some(
+ThisBuild / sonatypeProjectHosting := Some(
GitHubHosting("Azure", "SynapseML", "mmlspark-support@microsot.com"))
-homepage := Some(url("https://github.com/Microsoft/SynapseML"))
-developers := List(
+ThisBuild / homepage := Some(url("https://github.com/Microsoft/SynapseML"))
+ThisBuild / scmInfo := Some(
+ ScmInfo(
+ url("https://github.com/Azure/SynapseML"),
+ "scm:git@github.com:Azure/SynapseML.git"
+ )
+)
+ThisBuild / developers := List(
Developer("mhamilton723", "Mark Hamilton",
"mmlspark-support@microsoft.com", url("https://github.com/mhamilton723")),
Developer("imatiach-msft", "Ilya Matiach",
@@ -373,9 +380,9 @@ developers := List(
"mmlspark-support@microsoft.com", url("https://github.com/drdarshan"))
)
-licenses += ("MIT", url("https://github.com/Microsoft/SynapseML/blob/master/LICENSE"))
+ThisBuild / licenses += ("MIT", url("https://github.com/Microsoft/SynapseML/blob/master/LICENSE"))
-credentials += Credentials("Sonatype Nexus Repository Manager",
+ThisBuild / credentials += Credentials("Sonatype Nexus Repository Manager",
"oss.sonatype.org",
Secrets.nexusUsername,
Secrets.nexusPassword)
diff --git a/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/CognitiveServiceBase.scala b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/CognitiveServiceBase.scala
index 4f684687bd..0d7eb466a3 100644
--- a/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/CognitiveServiceBase.scala
+++ b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/CognitiveServiceBase.scala
@@ -332,10 +332,4 @@ abstract class CognitiveServicesBaseNoHandler(val uid: String) extends Transform
}
abstract class CognitiveServicesBase(uid: String) extends
- CognitiveServicesBaseNoHandler(uid) with HasHandler {
- setDefault(handler -> HandlingUtils.advancedUDF(100)) //scalastyle:ignore magic.number
-
- override def handlingFunc(client: CloseableHttpClient,
- request: HTTPRequestData): HTTPResponseData =
- getHandler(client, request)
-}
+ CognitiveServicesBaseNoHandler(uid) with HasHandler
diff --git a/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/ComputerVision.scala b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/ComputerVision.scala
index f990e82a8e..0519242e2a 100644
--- a/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/ComputerVision.scala
+++ b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/ComputerVision.scala
@@ -232,7 +232,7 @@ trait BasicAsyncReply extends HasAsyncReply {
request: HTTPRequestData): HTTPResponseData = {
val response = HandlingUtils.advanced(getBackoffs: _*)(client, request)
if (response.statusLine.statusCode == 202) {
- val location = new URI(response.headers.filter(_.name == "Operation-Location").head.value)
+ val location = new URI(response.headers.filter(_.name.toLowerCase() == "operation-location").head.value)
val maxTries = getMaxPollingRetries
val key = request.headers.find(_.name == "Ocp-Apim-Subscription-Key").map(_.value)
val it = (0 to maxTries).toIterator.flatMap { _ =>
@@ -255,7 +255,6 @@ trait BasicAsyncReply extends HasAsyncReply {
}
}
-
trait HasAsyncReply extends Params {
val backoffs: IntArrayParam = new IntArrayParam(
this, "backoffs", "array of backoffs to use in the handler")
diff --git a/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/FormOntologyLearner.scala b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/FormOntologyLearner.scala
index 4dd0fe5218..a53d787b64 100644
--- a/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/FormOntologyLearner.scala
+++ b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/FormOntologyLearner.scala
@@ -53,21 +53,23 @@ class FormOntologyLearner(override val uid: String) extends Estimator[FormOntolo
}
override def fit(dataset: Dataset[_]): FormOntologyTransformer = {
- val fromRow = AnalyzeResponse.makeFromRowConverter
-
- def combine(st1: StructType, st2: StructType): StructType = {
- FormOntologyLearner.combineDataTypes(st1, st2).asInstanceOf[StructType]
- }
-
- val mergedSchema = dataset.toDF()
- .select(col(getInputCol))
- .map(extractOntology(fromRow))(Encoders.kryo[StructType])
- .reduce(combine _)
-
- new FormOntologyTransformer()
- .setInputCol(getInputCol)
- .setOutputCol(getOutputCol)
- .setOntology(mergedSchema)
+ logFit({
+ val fromRow = AnalyzeResponse.makeFromRowConverter
+
+ def combine(st1: StructType, st2: StructType): StructType = {
+ FormOntologyLearner.combineDataTypes(st1, st2).asInstanceOf[StructType]
+ }
+
+ val mergedSchema = dataset.toDF()
+ .select(col(getInputCol))
+ .map(extractOntology(fromRow))(Encoders.kryo[StructType])
+ .reduce(combine _)
+
+ new FormOntologyTransformer()
+ .setInputCol(getInputCol)
+ .setOutputCol(getOutputCol)
+ .setOntology(mergedSchema)
+ })
}
override def copy(extra: ParamMap): Estimator[FormOntologyTransformer] = defaultCopy(extra)
@@ -109,11 +111,13 @@ class FormOntologyTransformer(override val uid: String) extends Model[FormOntolo
}
override def transform(dataset: Dataset[_]): DataFrame = {
- val fromRow = AnalyzeResponse.makeFromRowConverter
- val convertToOntologyUDF = UDFUtils.oldUdf(convertToOntology(fromRow) _, getOntology)
+ logTransform[DataFrame]({
+ val fromRow = AnalyzeResponse.makeFromRowConverter
+ val convertToOntologyUDF = UDFUtils.oldUdf(convertToOntology(fromRow) _, getOntology)
- dataset.toDF()
- .withColumn(getOutputCol, convertToOntologyUDF(col(getInputCol)))
+ dataset.toDF()
+ .withColumn(getOutputCol, convertToOntologyUDF(col(getInputCol)))
+ })
}
override def transformSchema(schema: StructType): StructType = {
diff --git a/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/MultivariateAnomalyDetection.scala b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/MultivariateAnomalyDetection.scala
new file mode 100644
index 0000000000..a47f197b7d
--- /dev/null
+++ b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/MultivariateAnomalyDetection.scala
@@ -0,0 +1,397 @@
+// Copyright (C) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License. See LICENSE in project root for information.
+
+package com.microsoft.azure.synapse.ml.cognitive
+
+import com.microsoft.azure.synapse.ml.build.BuildInfo
+import com.microsoft.azure.synapse.ml.codegen.Wrappable
+import com.microsoft.azure.synapse.ml.cognitive.MADJsonProtocol._
+import com.microsoft.azure.synapse.ml.core.contracts.HasOutputCol
+import com.microsoft.azure.synapse.ml.core.schema.DatasetExtensions
+import com.microsoft.azure.synapse.ml.io.http.HandlingUtils.{convertAndClose, sendWithRetries}
+import com.microsoft.azure.synapse.ml.io.http._
+import com.microsoft.azure.synapse.ml.logging.BasicLogging
+import com.microsoft.azure.synapse.ml.stages.{DropColumns, Lambda}
+import org.apache.commons.io.IOUtils
+import org.apache.http.client.methods.HttpGet
+import org.apache.http.entity.{AbstractHttpEntity, ContentType, StringEntity}
+import org.apache.http.impl.client.CloseableHttpClient
+import org.apache.spark.ml._
+import org.apache.spark.ml.param.{Param, ParamMap, ServiceParam}
+import org.apache.spark.ml.util._
+import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
+import org.apache.spark.sql.functions.{col, lit, struct}
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.{DataFrame, Dataset, Row}
+import spray.json._
+
+import java.net.URI
+import java.util.concurrent.TimeoutException
+import scala.concurrent.blocking
+import scala.language.existentials
+
+
+trait HasMADSource extends HasServiceParams {
+ val source = new ServiceParam[String](this, "source", "The blob link to the input data. " +
+ "It should be a zipped folder containing csv files. Each csv file should has two columns with header 'timestamp'" +
+ " and 'value' (case sensitive). The file name will be used as the variable name. The variables used for" +
+ " detection should be exactly the same as for training. Please refer to the sample data to prepare your" +
+ " own data accordingly.", isRequired = true)
+
+ def setSource(v: String): this.type = setScalarParam(source, v)
+
+ def setSourceCol(v: String): this.type = setVectorParam(source, v)
+
+ def getSource: String = getScalarParam(source)
+
+ def getSourceCol: String = getVectorParam(source)
+}
+
+trait HasMADStartTime extends HasServiceParams {
+ val startTime = new ServiceParam[String](this, "startTime", "A required field, start time" +
+ " of data to be used for detection/generating multivariate anomaly detection model, should be date-time.",
+ isRequired = true)
+
+ def setStartTime(v: String): this.type = setScalarParam(startTime, v)
+
+ def setStartTimeCol(v: String): this.type = setVectorParam(startTime, v)
+
+ def getStartTime: String = getScalarParam(startTime)
+
+ def getStartTimeCol: String = getVectorParam(startTime)
+
+}
+
+trait HasMADEndTime extends HasServiceParams {
+ val endTime = new ServiceParam[String](this, "endTime", "A required field, end time of data" +
+ " to be used for detection/generating multivariate anomaly detection model, should be date-time.",
+ isRequired = true)
+
+ def setEndTime(v: String): this.type = setScalarParam(endTime, v)
+
+ def setEndTimeCol(v: String): this.type = setVectorParam(endTime, v)
+
+ def getEndTime: String = getScalarParam(endTime)
+
+ def getEndTimeCol: String = getVectorParam(endTime)
+}
+
+trait MADBase extends HasAsyncReply with HasMADSource with HasMADStartTime with HasMADEndTime
+ with HasCognitiveServiceInput with HasInternalJsonOutputParser with HasSetLocation with Wrappable
+ with HTTPParams with HasOutputCol with HasURL with ComplexParamsWritable
+ with HasSubscriptionKey with HasErrorCol with BasicLogging {
+
+ setDefault(
+ outputCol -> (this.uid + "_output"),
+ errorCol -> (this.uid + "_error"))
+
+ protected def queryForResult(key: Option[String],
+ client: CloseableHttpClient,
+ location: URI): Option[HTTPResponseData] = {
+ val get = new HttpGet()
+ get.setURI(location)
+ key.foreach(get.setHeader("Ocp-Apim-Subscription-Key", _))
+ get.setHeader("User-Agent", s"synapseml/${BuildInfo.version}${HeaderValues.PlatformInfo}")
+ val resp = convertAndClose(sendWithRetries(client, get, getBackoffs))
+ get.releaseConnection()
+ val fields = IOUtils.toString(resp.entity.get.content, "UTF-8").parseJson.asJsObject.fields
+ val status = if (fields.keySet.contains("modelInfo")) {
+ fields("modelInfo").asInstanceOf[JsObject].fields
+ .get("status").map(_.convertTo[String]).get.toLowerCase()
+ } else if (fields.keySet.contains("summary")) {
+ fields("summary").asInstanceOf[JsObject]
+ .fields.get("status").map(_.convertTo[String]).get.toLowerCase()
+ } else {
+ "None"
+ }
+ status match {
+ case "ready" | "failed" => Some(resp)
+ case "created" | "running" => None
+ case s => throw new RuntimeException(s"Received unknown status code: $s")
+ }
+ }
+
+ protected def handlingFunc(client: CloseableHttpClient,
+ request: HTTPRequestData): HTTPResponseData = {
+ val response = HandlingUtils.advanced(getBackoffs: _*)(client, request)
+ if (response.statusLine.statusCode == 201) {
+ val location = new URI(response.headers.filter(_.name == "Location").head.value)
+ val maxTries = getMaxPollingRetries
+ val key = request.headers.find(_.name == "Ocp-Apim-Subscription-Key").map(_.value)
+ val it = (0 to maxTries).toIterator.flatMap { _ =>
+ queryForResult(key, client, location).orElse({
+ blocking {
+ Thread.sleep(getPollingDelay.toLong)
+ }
+ None
+ })
+ }
+ if (it.hasNext) {
+ it.next()
+ } else {
+ throw new TimeoutException(
+ s"Querying for results did not complete within $maxTries tries")
+ }
+ } else {
+ response
+ }
+ }
+}
+
+object MultivariateAnomalyEstimator extends ComplexParamsReadable[MultivariateAnomalyEstimator] with Serializable
+
+class MultivariateAnomalyEstimator(override val uid: String) extends Estimator[DetectMultivariateAnomaly]
+ with MADBase {
+ logClass()
+
+ def this() = this(Identifiable.randomUID("MultivariateAnomalyModel"))
+
+ def urlPath: String = "anomalydetector/v1.1-preview/multivariate/models"
+
+ val slidingWindow = new ServiceParam[Int](this, "slidingWindow", "An optional field, indicates" +
+ " how many history points will be used to determine the anomaly score of one subsequent point.", {
+ case Left(x) => (x >= 28) && (x <= 2880)
+ case Right(_) => true
+ }, isRequired = true)
+
+ def setSlidingWindow(v: Int): this.type = setScalarParam(slidingWindow, v)
+
+ def setSlidingWindowCol(v: String): this.type = setVectorParam(slidingWindow, v)
+
+ def getSlidingWindow: Int = getScalarParam(slidingWindow)
+
+ def getSlidingWindowCol: String = getVectorParam(slidingWindow)
+
+ val alignMode = new ServiceParam[String](this, "alignMode", "An optional field, indicates how " +
+ "we align different variables into the same time-range which is required by the model.{Inner, Outer}", {
+ case Left(s) => Set("inner", "outer")(s.toLowerCase)
+ case Right(_) => true
+ })
+
+ def setAlignMode(v: String): this.type = setScalarParam(alignMode, v.toLowerCase.capitalize)
+
+ def setAlignModeCol(v: String): this.type = setVectorParam(alignMode, v)
+
+ def getAlignMode: String = getScalarParam(alignMode)
+
+ def getAlignModeCol: String = getVectorParam(alignMode)
+
+ val fillNAMethod = new ServiceParam[String](this, "fillNAMethod", "An optional field, indicates how missed " +
+ "values will be filled with. Can not be set to NotFill, when alignMode is Outer.{Previous, Subsequent," +
+ " Linear, Zero, Fixed}", {
+ case Left(s) => Set("previous", "subsequent", "linear", "zero", "fixed")(s.toLowerCase)
+ case Right(_) => true
+ })
+
+ def setFillNAMethod(v: String): this.type = setScalarParam(fillNAMethod, v.toLowerCase.capitalize)
+
+ def setFillNAMethodCol(v: String): this.type = setVectorParam(fillNAMethod, v)
+
+ def getFillNAMethod: String = getScalarParam(fillNAMethod)
+
+ def getFillNAMethodCol: String = getVectorParam(fillNAMethod)
+
+ val paddingValue = new ServiceParam[Int](this, "paddingValue", "optional field, is only useful" +
+ " if FillNAMethod is set to Fixed.")
+
+ def setPaddingValue(v: Int): this.type = setScalarParam(paddingValue, v)
+
+ def setPaddingValueCol(v: String): this.type = setVectorParam(paddingValue, v)
+
+ def getPaddingValue: Int = getScalarParam(paddingValue)
+
+ def getPaddingValueCol: String = getVectorParam(paddingValue)
+
+ val displayName = new ServiceParam[String](this, "displayName", "optional field," +
+ " name of the model")
+
+ def setDisplayName(v: String): this.type = setScalarParam(displayName, v)
+
+ def setDisplayNameCol(v: String): this.type = setVectorParam(displayName, v)
+
+ def getDisplayName: String = getScalarParam(displayName)
+
+ def getDisplayNameCol: String = getVectorParam(displayName)
+
+ val diagnosticsInfo = new ServiceParam[DiagnosticsInfo](this, "diagnosticsInfo",
+ "diagnosticsInfo for training a multivariate anomaly detection model")
+
+ def setDiagnosticsInfo(v: DiagnosticsInfo): this.type = setScalarParam(diagnosticsInfo, v)
+
+ def getDiagnosticsInfo: DiagnosticsInfo = getScalarParam(diagnosticsInfo)
+
+ override protected def prepareEntity: Row => Option[AbstractHttpEntity] = {
+ r =>
+ Some(new StringEntity(Map("source" -> getValue(r, source).toJson,
+ "startTime" -> getValue(r, startTime).toJson,
+ "endTime" -> getValue(r, endTime).toJson,
+ "slidingWindow" -> getValue(r, slidingWindow).toJson,
+ "alignPolicy" -> Map("alignMode" -> getValueOpt(r, alignMode).toJson,
+ "fillNAMethod" -> getValueOpt(r, fillNAMethod).toJson,
+ "paddingValue" -> getValueOpt(r, paddingValue).toJson).toJson,
+ "displayName" -> getValueOpt(r, displayName).toJson)
+ .toJson.compactPrint, ContentType.APPLICATION_JSON))
+ }
+
+ override def responseDataType: DataType = MAEResponse.schema
+
+ protected def getInternalTransformer(schema: StructType): PipelineModel = {
+ val dynamicParamColName = DatasetExtensions.findUnusedColumnName("dynamic", schema)
+ val badColumns = getVectorParamMap.values.toSet.diff(schema.fieldNames.toSet)
+ assert(badColumns.isEmpty,
+ s"Could not find dynamic columns: $badColumns in columns: ${schema.fieldNames.toSet}")
+
+ val missingRequiredParams = this.getRequiredParams.filter {
+ p => this.get(p).isEmpty && this.getDefault(p).isEmpty
+ }
+ assert(missingRequiredParams.isEmpty,
+ s"Missing required params: ${missingRequiredParams.map(s => s.name).mkString("(", ", ", ")")}")
+
+ val dynamicParamCols = getVectorParamMap.values.toList.map(col) match {
+ case Nil => Seq(lit(false).alias("placeholder"))
+ case l => l
+ }
+
+ val stages = Array(
+ Lambda(_.withColumn(dynamicParamColName, struct(dynamicParamCols: _*))),
+ new SimpleHTTPTransformer()
+ .setInputCol(dynamicParamColName)
+ .setOutputCol(getOutputCol)
+ .setInputParser(getInternalInputParser(schema))
+ .setOutputParser(getInternalOutputParser(schema))
+ .setHandler(handlingFunc)
+ .setConcurrency(getConcurrency)
+ .setConcurrentTimeout(get(concurrentTimeout))
+ .setErrorCol(getErrorCol),
+ new DropColumns().setCol(dynamicParamColName)
+ )
+
+ NamespaceInjections.pipelineModel(stages)
+ }
+
+ override def fit(dataset: Dataset[_]): DetectMultivariateAnomaly = {
+ logFit({
+ import MADJsonProtocol._
+
+ val df = getInternalTransformer(dataset.schema)
+ .transform(dataset)
+ .withColumn("diagnosticsInfo", col(getOutputCol)
+ .getField("modelInfo").getField("diagnosticsInfo"))
+ .withColumn("modelId", col(getOutputCol).getField("modelId"))
+ .select(getOutputCol, "modelId", "diagnosticsInfo")
+ .collect()
+ this.setDiagnosticsInfo(df.head.get(2).asInstanceOf[GenericRowWithSchema]
+ .json.parseJson.convertTo[DiagnosticsInfo])
+ val modelId = df.head.getString(1)
+ new DetectMultivariateAnomaly()
+ .setSubscriptionKey(getSubscriptionKey)
+ .setLocation(getUrl.split("/".toCharArray)(2).split(".".toCharArray).head)
+ .setModelId(modelId)
+ .setSourceCol(getSourceCol)
+ .setStartTime(getStartTime)
+ .setEndTime(getEndTime)
+ })
+ }
+
+ override def copy(extra: ParamMap): Estimator[DetectMultivariateAnomaly] = defaultCopy(extra)
+
+ override def transformSchema(schema: StructType): StructType = {
+ getInternalTransformer(schema).transformSchema(schema)
+ }
+}
+
+object DetectMultivariateAnomaly extends ComplexParamsReadable[DetectMultivariateAnomaly] with Serializable
+
+class DetectMultivariateAnomaly(override val uid: String) extends Model[DetectMultivariateAnomaly]
+ with MADBase {
+ logClass()
+
+ def this() = this(Identifiable.randomUID("DetectMultivariateAnomaly"))
+
+ def urlPath: String = "anomalydetector/v1.1-preview/multivariate/models/"
+
+ val modelId = new ServiceParam[String](this, "modelId", "Format - uuid. Model identifier.",
+ isRequired = true)
+
+ def setModelId(v: String): this.type = setScalarParam(modelId, v)
+
+ def setModelIdCol(v: String): this.type = setVectorParam(modelId, v)
+
+ def getModelId: String = getScalarParam(modelId)
+
+ def getModelIdCol: String = getVectorParam(modelId)
+
+ override protected def prepareEntity: Row => Option[AbstractHttpEntity] = { row =>
+ Some(new StringEntity(DMARequest(
+ getValue(row, source),
+ getValue(row, startTime),
+ getValue(row, endTime)
+ ).toJson.compactPrint))
+ }
+
+ override protected def prepareUrl: Row => String = {
+ val urlParams: Array[ServiceParam[Any]] =
+ getUrlParams.asInstanceOf[Array[ServiceParam[Any]]];
+ // This semicolon is needed to avoid argument confusion
+ { row: Row =>
+ val base = getUrl + s"${getValue(row, modelId)}/detect"
+ val appended = if (!urlParams.isEmpty) {
+ "?" + URLEncodingUtils.format(urlParams.flatMap(p =>
+ getValueOpt(row, p).map(v => p.name -> p.toValueString(v))
+ ).toMap)
+ } else {
+ ""
+ }
+ base + appended
+ }
+ }
+
+ override def responseDataType: DataType = DMAResponse.schema
+
+ protected def getInternalTransformer(schema: StructType): PipelineModel = {
+ val dynamicParamColName = DatasetExtensions.findUnusedColumnName("dynamic", schema)
+ val badColumns = getVectorParamMap.values.toSet.diff(schema.fieldNames.toSet)
+ assert(badColumns.isEmpty,
+ s"Could not find dynamic columns: $badColumns in columns: ${schema.fieldNames.toSet}")
+
+ val missingRequiredParams = this.getRequiredParams.filter {
+ p => this.get(p).isEmpty && this.getDefault(p).isEmpty
+ }
+ assert(missingRequiredParams.isEmpty,
+ s"Missing required params: ${missingRequiredParams.map(s => s.name).mkString("(", ", ", ")")}")
+
+ val dynamicParamCols = getVectorParamMap.values.toList.map(col) match {
+ case Nil => Seq(lit(false).alias("placeholder"))
+ case l => l
+ }
+
+ val stages = Array(
+ Lambda(_.withColumn(dynamicParamColName, struct(dynamicParamCols: _*))),
+ new SimpleHTTPTransformer()
+ .setInputCol(dynamicParamColName)
+ .setOutputCol(getOutputCol)
+ .setInputParser(getInternalInputParser(schema))
+ .setOutputParser(getInternalOutputParser(schema))
+ .setHandler(handlingFunc)
+ .setConcurrency(getConcurrency)
+ .setConcurrentTimeout(get(concurrentTimeout))
+ .setErrorCol(getErrorCol),
+ new DropColumns().setCol(dynamicParamColName)
+ )
+
+ NamespaceInjections.pipelineModel(stages)
+ }
+
+ override def transform(dataset: Dataset[_]): DataFrame = {
+ logTransform[DataFrame](
+ getInternalTransformer(dataset.schema).transform(dataset)
+ )
+ }
+
+ override def copy(extra: ParamMap): DetectMultivariateAnomaly = defaultCopy(extra)
+
+ override def transformSchema(schema: StructType): StructType = {
+ getInternalTransformer(schema).transformSchema(schema)
+ }
+
+}
diff --git a/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/MultivariateAnomalyDetectorSchemas.scala b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/MultivariateAnomalyDetectorSchemas.scala
new file mode 100644
index 0000000000..cdcf9d635d
--- /dev/null
+++ b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/MultivariateAnomalyDetectorSchemas.scala
@@ -0,0 +1,93 @@
+// Copyright (C) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License. See LICENSE in project root for information.
+
+package com.microsoft.azure.synapse.ml.cognitive
+
+import com.microsoft.azure.synapse.ml.core.schema.SparkBindings
+import spray.json.{DefaultJsonProtocol, RootJsonFormat}
+
+// DMA stands for DetectMultivariateAnomaly
+object DMARequest extends SparkBindings[DMARequest]
+
+case class DMARequest(source: String,
+ startTime: String,
+ endTime: String)
+
+object DMAResponse extends SparkBindings[DMAResponse]
+
+case class DMAResponse(resultId: String,
+ summary: DMASummary,
+ results: Seq[DMAResult])
+
+case class DMASummary(status: String,
+ errors: Option[Seq[DMAError]],
+ variableStates: Option[Seq[DMAVariableState]],
+ setupInfo: DMASetupInfo)
+
+case class DMAError(code: String, message: String)
+
+case class DMAVariableState(variable: Option[String],
+ filledNARatio: Option[Double],
+ effectiveCount: Option[Int],
+ startTime: Option[String],
+ endTime: Option[String],
+ errors: Option[Seq[DMAError]])
+
+case class DMASetupInfo(source: String,
+ startTime: String,
+ endTime: String)
+
+case class DMAResult(timestamp: String, value: Option[DMAValue], errors: Option[Seq[DMAError]])
+
+case class DMAValue(contributors: Option[Seq[DMAContributor]],
+ isAnomaly: Boolean,
+ severity: Double,
+ score: Double)
+
+case class DMAContributor(contributionScore: Option[Double], variable: Option[String])
+
+// MAE stands for MultivariateAnomalyEstimator
+object MAERequest extends SparkBindings[MAERequest]
+
+case class MAERequest(source: String,
+ startTime: String,
+ endTime: String,
+ slidingWindow: Option[Int],
+ alignPolicy: Option[AlignPolicy],
+ displayName: Option[String])
+
+object MAEResponse extends SparkBindings[MAEResponse]
+
+case class MAEResponse(modelId: String,
+ createdTime: String,
+ lastUpdatedTime: String,
+ modelInfo: MAEModelInfo)
+
+case class MAEModelInfo(slidingWindow: Option[Int],
+ alignPolicy: Option[AlignPolicy],
+ source: String,
+ startTime: String,
+ endTime: String,
+ displayName: Option[String],
+ status: String,
+ errors: Option[Seq[DMAError]],
+ diagnosticsInfo: Option[DiagnosticsInfo])
+
+case class AlignPolicy(alignMode: Option[String], fillNAMethod: Option[String], paddingValue: Option[Int])
+
+case class DiagnosticsInfo(modelState: Option[ModelState], variableStates: Option[Seq[DMAVariableState]])
+
+case class ModelState(epochIds: Option[Seq[Int]],
+ trainLosses: Option[Seq[Double]],
+ validationLosses: Option[Seq[Double]],
+ latenciesInSeconds: Option[Seq[Double]])
+
+object MADJsonProtocol extends DefaultJsonProtocol {
+ implicit val DMAReqEnc: RootJsonFormat[DMARequest] = jsonFormat3(DMARequest.apply)
+ implicit val EEnc: RootJsonFormat[DMAError] = jsonFormat2(DMAError.apply)
+ implicit val VSEnc: RootJsonFormat[DMAVariableState] = jsonFormat6(DMAVariableState.apply)
+ implicit val MSEnc: RootJsonFormat[ModelState] = jsonFormat4(ModelState.apply)
+ implicit val DIEnc: RootJsonFormat[DiagnosticsInfo] = jsonFormat2(DiagnosticsInfo.apply)
+ implicit val APEnc: RootJsonFormat[AlignPolicy] = jsonFormat3(AlignPolicy.apply)
+ implicit val MAEReqEnc: RootJsonFormat[MAERequest] = jsonFormat6(MAERequest.apply)
+}
diff --git a/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/TextAnalytics.scala b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/TextAnalytics.scala
index 57e9185b9d..d398f25b8e 100644
--- a/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/TextAnalytics.scala
+++ b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/TextAnalytics.scala
@@ -4,24 +4,28 @@
package com.microsoft.azure.synapse.ml.cognitive
import com.microsoft.azure.synapse.ml.core.schema.DatasetExtensions
-import com.microsoft.azure.synapse.ml.io.http.SimpleHTTPTransformer
+import com.microsoft.azure.synapse.ml.io.http.{HasHandler, SimpleHTTPTransformer}
import com.microsoft.azure.synapse.ml.logging.BasicLogging
import com.microsoft.azure.synapse.ml.stages.{DropColumns, Lambda, UDFTransformer}
import org.apache.http.client.methods.{HttpPost, HttpRequestBase}
import org.apache.http.entity.{AbstractHttpEntity, StringEntity}
import org.apache.spark.injections.UDFUtils
-import org.apache.spark.ml.param.ServiceParam
+import org.apache.spark.ml.param._
import org.apache.spark.ml.util._
import org.apache.spark.ml.{ComplexParamsReadable, NamespaceInjections, PipelineModel, Transformer}
import org.apache.spark.sql.Row
+import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
+import org.apache.spark.sql.expressions.UserDefinedFunction
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
import spray.json.DefaultJsonProtocol._
import spray.json._
import java.net.URI
+import java.util
+import scala.collection.JavaConverters._
-abstract class TextAnalyticsBase(override val uid: String) extends CognitiveServicesBase(uid)
+abstract class TextAnalyticsBase(override val uid: String) extends CognitiveServicesBaseNoHandler(uid)
with HasCognitiveServiceInput with HasInternalJsonOutputParser with HasSetLocation
with HasSetLinkedService {
@@ -89,40 +93,20 @@ abstract class TextAnalyticsBase(override val uid: String) extends CognitiveServ
override protected def prepareEntity: Row => Option[AbstractHttpEntity] = { _ => None }
- override protected def getInternalTransformer(schema: StructType): PipelineModel = {
- val dynamicParamColName = DatasetExtensions.findUnusedColumnName("dynamic", schema)
-
- val missingRequiredParams = this.getRequiredParams.filter {
- p => this.get(p).isEmpty && this.getDefault(p).isEmpty
- }
- assert(missingRequiredParams.isEmpty,
- s"Missing required params: ${missingRequiredParams.map(s => s.name).mkString("(", ", ", ")")}")
-
- def reshapeToArray(parameterName: String): Option[(Transformer, String, String)] = {
- val reshapedColName = DatasetExtensions.findUnusedColumnName(parameterName, schema)
- getVectorParamMap.get(parameterName).flatMap {
- case c if schema(c).dataType == StringType =>
- Some((Lambda(_.withColumn(reshapedColName, array(col(getVectorParam(parameterName))))),
- getVectorParam(parameterName),
- reshapedColName))
- case _ => None
- }
+ protected def reshapeToArray(schema: StructType, parameterName: String): Option[(Transformer, String, String)] = {
+ val reshapedColName = DatasetExtensions.findUnusedColumnName(parameterName, schema)
+ getVectorParamMap.get(parameterName).flatMap {
+ case c if schema(c).dataType == StringType =>
+ Some((Lambda(_.withColumn(reshapedColName, array(col(getVectorParam(parameterName))))),
+ getVectorParam(parameterName),
+ reshapedColName))
+ case _ => None
}
+ }
- val reshapeCols = Seq(reshapeToArray("text"), reshapeToArray("language")).flatten
-
- val newColumnMapping = reshapeCols.map {
- case (_, oldCol, newCol) => (oldCol, newCol)
- }.toMap
-
- val columnsToGroup = getVectorParamMap.map { case (_, oldCol) =>
- val newCol = newColumnMapping.getOrElse(oldCol, oldCol)
- col(newCol).alias(oldCol)
- }.toSeq
-
+ protected def unpackBatchUDF: UserDefinedFunction = {
val innerFields = innerResponseDataType.fields.filter(_.name != "id")
-
- val unpackBatchUDF = UDFUtils.oldUdf({ rowOpt: Row =>
+ UDFUtils.oldUdf({ rowOpt: Row =>
Option(rowOpt).map { row =>
val documents = row.getSeq[Row](1).map(doc =>
(doc.getString(0).toInt, doc)).toMap
@@ -141,6 +125,32 @@ abstract class TextAnalyticsBase(override val uid: String) extends CognitiveServ
}.add("error-message", StringType)
)
)
+ }
+
+ override protected def getInternalTransformer(schema: StructType): PipelineModel = {
+ val dynamicParamColName = DatasetExtensions.findUnusedColumnName("dynamic", schema)
+ val badColumns = getVectorParamMap.values.toSet.diff(schema.fieldNames.toSet)
+ assert(badColumns.isEmpty,
+ s"Could not find dynamic input columns: $badColumns in columns: ${schema.fieldNames.toSet}")
+
+ val missingRequiredParams = this.getRequiredParams.filter {
+ p => this.get(p).isEmpty && this.getDefault(p).isEmpty
+ }
+ assert(missingRequiredParams.isEmpty,
+ s"Missing required params: ${missingRequiredParams.map(s => s.name).mkString("(", ", ", ")")}")
+
+ val reshapeCols = Seq(
+ reshapeToArray(schema, "text"),
+ reshapeToArray(schema, "language")).flatten
+
+ val newColumnMapping = reshapeCols.map {
+ case (_, oldCol, newCol) => (oldCol, newCol)
+ }.toMap
+
+ val columnsToGroup = getVectorParamMap.map { case (_, oldCol) =>
+ val newCol = newColumnMapping.getOrElse(oldCol, oldCol)
+ col(newCol).alias(oldCol)
+ }.toSeq
val stages = reshapeCols.map(_._1).toArray ++ Array(
Lambda(_.withColumn(
@@ -151,7 +161,7 @@ abstract class TextAnalyticsBase(override val uid: String) extends CognitiveServ
.setOutputCol(getOutputCol)
.setInputParser(getInternalInputParser(schema))
.setOutputParser(getInternalOutputParser(schema))
- .setHandler(getHandler)
+ .setHandler(handlingFunc)
.setConcurrency(getConcurrency)
.setConcurrentTimeout(get(concurrentTimeout))
.setErrorCol(getErrorCol),
@@ -204,7 +214,7 @@ trait HasStringIndexType extends HasServiceParams {
object TextSentimentV2 extends ComplexParamsReadable[TextSentimentV2]
class TextSentimentV2(override val uid: String)
- extends TextAnalyticsBase(uid) with BasicLogging {
+ extends TextAnalyticsBase(uid) with BasicLogging with HasHandler {
logClass()
def this() = this(Identifiable.randomUID("TextSentimentV2"))
@@ -218,7 +228,7 @@ class TextSentimentV2(override val uid: String)
object LanguageDetectorV2 extends ComplexParamsReadable[LanguageDetectorV2]
class LanguageDetectorV2(override val uid: String)
- extends TextAnalyticsBase(uid) with BasicLogging {
+ extends TextAnalyticsBase(uid) with BasicLogging with HasHandler {
logClass()
def this() = this(Identifiable.randomUID("LanguageDetectorV2"))
@@ -231,7 +241,7 @@ class LanguageDetectorV2(override val uid: String)
object EntityDetectorV2 extends ComplexParamsReadable[EntityDetectorV2]
class EntityDetectorV2(override val uid: String)
- extends TextAnalyticsBase(uid) with BasicLogging {
+ extends TextAnalyticsBase(uid) with BasicLogging with HasHandler {
logClass()
def this() = this(Identifiable.randomUID("EntityDetectorV2"))
@@ -243,7 +253,7 @@ class EntityDetectorV2(override val uid: String)
object NERV2 extends ComplexParamsReadable[NERV2]
-class NERV2(override val uid: String) extends TextAnalyticsBase(uid) with BasicLogging {
+class NERV2(override val uid: String) extends TextAnalyticsBase(uid) with BasicLogging with HasHandler {
logClass()
def this() = this(Identifiable.randomUID("NERV2"))
@@ -256,7 +266,7 @@ class NERV2(override val uid: String) extends TextAnalyticsBase(uid) with BasicL
object KeyPhraseExtractorV2 extends ComplexParamsReadable[KeyPhraseExtractorV2]
class KeyPhraseExtractorV2(override val uid: String)
- extends TextAnalyticsBase(uid) with BasicLogging {
+ extends TextAnalyticsBase(uid) with BasicLogging with HasHandler {
logClass()
def this() = this(Identifiable.randomUID("KeyPhraseExtractorV2"))
@@ -266,10 +276,12 @@ class KeyPhraseExtractorV2(override val uid: String)
def urlPath: String = "/text/analytics/v2.0/keyPhrases"
}
+trait TAV3Mixins extends HasModelVersion with HasShowStats with HasStringIndexType with BasicLogging with HasHandler
+
object TextSentiment extends ComplexParamsReadable[TextSentiment]
class TextSentiment(override val uid: String)
- extends TextAnalyticsBase(uid) with HasModelVersion with HasShowStats with HasStringIndexType with BasicLogging {
+ extends TextAnalyticsBase(uid) with TAV3Mixins {
logClass()
def this() = this(Identifiable.randomUID("TextSentiment"))
@@ -295,7 +307,7 @@ class TextSentiment(override val uid: String)
object KeyPhraseExtractor extends ComplexParamsReadable[KeyPhraseExtractor]
class KeyPhraseExtractor(override val uid: String)
- extends TextAnalyticsBase(uid) with HasModelVersion with HasShowStats with BasicLogging {
+ extends TextAnalyticsBase(uid) with TAV3Mixins {
logClass()
def this() = this(Identifiable.randomUID("KeyPhraseExtractor"))
@@ -308,7 +320,8 @@ class KeyPhraseExtractor(override val uid: String)
object NER extends ComplexParamsReadable[NER]
class NER(override val uid: String)
- extends TextAnalyticsBase(uid) with HasModelVersion with HasShowStats with HasStringIndexType with BasicLogging {
+ extends TextAnalyticsBase(uid) with HasModelVersion
+ with HasShowStats with HasStringIndexType with BasicLogging with HasHandler {
logClass()
def this() = this(Identifiable.randomUID("NER"))
@@ -321,7 +334,7 @@ class NER(override val uid: String)
object PII extends ComplexParamsReadable[PII]
class PII(override val uid: String)
- extends TextAnalyticsBase(uid) with HasModelVersion with HasShowStats with HasStringIndexType with BasicLogging {
+ extends TextAnalyticsBase(uid) with TAV3Mixins {
logClass()
def this() = this(Identifiable.randomUID("PII"))
@@ -344,7 +357,7 @@ class PII(override val uid: String)
object LanguageDetector extends ComplexParamsReadable[LanguageDetector]
class LanguageDetector(override val uid: String)
- extends TextAnalyticsBase(uid) with HasModelVersion with HasShowStats with BasicLogging {
+ extends TextAnalyticsBase(uid) with HasModelVersion with HasShowStats with BasicLogging with HasHandler {
logClass()
def this() = this(Identifiable.randomUID("LanguageDetector"))
@@ -357,7 +370,7 @@ class LanguageDetector(override val uid: String)
object EntityDetector extends ComplexParamsReadable[EntityDetector]
class EntityDetector(override val uid: String)
- extends TextAnalyticsBase(uid) with HasModelVersion with HasShowStats with HasStringIndexType with BasicLogging {
+ extends TextAnalyticsBase(uid) with TAV3Mixins {
logClass()
def this() = this(Identifiable.randomUID("EntityDetector"))
@@ -366,3 +379,211 @@ class EntityDetector(override val uid: String)
def urlPath: String = "/text/analytics/v3.1/entities/linking"
}
+
+
+class TextAnalyzeTaskParam(parent: Params,
+ name: String,
+ doc: String,
+ isValid: Seq[TAAnalyzeTask] => Boolean = (_: Seq[TAAnalyzeTask]) => true)
+ (@transient implicit val dataFormat: JsonFormat[TAAnalyzeTask])
+ extends JsonEncodableParam[Seq[TAAnalyzeTask]](parent, name, doc, isValid) {
+ type ValueType = TAAnalyzeTask
+
+ override def w(value: Seq[TAAnalyzeTask]): ParamPair[Seq[TAAnalyzeTask]] = super.w(value)
+
+ def w(value: java.util.ArrayList[util.HashMap[String, Any]]): ParamPair[Seq[TAAnalyzeTask]] =
+ super.w(value.asScala.toArray.map(hashMapToTAAnalyzeTask))
+
+ def hashMapToTAAnalyzeTask(value: util.HashMap[String, Any]): TAAnalyzeTask = {
+ if (!value.containsKey("parameters")) {
+ throw new IllegalArgumentException("Task optiosn must include 'parameters' value")
+ }
+ if (value.size() > 1) {
+ throw new IllegalArgumentException("Task options should only include 'parameters' value")
+ }
+ val valParameters = value.get("parameters").asInstanceOf[util.HashMap[String, Any]]
+ val parameters = valParameters.asScala.toMap.map { x => (x._1, x._2.toString) }
+ TAAnalyzeTask(parameters)
+ }
+}
+
+object TextAnalyze extends ComplexParamsReadable[TextAnalyze]
+
+class TextAnalyze(override val uid: String) extends TextAnalyticsBase(uid)
+ with HasCognitiveServiceInput with HasInternalJsonOutputParser with HasSetLocation
+ with HasSetLinkedService with BasicAsyncReply {
+
+ import TAJSONFormat._
+
+ def this() = this(Identifiable.randomUID("TextAnalyze"))
+
+ val entityRecognitionTasks = new TextAnalyzeTaskParam(
+ this,
+ "entityRecognitionTasks",
+ "the entity recognition tasks to perform on submitted documents"
+ )
+
+ def getEntityRecognitionTasks: Seq[TAAnalyzeTask] = $(entityRecognitionTasks)
+
+ def setEntityRecognitionTasks(v: Seq[TAAnalyzeTask]): this.type = set(entityRecognitionTasks, v)
+
+ setDefault(entityRecognitionTasks -> Seq[TAAnalyzeTask]())
+
+ val entityRecognitionPiiTasks = new TextAnalyzeTaskParam(
+ this,
+ "entityRecognitionPiiTasks",
+ "the entity recognition pii tasks to perform on submitted documents"
+ )
+
+ def getEntityRecognitionPiiTasks: Seq[TAAnalyzeTask] = $(entityRecognitionPiiTasks)
+
+ def setEntityRecognitionPiiTasks(v: Seq[TAAnalyzeTask]): this.type = set(entityRecognitionPiiTasks, v)
+
+ setDefault(entityRecognitionPiiTasks -> Seq[TAAnalyzeTask]())
+
+ val entityLinkingTasks = new TextAnalyzeTaskParam(
+ this,
+ "entityLinkingTasks",
+ "the entity linking tasks to perform on submitted documents"
+ )
+
+ def getEntityLinkingTasks: Seq[TAAnalyzeTask] = $(entityLinkingTasks)
+
+ def setEntityLinkingTasks(v: Seq[TAAnalyzeTask]): this.type = set(entityLinkingTasks, v)
+
+ setDefault(entityLinkingTasks -> Seq[TAAnalyzeTask]())
+
+ val keyPhraseExtractionTasks = new TextAnalyzeTaskParam(
+ this,
+ "keyPhraseExtractionTasks",
+ "the key phrase extraction tasks to perform on submitted documents"
+ )
+
+ def getKeyPhraseExtractionTasks: Seq[TAAnalyzeTask] = $(keyPhraseExtractionTasks)
+
+ def setKeyPhraseExtractionTasks(v: Seq[TAAnalyzeTask]): this.type = set(keyPhraseExtractionTasks, v)
+
+ setDefault(keyPhraseExtractionTasks -> Seq[TAAnalyzeTask]())
+
+ val sentimentAnalysisTasks = new TextAnalyzeTaskParam(
+ this,
+ "sentimentAnalysisTasks",
+ "the sentiment analysis tasks to perform on submitted documents"
+ )
+
+ def getSentimentAnalysisTasks: Seq[TAAnalyzeTask] = $(sentimentAnalysisTasks)
+
+ def setSentimentAnalysisTasks(v: Seq[TAAnalyzeTask]): this.type = set(sentimentAnalysisTasks, v)
+
+ setDefault(sentimentAnalysisTasks -> Seq[TAAnalyzeTask]())
+
+ override protected def responseDataType: StructType = TAAnalyzeResponse.schema
+
+ def urlPath: String = "/text/analytics/v3.1/analyze"
+
+ override protected def prepareEntity: Row => Option[AbstractHttpEntity] = { _ => None }
+
+ override protected def inputFunc(schema: StructType): Row => Option[HttpRequestBase] = {
+ { row: Row =>
+ if (shouldSkip(row)) {
+ None
+ } else if (getValue(row, text).forall(Option(_).isEmpty)) {
+ None
+ } else {
+ import TAJSONFormat._
+ val post = new HttpPost(getUrl)
+ getValueOpt(row, subscriptionKey).foreach(post.setHeader("Ocp-Apim-Subscription-Key", _))
+ post.setHeader("Content-Type", "application/json")
+ val texts = getValue(row, text)
+
+ val languages: Option[Seq[String]] = (getValueOpt(row, language) match {
+ case Some(Seq(lang)) => Some(Seq.fill(texts.size)(lang))
+ case s => s
+ })
+
+ val documents = texts.zipWithIndex.map { case (t, i) =>
+ TADocument(languages.flatMap(ls => Option(ls(i))), i.toString, Option(t).getOrElse(""))
+ }
+ val displayName = "SynapseML"
+ val analysisInput = TAAnalyzeAnalysisInput(documents)
+ val tasks = TAAnalyzeTasks(
+ entityRecognitionTasks = getEntityRecognitionTasks,
+ entityLinkingTasks = getEntityLinkingTasks,
+ entityRecognitionPiiTasks = getEntityRecognitionPiiTasks,
+ keyPhraseExtractionTasks = getKeyPhraseExtractionTasks,
+ sentimentAnalysisTasks = getSentimentAnalysisTasks
+ )
+ val json = TAAnalyzeRequest(displayName, analysisInput, tasks).toJson.compactPrint
+ post.setEntity(new StringEntity(json, "UTF-8"))
+ Some(post)
+ }
+ }
+ }
+
+ // TODO refactor to remove duplicate from TextAnalyticsBase
+ private def getTaskRows(tasksRow: GenericRowWithSchema, taskName: String, documentIndex: Int): Option[Seq[Row]] = {
+ val namedTaskRow = tasksRow
+ .getAs[Seq[GenericRowWithSchema]](taskName)
+ if (namedTaskRow == null) {
+ None
+ } else {
+ val taskResults = namedTaskRow
+ .map(x => x.getAs[GenericRowWithSchema]("results"))
+ val rows = taskResults.map(result => {
+ val documents = result.getAs[Seq[GenericRowWithSchema]]("documents")
+ val errors = result.getAs[Seq[GenericRowWithSchema]]("errors")
+ val doc = documents.find { d => d.getAs[String]("id").toInt == documentIndex }
+ val error = errors.find { e => e.getAs[String]("id").toInt == documentIndex }
+ val resultRow = Row.fromSeq(Seq(doc, error)) // result/errors per task, per document
+ resultRow
+ })
+ Some(rows)
+ }
+ }
+
+ override protected def unpackBatchUDF: UserDefinedFunction = {
+ val innerResponseDataType = TAAnalyzeResults.schema
+
+ UDFUtils.oldUdf({ rowOpt: Row =>
+ Option(rowOpt).map { row =>
+ val tasks = row.getAs[GenericRowWithSchema]("tasks")
+
+ // Determine the total number of documents (successful docs + errors)
+ // We need to handle the fact that entityRecognition might not have been specified
+ // - i.e. find the first task with results
+ val taskNames = Seq(
+ "entityRecognitionTasks",
+ "entityLinkingTasks",
+ "entityRecognitionPiiTasks",
+ "keyPhraseExtractionTasks",
+ "sentimentAnalysisTasks")
+ val taskSet = taskNames.map(name => tasks.getAs[Seq[GenericRowWithSchema]](name))
+ .filter(r => r != null)
+ .head
+ val results = taskSet.map(x => x.getAs[GenericRowWithSchema]("results"))
+ val docCount = results.head.getAs[Seq[GenericRowWithSchema]]("documents").size
+ val errorCount = results.head.getAs[Seq[GenericRowWithSchema]]("errors").size
+
+ val rows: Seq[Row] = (0 until (docCount + errorCount)).map(i => {
+ val entityRecognitionRows = getTaskRows(tasks, "entityRecognitionTasks", i)
+ val entityLinkingRows = getTaskRows(tasks, "entityLinkingTasks", i)
+ val entityRecognitionPiiRows = getTaskRows(tasks, "entityRecognitionPiiTasks", i)
+ val keyPhraseRows = getTaskRows(tasks, "keyPhraseExtractionTasks", i)
+ val sentimentAnalysisRows = getTaskRows(tasks, "sentimentAnalysisTasks", i)
+
+ val taaResult = Seq(
+ entityRecognitionRows,
+ entityLinkingRows,
+ entityRecognitionPiiRows,
+ keyPhraseRows,
+ sentimentAnalysisRows)
+ val resultRow = Row.fromSeq(taaResult)
+ resultRow
+ })
+ rows
+ }
+ }, ArrayType(innerResponseDataType)
+ )
+ }
+
+}
diff --git a/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/TextAnalyticsAnalyzeSchemas.scala b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/TextAnalyticsAnalyzeSchemas.scala
new file mode 100644
index 0000000000..fdd3afd529
--- /dev/null
+++ b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/TextAnalyticsAnalyzeSchemas.scala
@@ -0,0 +1,68 @@
+// Copyright (C) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License. See LICENSE in project root for information.
+
+package com.microsoft.azure.synapse.ml.cognitive
+
+import com.microsoft.azure.synapse.ml.core.schema.SparkBindings
+
+// Text Analytics /analyze endpoint schemas
+
+case class TAAnalyzeAnalysisInput(documents: Seq[TADocument])
+
+object TAAnalyzeAnalysisInput extends SparkBindings[TAAnalyzeAnalysisInput]
+
+case class TAAnalyzeTask(parameters: Map[String, String])
+
+object TAAnalyzeTask extends SparkBindings[TAAnalyzeTask]
+
+case class TAAnalyzeTasks(entityRecognitionTasks: Seq[TAAnalyzeTask],
+ entityLinkingTasks: Seq[TAAnalyzeTask],
+ entityRecognitionPiiTasks: Seq[TAAnalyzeTask],
+ keyPhraseExtractionTasks: Seq[TAAnalyzeTask],
+ sentimentAnalysisTasks: Seq[TAAnalyzeTask])
+
+object TAAnalyzeTasks extends SparkBindings[TAAnalyzeTasks]
+
+case class TAAnalyzeRequest(displayName: String,
+ analysisInput: TAAnalyzeAnalysisInput,
+ tasks: TAAnalyzeTasks)
+
+object TAAnalyzeRequest extends SparkBindings[TAAnalyzeRequest]
+
+
+case class TAAnalyzeResponseTaskResults[T](documents: Seq[T],
+ errors: Seq[TAError],
+ modelVersion: String)
+
+case class TAAnalyzeResponseTask[T](state: String,
+ results: TAAnalyzeResponseTaskResults[T])
+
+case class TAAnalyzeResponseTasks(completed: Int,
+ failed: Int,
+ inProgress: Int,
+ total: Int,
+ entityRecognitionTasks: Option[Seq[TAAnalyzeResponseTask[NERDocV3]]],
+ entityLinkingTasks: Option[Seq[TAAnalyzeResponseTask[DetectEntitiesScoreV3]]],
+ entityRecognitionPiiTasks: Option[Seq[TAAnalyzeResponseTask[PIIDocV3]]],
+ keyPhraseExtractionTasks: Option[Seq[TAAnalyzeResponseTask[KeyPhraseScoreV3]]],
+ sentimentAnalysisTasks: Option[Seq[TAAnalyzeResponseTask[SentimentScoredDocumentV3]]]
+ )
+
+// API call response
+case class TAAnalyzeResponse(status: String,
+ errors: Option[Seq[TAError]],
+ displayName: String,
+ tasks: TAAnalyzeResponseTasks)
+
+object TAAnalyzeResponse extends SparkBindings[TAAnalyzeResponse]
+
+case class TAAnalyzeResultTaskResults[T](result: Option[T],
+ error: Option[TAError])
+
+case class TAAnalyzeResult(entityRecognition: Option[Seq[TAAnalyzeResultTaskResults[NERDocV3]]],
+ entityLinking: Option[Seq[TAAnalyzeResultTaskResults[DetectEntitiesScoreV3]]],
+ entityRecognitionPii: Option[Seq[TAAnalyzeResultTaskResults[PIIDocV3]]],
+ keyPhraseExtraction: Option[Seq[TAAnalyzeResultTaskResults[KeyPhraseScoreV3]]],
+ sentimentAnalysis: Option[Seq[TAAnalyzeResultTaskResults[SentimentScoredDocumentV3]]])
+
+object TAAnalyzeResults extends SparkBindings[TAAnalyzeResult]
diff --git a/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/TextAnalyticsSchemas.scala b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/TextAnalyticsSchemas.scala
index 78bfd6b4ec..253406af62 100644
--- a/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/TextAnalyticsSchemas.scala
+++ b/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/TextAnalyticsSchemas.scala
@@ -5,6 +5,8 @@ package com.microsoft.azure.synapse.ml.cognitive
import com.microsoft.azure.synapse.ml.core.schema.SparkBindings
import spray.json.RootJsonFormat
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.param.ParamValidators
// General Text Analytics Schemas
@@ -36,7 +38,10 @@ object TAJSONFormat {
implicit val DocumentFormat: RootJsonFormat[TADocument] = jsonFormat3(TADocument.apply)
implicit val RequestFormat: RootJsonFormat[TARequest] = jsonFormat1(TARequest.apply)
-
+ implicit val AnalysisInputsFormat: RootJsonFormat[TAAnalyzeAnalysisInput] = jsonFormat1(TAAnalyzeAnalysisInput.apply)
+ implicit val AnalysisTaskFormat: RootJsonFormat[TAAnalyzeTask] = jsonFormat1(TAAnalyzeTask.apply)
+ implicit val AnalysisTasksFormat: RootJsonFormat[TAAnalyzeTasks] = jsonFormat5(TAAnalyzeTasks.apply)
+ implicit val AnalyzeRequestFormat: RootJsonFormat[TAAnalyzeRequest] = jsonFormat3(TAAnalyzeRequest.apply)
}
// SentimentV3 Schemas
@@ -111,6 +116,7 @@ object PIIResponseV3 extends SparkBindings[TAResponse[PIIDocV3]]
case class PIIDocV3(id: String,
entities: Seq[PIIEntityV3],
+ redactedText: String,
warnings: Seq[TAWarning],
statistics: Option[DocumentStatistics])
diff --git a/cognitive/src/test/scala/com/microsoft/azure/synapse/ml/cognitive/split1/MultivariateAnamolyDetectionSuite.scala b/cognitive/src/test/scala/com/microsoft/azure/synapse/ml/cognitive/split1/MultivariateAnamolyDetectionSuite.scala
new file mode 100644
index 0000000000..93f3f64645
--- /dev/null
+++ b/cognitive/src/test/scala/com/microsoft/azure/synapse/ml/cognitive/split1/MultivariateAnamolyDetectionSuite.scala
@@ -0,0 +1,200 @@
+// Copyright (C) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License. See LICENSE in project root for information.
+
+package com.microsoft.azure.synapse.ml.cognitive.split1
+
+import com.microsoft.azure.synapse.ml.cognitive._
+import com.microsoft.azure.synapse.ml.core.env.StreamUtilities.using
+import com.microsoft.azure.synapse.ml.core.test.base.TestBase
+import com.microsoft.azure.synapse.ml.core.test.fuzzing.{EstimatorFuzzing, TestObject}
+import org.apache.commons.io.IOUtils
+import org.apache.http.client.methods.{HttpDelete, HttpEntityEnclosingRequestBase, HttpGet, HttpRequestBase}
+import org.apache.spark.ml.util.MLReadable
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.functions.col
+import org.scalactic.Equality
+import spray.json._
+
+import java.net.URI
+
+case class MADListModelsResponse(models: Seq[MADModel],
+ currentCount: Int,
+ maxCount: Int,
+ nextLink: Option[String])
+
+case class MADModel(modelId: String,
+ createdTime: String,
+ lastUpdatedTime: String,
+ status: String,
+ displayName: Option[String],
+ variablesCount: Int)
+
+object MADListModelsProtocol extends DefaultJsonProtocol {
+ implicit val MADModelEnc: RootJsonFormat[MADModel] = jsonFormat6(MADModel)
+ implicit val MADLMRespEnc: RootJsonFormat[MADListModelsResponse] = jsonFormat4(MADListModelsResponse)
+}
+
+import com.microsoft.azure.synapse.ml.cognitive.split1.MADListModelsProtocol._
+
+object MADUtils extends AnomalyKey {
+
+ import com.microsoft.azure.synapse.ml.cognitive.RESTHelpers._
+
+ def madSend(request: HttpRequestBase, path: String,
+ params: Map[String, String] = Map()): String = {
+
+ val paramString = if (params.isEmpty) {
+ ""
+ } else {
+ "?" + URLEncodingUtils.format(params)
+ }
+ request.setURI(new URI(path + paramString))
+
+ retry(List(100, 500, 1000), { () =>
+ request.addHeader("Ocp-Apim-Subscription-Key", anomalyKey)
+ request.addHeader("Content-Type", "application/json")
+ using(Client.execute(request)) { response =>
+ if (!response.getStatusLine.getStatusCode.toString.startsWith("2")) {
+ val bodyOpt = request match {
+ case er: HttpEntityEnclosingRequestBase => IOUtils.toString(er.getEntity.getContent, "UTF-8")
+ case _ => ""
+ }
+ if (response.getStatusLine.getStatusCode.toString.equals("429")) {
+ val retryTime = response.getHeaders("Retry-After").head.getValue.toInt * 1000
+ Thread.sleep(retryTime.toLong)
+ }
+ throw new RuntimeException(s"Failed: response: $response " + s"requestUrl: ${request.getURI}" +
+ s"requestBody: $bodyOpt")
+ }
+ if (response.getStatusLine.getReasonPhrase == "No Content") {
+ ""
+ }
+ else if (response.getStatusLine.getReasonPhrase == "Created") {
+ response.getHeaders("Location").head.getValue
+ }
+ else {
+ IOUtils.toString(response.getEntity.getContent, "UTF-8")
+ }
+ }.get
+ })
+ }
+
+ def madDelete(path: String, params: Map[String, String] = Map()): String = {
+ madSend(new HttpDelete(), "https://westus2.api.cognitive.microsoft.com/anomalydetector/" +
+ "v1.1-preview/multivariate/models/" + path, params)
+ }
+
+ def madListModels(params: Map[String, String] = Map()): String = {
+ madSend(new HttpGet(), "https://westus2.api.cognitive.microsoft.com/anomalydetector/" +
+ "v1.1-preview/multivariate/models", params)
+ }
+}
+
+trait MADUtils extends TestBase with AnomalyKey {
+
+ import spark.implicits._
+
+ lazy val df: DataFrame = Seq(
+ "https://mmlspark.blob.core.windows.net/datasets/sample_data_5_3000.zip"
+ ).toDF("source")
+
+ lazy val startTime: String = "2021-01-01T00:00:00Z"
+
+ lazy val endTime: String = "2021-01-02T12:00:00Z"
+}
+
+class MultiAnomalyEstimatorSuite extends EstimatorFuzzing[MultivariateAnomalyEstimator] with MADUtils {
+
+ import MADUtils._
+
+ override def assertDFEq(df1: DataFrame, df2: DataFrame)(implicit eq: Equality[DataFrame]): Unit = {
+ def prep(df: DataFrame) = {
+ df.select("source")
+ }
+
+ super.assertDFEq(prep(df1), prep(df2))(eq)
+ }
+
+ def mae: MultivariateAnomalyEstimator = new MultivariateAnomalyEstimator()
+ .setSubscriptionKey(anomalyKey)
+ .setLocation("westus2")
+ .setOutputCol("result")
+ .setSourceCol("source")
+ .setStartTime(startTime)
+ .setEndTime(endTime)
+ .setConcurrency(5)
+
+ test("Basic Usage") {
+ val mam = mae.setSlidingWindow(200).setAlignMode("outer").setFillNAMethod("linear")
+ val model = mam.fit(df)
+ val diagnosticsInfo = mam.getDiagnosticsInfo
+ assert(diagnosticsInfo.variableStates.get.length.equals(5))
+
+ val result = model
+ .setSourceCol("source").setStartTime(startTime).setEndTime(endTime)
+ .setOutputCol("result")
+ .transform(df)
+ .withColumn("status", col("result.summary.status"))
+ .withColumn("variableStates", col("result.summary.variableStates"))
+ .select("status", "variableStates")
+ .collect()
+
+ assert(result.head.getString(0).equals("READY"))
+ assert(result.head.getSeq(1).length.equals(5))
+
+ madDelete(model.getModelId)
+ }
+
+ test("Throw errors if alignMode is not set correctly") {
+ val caught = intercept[IllegalArgumentException] {
+ mae.setSlidingWindow(200).setAlignMode("alignMode").fit(df)
+ }
+ assert(caught.getMessage.contains("parameter alignMode given invalid value"))
+ }
+
+ test("Throw errors if slidingWindow is not between 28 and 2880") {
+ val caught = intercept[IllegalArgumentException] {
+ mae.setSlidingWindow(20).fit(df)
+ }
+ assert(caught.getMessage.contains("parameter slidingWindow given invalid value"))
+ }
+
+ test("Throw errors if required fields not set") {
+ val caught = intercept[AssertionError] {
+ new MultivariateAnomalyEstimator()
+ .setSubscriptionKey(anomalyKey)
+ .setLocation("westus2")
+ .setOutputCol("result")
+ .fit(df)
+ }
+ assert(caught.getMessage.contains("Missing required params"))
+ assert(caught.getMessage.contains("slidingWindow"))
+ assert(caught.getMessage.contains("source"))
+ assert(caught.getMessage.contains("startTime"))
+ assert(caught.getMessage.contains("endTime"))
+ }
+
+ override def testSerialization(): Unit = {
+ println("ignore the Serialization Fuzzing test because fitting process takes more than 3 minutes")
+ }
+
+ override def testExperiments(): Unit = {
+ println("ignore the Experiment Fuzzing test because fitting process takes more than 3 minutes")
+ }
+
+ override def afterAll(): Unit = {
+ val models = madListModels().parseJson.convertTo[MADListModelsResponse].models.map(_.modelId)
+ for (modelId <- models) {
+ madDelete(modelId)
+ }
+ super.afterAll()
+ }
+
+ override def testObjects(): Seq[TestObject[MultivariateAnomalyEstimator]] =
+ Seq(new TestObject(mae.setSlidingWindow(200), df))
+
+ override def reader: MLReadable[_] = MultivariateAnomalyEstimator
+
+ override def modelReader: MLReadable[_] = DetectMultivariateAnomaly
+}
+
diff --git a/cognitive/src/test/scala/com/microsoft/azure/synapse/ml/cognitive/split1/TextAnalyticsSuite.scala b/cognitive/src/test/scala/com/microsoft/azure/synapse/ml/cognitive/split1/TextAnalyticsSuite.scala
index ea9b76ca7c..fa5c600d3f 100644
--- a/cognitive/src/test/scala/com/microsoft/azure/synapse/ml/cognitive/split1/TextAnalyticsSuite.scala
+++ b/cognitive/src/test/scala/com/microsoft/azure/synapse/ml/cognitive/split1/TextAnalyticsSuite.scala
@@ -11,13 +11,14 @@ import com.microsoft.azure.synapse.ml.stages.FixedMiniBatchTransformer
import org.apache.spark.ml.util.MLReadable
import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
import org.apache.spark.sql.functions.col
-import org.apache.spark.sql.{DataFrame, Row}
+import org.apache.spark.sql.{DataFrame, Dataset, Row}
-trait TextKey {
- lazy val textKey = sys.env.getOrElse("TEXT_API_KEY", Secrets.CognitiveApiKey)
+trait TextEndpoint {
+ lazy val textKey: String = sys.env.getOrElse("TEXT_API_KEY", Secrets.CognitiveApiKey)
+ lazy val textApiLocation: String = sys.env.getOrElse("TEXT_API_LOCATION", "eastus")
}
-class LanguageDetectorSuite extends TransformerFuzzing[LanguageDetectorV2] with TextKey {
+class LanguageDetectorSuite extends TransformerFuzzing[LanguageDetectorV2] with TextEndpoint {
import spark.implicits._
@@ -30,7 +31,7 @@ class LanguageDetectorSuite extends TransformerFuzzing[LanguageDetectorV2] with
lazy val detector: LanguageDetectorV2 = new LanguageDetectorV2()
.setSubscriptionKey(textKey)
- .setUrl("https://eastus.api.cognitive.microsoft.com/text/analytics/v2.0/languages")
+ .setUrl(s"https://$textApiLocation.api.cognitive.microsoft.com/text/analytics/v2.0/languages")
.setTextCol("text2")
.setOutputCol("replies")
@@ -41,7 +42,7 @@ class LanguageDetectorSuite extends TransformerFuzzing[LanguageDetectorV2] with
.getItem("name"))
.select("lang")
.collect().toList
- assert(replies(0).getString(0) == "English" && replies(2).getString(0) == "Spanish")
+ assert(replies.head.getString(0) == "English" && replies(2).getString(0) == "Spanish")
}
test("Is serializable ") {
@@ -49,7 +50,7 @@ class LanguageDetectorSuite extends TransformerFuzzing[LanguageDetectorV2] with
.withColumn("lang", col("replies").getItem(0)
.getItem("detectedLanguages").getItem(0)
.getItem("name"))
- .select("text2","lang")
+ .select("text2", "lang")
.sort("text2")
.collect().toList
assert(replies(2).getString(1) == "English" && replies(3).getString(1) == "Spanish")
@@ -70,7 +71,7 @@ class LanguageDetectorSuite extends TransformerFuzzing[LanguageDetectorV2] with
override def reader: MLReadable[_] = LanguageDetectorV2
}
-class LanguageDetectorV3Suite extends TransformerFuzzing[LanguageDetector] with TextKey {
+class LanguageDetectorV3Suite extends TransformerFuzzing[LanguageDetector] with TextEndpoint {
import spark.implicits._
@@ -83,7 +84,7 @@ class LanguageDetectorV3Suite extends TransformerFuzzing[LanguageDetector] with
lazy val detector: LanguageDetector = new LanguageDetector()
.setSubscriptionKey(textKey)
- .setUrl("https://eastus.api.cognitive.microsoft.com/text/analytics/v3.0/languages")
+ .setUrl(s"https://$textApiLocation.api.cognitive.microsoft.com/text/analytics/v3.0/languages")
.setOutputCol("replies")
test("Basic Usage") {
@@ -93,7 +94,7 @@ class LanguageDetectorV3Suite extends TransformerFuzzing[LanguageDetector] with
.getItem("name"))
.select("lang")
.collect().toList
- assert(replies(0).getString(0) == "English" && replies(2).getString(0) == "Spanish")
+ assert(replies.head.getString(0) == "English" && replies(2).getString(0) == "Spanish")
}
override def testObjects(): Seq[TestObject[LanguageDetector]] =
@@ -102,7 +103,7 @@ class LanguageDetectorV3Suite extends TransformerFuzzing[LanguageDetector] with
override def reader: MLReadable[_] = LanguageDetector
}
-class EntityDetectorSuite extends TransformerFuzzing[EntityDetectorV2] with TextKey {
+class EntityDetectorSuite extends TransformerFuzzing[EntityDetectorV2] with TextEndpoint {
import spark.implicits._
@@ -113,7 +114,7 @@ class EntityDetectorSuite extends TransformerFuzzing[EntityDetectorV2] with Text
lazy val detector: EntityDetectorV2 = new EntityDetectorV2()
.setSubscriptionKey(textKey)
- .setUrl("https://eastus.api.cognitive.microsoft.com/text/analytics/v2.0/entities")
+ .setUrl(s"https://$textApiLocation.api.cognitive.microsoft.com/text/analytics/v2.0/entities")
.setLanguage("en")
.setOutputCol("replies")
@@ -132,7 +133,7 @@ class EntityDetectorSuite extends TransformerFuzzing[EntityDetectorV2] with Text
override def reader: MLReadable[_] = EntityDetectorV2
}
-class EntityDetectorSuiteV3 extends TransformerFuzzing[EntityDetector] with TextKey {
+class EntityDetectorSuiteV3 extends TransformerFuzzing[EntityDetector] with TextEndpoint {
import spark.implicits._
@@ -163,7 +164,8 @@ class EntityDetectorSuiteV3 extends TransformerFuzzing[EntityDetector] with Text
override def reader: MLReadable[_] = EntityDetector
}
-trait TextSentimentBaseSuite extends TestBase with TextKey {
+trait TextSentimentBaseSuite extends TestBase with TextEndpoint {
+
import spark.implicits._
lazy val df: DataFrame = Seq(
@@ -190,13 +192,13 @@ class TextSentimentV3Suite extends TransformerFuzzing[TextSentiment] with TextSe
col("replies").alias("scoredDocuments")
).collect().toList
- assert(List(4,5).forall(results(_).get(0) == null))
+ assert(List(4, 5).forall(results(_).get(0) == null))
assert(
results(0).getSeq[Row](0).head.getString(0) == "positive" &&
- results(2).getSeq[Row](0).head.getString(0) == "negative")
+ results(2).getSeq[Row](0).head.getString(0) == "negative")
}
- test("batch usage"){
+ test("batch usage") {
val t = new TextSentiment()
.setSubscriptionKey(textKey)
.setLocation("eastus")
@@ -219,7 +221,7 @@ class TextSentimentSuite extends TransformerFuzzing[TextSentimentV2] with TextSe
lazy val t: TextSentimentV2 = new TextSentimentV2()
.setSubscriptionKey(textKey)
- .setUrl("https://eastus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment")
+ .setUrl(s"https://$textApiLocation.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment")
.setLanguageCol("lang")
.setOutputCol("replies")
@@ -228,11 +230,11 @@ class TextSentimentSuite extends TransformerFuzzing[TextSentimentV2] with TextSe
col("replies").getItem(0).getItem("score"))
.select("score").collect().toList
- assert(List(4,5).forall(results(_).get(0) == null))
- assert(results(0).getFloat(0) > .5 && results(2).getFloat(0) < .5)
+ assert(List(4, 5).forall(results(_).get(0) == null))
+ assert(results.head.getFloat(0) > .5 && results(2).getFloat(0) < .5)
}
- test("batch usage"){
+ test("batch usage") {
val t = new TextSentimentV2()
.setSubscriptionKey(textKey)
.setLocation("eastus")
@@ -251,7 +253,7 @@ class TextSentimentSuite extends TransformerFuzzing[TextSentimentV2] with TextSe
override def reader: MLReadable[_] = TextSentimentV2
}
-class KeyPhraseExtractorSuite extends TransformerFuzzing[KeyPhraseExtractorV2] with TextKey {
+class KeyPhraseExtractorSuite extends TransformerFuzzing[KeyPhraseExtractorV2] with TextEndpoint {
import spark.implicits._
@@ -264,7 +266,7 @@ class KeyPhraseExtractorSuite extends TransformerFuzzing[KeyPhraseExtractorV2] w
lazy val t: KeyPhraseExtractorV2 = new KeyPhraseExtractorV2()
.setSubscriptionKey(textKey)
- .setUrl("https://eastus.api.cognitive.microsoft.com/text/analytics/v2.0/keyPhrases")
+ .setUrl(s"https://$textApiLocation.api.cognitive.microsoft.com/text/analytics/v2.0/keyPhrases")
.setLanguageCol("lang")
.setOutputCol("replies")
@@ -275,7 +277,7 @@ class KeyPhraseExtractorSuite extends TransformerFuzzing[KeyPhraseExtractorV2] w
println(results)
- assert(results(0).getSeq[String](0).toSet === Set("world", "input text"))
+ assert(results.head.getSeq[String](0).toSet === Set("world", "input text"))
assert(results(2).getSeq[String](0).toSet === Set("carretera", "trĆ”fico", "dĆa"))
}
@@ -285,10 +287,11 @@ class KeyPhraseExtractorSuite extends TransformerFuzzing[KeyPhraseExtractorV2] w
override def reader: MLReadable[_] = KeyPhraseExtractorV2
}
-class KeyPhraseExtractorV3Suite extends TransformerFuzzing[KeyPhraseExtractor] with TextKey {
+class KeyPhraseExtractorV3Suite extends TransformerFuzzing[KeyPhraseExtractor] with TextEndpoint {
import spark.implicits._
+ //noinspection ScalaStyle
lazy val df: DataFrame = Seq(
("en", "Hello world. This is some input text that I love."),
("fr", "Bonjour tout le monde"),
@@ -298,7 +301,7 @@ class KeyPhraseExtractorV3Suite extends TransformerFuzzing[KeyPhraseExtractor] w
lazy val t: KeyPhraseExtractor = new KeyPhraseExtractor()
.setSubscriptionKey(textKey)
- .setUrl("https://eastus.api.cognitive.microsoft.com/text/analytics/v3.0/keyPhrases")
+ .setUrl(s"https://$textApiLocation.api.cognitive.microsoft.com/text/analytics/v3.0/keyPhrases")
.setLanguageCol("lang")
.setOutputCol("replies")
@@ -309,7 +312,7 @@ class KeyPhraseExtractorV3Suite extends TransformerFuzzing[KeyPhraseExtractor] w
println(results)
- assert(results(0).getSeq[String](0).toSet === Set("Hello world", "input text"))
+ assert(results.head.getSeq[String](0).toSet === Set("Hello world", "input text"))
assert(results(2).getSeq[String](0).toSet === Set("mucho trĆ”fico", "dĆa", "carretera", "ayer"))
}
@@ -319,17 +322,18 @@ class KeyPhraseExtractorV3Suite extends TransformerFuzzing[KeyPhraseExtractor] w
override def reader: MLReadable[_] = KeyPhraseExtractor
}
-class NERSuite extends TransformerFuzzing[NERV2] with TextKey {
+class NERSuite extends TransformerFuzzing[NERV2] with TextEndpoint {
+
import spark.implicits._
lazy val df: DataFrame = Seq(
- ("1", "en", "Jeff bought three dozen eggs because there was a 50% discount."),
- ("2", "en", "The Great Depression began in 1929. By 1933, the GDP in America fell by 25%.")
- ).toDF("id", "language", "text")
+ ("en", "Jeff bought three dozen eggs because there was a 50% discount."),
+ ("en", "The Great Depression began in 1929. By 1933, the GDP in America fell by 25%.")
+ ).toDF("language", "text")
lazy val n: NERV2 = new NERV2()
.setSubscriptionKey(textKey)
- .setLocation("eastus")
+ .setLocation(textApiLocation)
.setLanguage("en")
.setOutputCol("response")
@@ -358,17 +362,18 @@ class NERSuite extends TransformerFuzzing[NERV2] with TextKey {
override def reader: MLReadable[_] = NERV2
}
-class NERSuiteV3 extends TransformerFuzzing[NER] with TextKey {
+class NERSuiteV3 extends TransformerFuzzing[NER] with TextEndpoint {
+
import spark.implicits._
lazy val df: DataFrame = Seq(
- ("1", "en", "I had a wonderful trip to Seattle last week."),
- ("2", "en", "I visited Space Needle 2 times.")
- ).toDF("id", "language", "text")
+ ("en", "I had a wonderful trip to Seattle last week."),
+ ("en", "I visited Space Needle 2 times.")
+ ).toDF("language", "text")
lazy val n: NER = new NER()
.setSubscriptionKey(textKey)
- .setLocation("eastus")
+ .setLocation(textApiLocation)
.setLanguage("en")
.setOutputCol("response")
@@ -397,24 +402,34 @@ class NERSuiteV3 extends TransformerFuzzing[NER] with TextKey {
override def reader: MLReadable[_] = NER
}
-class PIISuiteV3 extends TransformerFuzzing[PII] with TextKey {
+class PIISuiteV3 extends TransformerFuzzing[PII] with TextEndpoint {
+
import spark.implicits._
lazy val df: DataFrame = Seq(
- ("1", "en", "My SSN is 859-98-0987"),
- ("2", "en",
+ ("en", "My SSN is 859-98-0987"),
+ ("en",
"Your ABA number - 111000025 - is the first 9 digits in the lower left hand corner of your personal check."),
- ("3", "en", "Is 998.214.865-68 your Brazilian CPF number?")
- ).toDF("id", "language", "text")
+ ("en", "Is 998.214.865-68 your Brazilian CPF number?")
+ ).toDF("language", "text")
lazy val n: PII = new PII()
.setSubscriptionKey(textKey)
- .setLocation("eastus")
+ .setLocation(textApiLocation)
.setLanguage("en")
.setOutputCol("response")
test("Basic Usage") {
val results = n.transform(df)
+
+ val redactedTexts = results.withColumn("redactedText",
+ col("response")
+ .getItem(0)
+ .getItem("redactedText"))
+ .select("redactedText")
+ val redactedText = redactedTexts.collect().head(0).toString
+ assert(redactedText === "My SSN is ***********")
+
val matches = results.withColumn("match",
col("response")
.getItem(0)
@@ -437,3 +452,208 @@ class PIISuiteV3 extends TransformerFuzzing[PII] with TextKey {
override def reader: MLReadable[_] = PII
}
+
+class TextAnalyzeSuite extends TransformerFuzzing[TextAnalyze] with TextEndpoint {
+
+ import spark.implicits._
+
+ lazy val dfBasic: DataFrame = Seq(
+ ("en", "I had a wonderful trip to Seattle last week and visited Microsoft."),
+ ("invalid", "This is irrelevant as the language is invalid")
+ ).toDF("language", "text")
+
+ lazy val dfBatched: DataFrame = Seq(
+ (
+ Seq("en", "invalid"),
+ Seq("I had a wonderful trip to Seattle last week and visited Microsoft.",
+ "This is irrelevant as the language is invalid")
+ )
+ ).toDF("language", "text")
+
+ lazy val n: TextAnalyze = new TextAnalyze()
+ .setSubscriptionKey(textKey)
+ .setLocation(textApiLocation)
+ .setLanguageCol("language")
+ .setOutputCol("response")
+ .setErrorCol("error")
+ .setEntityRecognitionTasks(Seq(TAAnalyzeTask(Map("model-version" -> "latest"))))
+ .setEntityLinkingTasks(Seq(TAAnalyzeTask(Map("model-version" -> "latest"))))
+ .setEntityRecognitionPiiTasks(Seq(TAAnalyzeTask(Map("model-version" -> "latest"))))
+ .setKeyPhraseExtractionTasks(Seq(TAAnalyzeTask(Map("model-version" -> "latest"))))
+ .setSentimentAnalysisTasks(Seq(TAAnalyzeTask(Map("model-version" -> "latest"))))
+
+ def getEntityRecognitionResults(results: Dataset[Row], resultIndex: Int): Array[Row] = {
+ results.withColumn("entityRecognition",
+ col("response")
+ .getItem(resultIndex)
+ .getItem("entityRecognition")
+ .getItem(0)
+ .getItem("result")
+ .getItem("entities")
+ .getItem(0)
+ ).select("entityRecognition")
+ .collect()
+ }
+
+ def getEntityRecognitionErrors(results: Dataset[Row], resultIndex: Int): Array[Row] = {
+ results.withColumn("error",
+ col("response")
+ .getItem(resultIndex)
+ .getItem("entityRecognition")
+ .getItem(0)
+ .getItem("error")
+ .getItem("error")
+ ).select("error")
+ .collect()
+ }
+
+ def getEntityRecognitionPiiResults(results: Dataset[Row], resultIndex: Int): Array[Row] = {
+ results.withColumn("entityRecognitionPii",
+ col("response")
+ .getItem(resultIndex)
+ .getItem("entityRecognitionPii")
+ .getItem(0)
+ .getItem("result")
+ .getItem("entities")
+ .getItem(0)
+ ).select("entityRecognitionPii")
+ .collect()
+ }
+
+ def getKeyPhraseResults(results: Dataset[Row], resultIndex: Int): Array[Row] = {
+ results.withColumn("keyPhrase",
+ col("response")
+ .getItem(0)
+ .getItem("keyPhraseExtraction")
+ .getItem(0)
+ .getItem("result")
+ .getItem("keyPhrases")
+ .getItem(0)
+ ).select("keyPhrase")
+ .collect()
+ }
+
+ def getSentimentAnalysisResults(results: Dataset[Row], resultIndex: Int): Array[Row] = {
+ results.withColumn("sentimentAnalysis",
+ col("response")
+ .getItem(0)
+ .getItem("sentimentAnalysis")
+ .getItem(0)
+ .getItem("result")
+ .getItem("sentiment")
+ ).select("sentimentAnalysis")
+ .collect()
+ }
+
+ def getEntityLinkingResults(results: Dataset[Row], resultIndex: Int): Array[Row] = {
+ results.withColumn("entityLinking",
+ col("response")
+ .getItem(0)
+ .getItem("entityLinking")
+ .getItem(0)
+ .getItem("result")
+ .getItem("entities")
+ .getItem(0)
+ ).select("entityLinking")
+ .collect()
+ }
+
+ test("Basic Usage") {
+ val results = n.transform(dfBasic).cache()
+ // Validate first row (successful execution)
+
+ // entity recognition
+ val entityRows = getEntityRecognitionResults(results, resultIndex = 0)
+ val entityRow = entityRows(0)
+ val entityResult = entityRow(0).asInstanceOf[GenericRowWithSchema]
+
+ assert(entityResult.getAs[String]("text") === "trip")
+ assert(entityResult.getAs[Int]("offset") === 18)
+ assert(entityResult.getAs[Int]("length") === 4)
+ assert(entityResult.getAs[Double]("confidenceScore") > 0.66)
+ assert(entityResult.getAs[String]("category") === "Event")
+
+ // entity recognition pii
+ val entityPiiRows = getEntityRecognitionPiiResults(results, resultIndex = 0)
+ val entityPiiRow = entityPiiRows(0)
+ val entityPiiResult = entityPiiRow(0).asInstanceOf[GenericRowWithSchema]
+ assert(entityPiiResult.getAs[String]("text") === "last week")
+ assert(entityPiiResult.getAs[Int]("offset") === 34)
+ assert(entityPiiResult.getAs[Int]("length") === 9)
+ assert(entityPiiResult.getAs[Double]("confidenceScore") > 0.79)
+ assert(entityPiiResult.getAs[String]("category") === "DateTime")
+
+ // key phrases
+ val keyPhraseRows = getKeyPhraseResults(results, resultIndex = 0)
+ val keyPhraseRow = keyPhraseRows(0).asInstanceOf[GenericRowWithSchema]
+ assert(keyPhraseRow.getAs[String](0) === "wonderful trip")
+
+ // text sentiment
+ val sentimentAnalysisRows = getSentimentAnalysisResults(results, resultIndex = 0)
+ val sentimentAnalysisRow = sentimentAnalysisRows(0).asInstanceOf[GenericRowWithSchema]
+ assert(sentimentAnalysisRow.getAs[String](0) === "positive")
+
+ // entity linking
+ val entityLinkingRows = getEntityLinkingResults(results, resultIndex = 0)
+ val entityLinkingRow = entityLinkingRows(0).asInstanceOf[GenericRowWithSchema]
+ val entityLinkingResult = entityLinkingRow(0).asInstanceOf[GenericRowWithSchema]
+ assert(entityLinkingResult.getAs[String]("name") === "Seattle")
+
+ // Validate second row has error
+ val entityRows2 = getEntityRecognitionErrors(results, resultIndex = 0)
+ val entityRow2 = entityRows2(1).asInstanceOf[GenericRowWithSchema]
+ assert(entityRow2.getAs[String]("error").contains("\"code\":\"UnsupportedLanguageCode\""))
+ }
+
+ test("Batched Usage") {
+ val results = n.transform(dfBatched).cache()
+ // First batch entry
+
+ // entity recognition
+ val entityRows = getEntityRecognitionResults(results, resultIndex = 0)
+ val entityRow = entityRows(0)
+ val entityResult = entityRow(0).asInstanceOf[GenericRowWithSchema]
+
+ assert(entityResult.getAs[String]("text") === "trip")
+ assert(entityResult.getAs[Int]("offset") === 18)
+ assert(entityResult.getAs[Int]("length") === 4)
+ assert(entityResult.getAs[Double]("confidenceScore") > 0.66)
+ assert(entityResult.getAs[String]("category") === "Event")
+
+ // entity recognition pii
+ val entityPiiRows = getEntityRecognitionPiiResults(results, resultIndex = 0)
+ val entityPiiRow = entityPiiRows(0)
+ val entityPiiResult = entityPiiRow(0).asInstanceOf[GenericRowWithSchema]
+ assert(entityPiiResult.getAs[String]("text") === "last week")
+ assert(entityPiiResult.getAs[Int]("offset") === 34)
+ assert(entityPiiResult.getAs[Int]("length") === 9)
+ assert(entityPiiResult.getAs[Double]("confidenceScore") > 0.79)
+ assert(entityPiiResult.getAs[String]("category") === "DateTime")
+
+ // key phrases
+ val keyPhraseRows = getKeyPhraseResults(results, resultIndex = 0)
+ val keyPhraseRow = keyPhraseRows(0).asInstanceOf[GenericRowWithSchema]
+ assert(keyPhraseRow.getAs[String](0) === "wonderful trip")
+
+ // text sentiment
+ val sentimentAnalysisRows = getSentimentAnalysisResults(results, resultIndex = 0)
+ val sentimentAnalysisRow = sentimentAnalysisRows(0).asInstanceOf[GenericRowWithSchema]
+ assert(sentimentAnalysisRow.getAs[String](0) === "positive")
+
+ // entity linking
+ val entityLinkingRows = getEntityLinkingResults(results, resultIndex = 0)
+ val entityLinkingRow = entityLinkingRows(0).asInstanceOf[GenericRowWithSchema]
+ val entityLinkingResult = entityLinkingRow(0).asInstanceOf[GenericRowWithSchema]
+ assert(entityLinkingResult.getAs[String]("name") === "Seattle")
+
+ // Second batch entry
+ val entityRows2 = getEntityRecognitionErrors(results, resultIndex = 1)
+ val entityRow2 = entityRows2(0).asInstanceOf[GenericRowWithSchema]
+ assert(entityRow2.getAs[String]("error").contains("\"code\":\"UnsupportedLanguageCode\""))
+ }
+
+ override def testObjects(): Seq[TestObject[TextAnalyze]] =
+ Seq(new TestObject[TextAnalyze](n, dfBasic, dfBatched))
+
+ override def reader: MLReadable[_] = TextAnalyze
+}
diff --git a/cognitive/src/test/scala/com/microsoft/azure/synapse/ml/cognitive/split1/TranslatorSuite.scala b/cognitive/src/test/scala/com/microsoft/azure/synapse/ml/cognitive/split1/TranslatorSuite.scala
index 9ef46e99a5..af408dd63b 100644
--- a/cognitive/src/test/scala/com/microsoft/azure/synapse/ml/cognitive/split1/TranslatorSuite.scala
+++ b/cognitive/src/test/scala/com/microsoft/azure/synapse/ml/cognitive/split1/TranslatorSuite.scala
@@ -62,7 +62,7 @@ class TranslateSuite extends TransformerFuzzing[Translate]
test("Translate multiple pieces of text with language autodetection") {
val result1 = getTranslationTextResult(translate.setToLanguage(Seq("zh-Hans")), textDf2).collect()
- assert(result1(0).getSeq(0).mkString("\n") == "ä½ å„½ļ¼ä½ å«ä»ä¹ååļ¼\nåč§")
+ assert(result1(0).getSeq(0).mkString("\n") == "ęØ儽ļ¼ęØå«ä»ä¹ååļ¼\nåč§")
val translate1: Translate = new Translate()
.setSubscriptionKey(translatorKey)
@@ -71,7 +71,7 @@ class TranslateSuite extends TransformerFuzzing[Translate]
.setOutputCol("translation")
.setConcurrency(5)
val result3 = getTranslationTextResult(translate1.setToLanguage("zh-Hans"), emptyDf).collect()
- assert(result3(0).getSeq(0).mkString("\n") == "åØļ¼ čæęÆēŖ触ļ¼")
+ assert(result3(0).getSeq(0).mkString("\n") == "åØļ¼čæęÆēŖ触ļ¼")
val translate2: Translate = new Translate()
.setSubscriptionKey(translatorKey)
@@ -81,7 +81,7 @@ class TranslateSuite extends TransformerFuzzing[Translate]
.setOutputCol("translation")
.setConcurrency(5)
val result4 = getTranslationTextResult(translate2, textDf6).collect()
- assert(result4(0).getSeq(0).mkString("") == "åØļ¼ čæęÆēŖ触ļ¼")
+ assert(result4(0).getSeq(0).mkString("") == "åØļ¼čæęÆēŖ触ļ¼")
assert(result4(1).get(0) == null)
assert(result4(2).get(0) == null)
}
@@ -103,13 +103,13 @@ class TranslateSuite extends TransformerFuzzing[Translate]
.withColumn("transliteration", col("translation.transliteration.text"))
.withColumn("translation", col("translation.text"))
.select("translation", "transliteration").collect()
- assert(results.head.getSeq(0).mkString("\n") === "ä½ å„½ļ¼ä½ å«ä»ä¹ååļ¼")
- assert(results.head.getSeq(1).mkString("\n") === "nĒ hĒo ļ¼ nĒ jiĆ o shĆ©n me mĆng zƬ ļ¼")
+ assert(results.head.getSeq(0).mkString("\n") === "ęØ儽ļ¼ęØå«ä»ä¹ååļ¼")
+ assert(results.head.getSeq(1).mkString("\n") === "nĆn hĒo ļ¼ nĆn jiĆ o shĆ©n me mĆng zƬ ļ¼")
}
test("Translate to multiple languages") {
val result1 = getTranslationTextResult(translate.setToLanguage(Seq("zh-Hans", "de")), textDf1).collect()
- assert(result1(0).getSeq(0).mkString("\n") == "ä½ å„½ļ¼ä½ å«ä»ä¹ååļ¼\nHallo, wie heiĆt du?")
+ assert(result1(0).getSeq(0).mkString("\n") == "ęØ儽ļ¼ęØå«ä»ä¹ååļ¼\nHallo, wie heiĆt du?")
}
test("Handle profanity") {
diff --git a/core/src/main/python/mmlspark/__init__.py b/core/src/main/python/mmlspark/__init__.py
new file mode 100644
index 0000000000..4c807c6adc
--- /dev/null
+++ b/core/src/main/python/mmlspark/__init__.py
@@ -0,0 +1,5 @@
+import sys
+import warnings
+warnings.warn("The mmlspark namespace has been deprecated. Please change import statements to import from synapse.ml")
+import synapse.ml
+sys.modules['mmlspark'] = synapse.ml
\ No newline at end of file
diff --git a/core/src/main/scala/com/microsoft/azure/synapse/ml/codegen/CodeGen.scala b/core/src/main/scala/com/microsoft/azure/synapse/ml/codegen/CodeGen.scala
index 7dba32e973..9347a35c72 100644
--- a/core/src/main/scala/com/microsoft/azure/synapse/ml/codegen/CodeGen.scala
+++ b/core/src/main/scala/com/microsoft/azure/synapse/ml/codegen/CodeGen.scala
@@ -5,6 +5,7 @@ package com.microsoft.azure.synapse.ml.codegen
import java.io.File
import CodegenConfigProtocol._
+import com.microsoft.azure.synapse.ml.build.BuildInfo
import com.microsoft.azure.synapse.ml.core.env.FileUtilities._
import org.apache.commons.io.FileUtils
import org.apache.commons.io.FilenameUtils._
@@ -87,13 +88,14 @@ object CodeGen {
|Config/testthat/edition: 3
|""".stripMargin)
+ val scalaVersion = BuildInfo.scalaVersion.split(".".toCharArray).dropRight(1).mkString(".")
writeFile(new File(conf.rSrcDir, "package_register.R"),
s"""|#' @import sparklyr
|spark_dependencies <- function(spark_version, scala_version, ...) {
| spark_dependency(
| jars = c(),
| packages = c(
- | "com.microsoft.azure:${conf.name}:${conf.version}"
+ | "com.microsoft.azure:${conf.name}_${scalaVersion}:${conf.version}"
| ),
| repositories = c("https://mmlspark.azureedge.net/maven")
| )
@@ -134,6 +136,7 @@ object CodeGen {
if (!conf.pySrcDir.exists()) {
conf.pySrcDir.mkdir()
}
+ val extraPackage = if (conf.name.endsWith("core")){" + [\"mmlspark\"]"}else{""}
writeFile(join(conf.pySrcDir, "setup.py"),
s"""
|# Copyright (C) Microsoft Corporation. All rights reserved.
@@ -151,7 +154,7 @@ object CodeGen {
| long_description="SynapseML contains Microsoft's open source "
| + "contributions to the Apache Spark ecosystem",
| license="MIT",
- | packages=find_namespace_packages(include=['synapse.ml.*']),
+ | packages=find_namespace_packages(include=['synapse.ml.*']) ${extraPackage},
| url="https://github.com/Microsoft/SynapseML",
| author="Microsoft",
| author_email="mmlspark-support@microsoft.com",
diff --git a/core/src/main/scala/com/microsoft/azure/synapse/ml/explainers/KernelSHAPBase.scala b/core/src/main/scala/com/microsoft/azure/synapse/ml/explainers/KernelSHAPBase.scala
index 9ae74e7138..74a88a4054 100644
--- a/core/src/main/scala/com/microsoft/azure/synapse/ml/explainers/KernelSHAPBase.scala
+++ b/core/src/main/scala/com/microsoft/azure/synapse/ml/explainers/KernelSHAPBase.scala
@@ -90,7 +90,7 @@ abstract class KernelSHAPBase(override val uid: String)
(id, coefficientsMatrix, metrics.toSpark)
}.toDF(idCol, this.getOutputCol, this.getMetricsCol)
- preprocessed.hint("broadcast").join(fitted, Seq(idCol), "inner").drop(idCol)
+ preprocessed.join(fitted, Seq(idCol), "inner").drop(idCol)
}
override def copy(extra: ParamMap): Transformer = defaultCopy(extra)
diff --git a/core/src/main/scala/com/microsoft/azure/synapse/ml/explainers/LIMEBase.scala b/core/src/main/scala/com/microsoft/azure/synapse/ml/explainers/LIMEBase.scala
index 67afb5c103..39b3ba7d91 100644
--- a/core/src/main/scala/com/microsoft/azure/synapse/ml/explainers/LIMEBase.scala
+++ b/core/src/main/scala/com/microsoft/azure/synapse/ml/explainers/LIMEBase.scala
@@ -111,7 +111,7 @@ abstract class LIMEBase(override val uid: String)
(id, coefficientsMatrix, metrics.toSpark)
}.toDF(idCol, this.getOutputCol, this.getMetricsCol)
- preprocessed.hint("broadcast").join(fitted, Seq(idCol), "inner").drop(idCol)
+ preprocessed.join(fitted, Seq(idCol), "inner").drop(idCol)
}
override def copy(extra: ParamMap): Transformer = this.defaultCopy(extra)
diff --git a/core/src/main/scala/com/microsoft/azure/synapse/ml/io/http/HTTPTransformer.scala b/core/src/main/scala/com/microsoft/azure/synapse/ml/io/http/HTTPTransformer.scala
index c905468fab..cc117fc082 100644
--- a/core/src/main/scala/com/microsoft/azure/synapse/ml/io/http/HTTPTransformer.scala
+++ b/core/src/main/scala/com/microsoft/azure/synapse/ml/io/http/HTTPTransformer.scala
@@ -7,6 +7,7 @@ import com.microsoft.azure.synapse.ml.codegen.Wrappable
import com.microsoft.azure.synapse.ml.core.contracts.{HasInputCol, HasOutputCol}
import com.microsoft.azure.synapse.ml.io.http.HandlingUtils.HandlerFunc
import com.microsoft.azure.synapse.ml.logging.BasicLogging
+import org.apache.http.impl.client.CloseableHttpClient
import org.apache.spark.injections.UDFUtils
import org.apache.spark.ml.param._
import org.apache.spark.ml.util.Identifiable
@@ -28,6 +29,12 @@ trait HasHandler extends Params {
def setHandler(v: HandlerFunc): HasHandler.this.type = {
set(handler, UDFUtils.oldUdf(v, StringType))
}
+
+ setDefault(handler -> HandlingUtils.advancedUDF(100)) //scalastyle:ignore magic.number
+
+ def handlingFunc(client: CloseableHttpClient,
+ request: HTTPRequestData): HTTPResponseData =
+ getHandler(client, request)
}
trait HTTPParams extends Wrappable {
@@ -88,19 +95,19 @@ class HTTPTransformer(val uid: String)
with ComplexParamsWritable with BasicLogging {
logClass()
- setDefault(handler -> HandlingUtils.advancedUDF(100,500,1000)) //scalastyle:ignore magic.number
+ setDefault(handler -> HandlingUtils.advancedUDF(100, 500, 1000)) //scalastyle:ignore magic.number
def this() = this(Identifiable.randomUID("HTTPTransformer"))
val clientHolder = SharedVariable {
getConcurrency match {
- case 1 => new SingleThreadedHTTPClient(getHandler, (getTimeout*1000).toInt)
+ case 1 => new SingleThreadedHTTPClient(getHandler, (getTimeout * 1000).toInt)
case n if n > 1 =>
val dur = get(concurrentTimeout)
- .map(ct => Duration.fromNanos((ct* math.pow(10, 9)).toLong)) //scalastyle:ignore magic.number
+ .map(ct => Duration.fromNanos((ct * math.pow(10, 9)).toLong)) //scalastyle:ignore magic.number
.getOrElse(Duration.Inf)
val ec = ExecutionContext.global
- new AsyncHTTPClient(getHandler,n, dur, (getTimeout*1000).toInt)(ec)
+ new AsyncHTTPClient(getHandler, n, dur, (getTimeout * 1000).toInt)(ec)
}
}
@@ -134,7 +141,7 @@ class HTTPTransformer(val uid: String)
def transformSchema(schema: StructType): StructType = {
assert(schema(getInputCol).dataType == HTTPSchema.Request)
- schema.add(getOutputCol, HTTPSchema.Response, nullable=true)
+ schema.add(getOutputCol, HTTPSchema.Response, nullable = true)
}
}
diff --git a/core/src/main/scala/com/microsoft/azure/synapse/ml/stages/MiniBatchTransformer.scala b/core/src/main/scala/com/microsoft/azure/synapse/ml/stages/MiniBatchTransformer.scala
index b4eec0b6ba..90c6ea454f 100644
--- a/core/src/main/scala/com/microsoft/azure/synapse/ml/stages/MiniBatchTransformer.scala
+++ b/core/src/main/scala/com/microsoft/azure/synapse/ml/stages/MiniBatchTransformer.scala
@@ -189,20 +189,23 @@ class FlattenBatch(val uid: String)
def this() = this(Identifiable.randomUID("FlattenBatch"))
- def transpose(nestedSeq: Seq[Seq[Any]]): Seq[Seq[Any]] = {
+ def transpose(nestedSeq: Seq[Any]): Seq[Seq[Any]] = {
val innerLength = nestedSeq.filter {
case null => false
- case _ => true
- }.head.length
+ case _ : Seq[Any] => true
+ case _ => false
+ }.head.asInstanceOf[Seq[Any]].length
assert(nestedSeq.forall{
case null => true
- case innerSeq => innerSeq.lengthCompare(innerLength) == 0
+ case innerSeq : Seq[Any] => innerSeq.lengthCompare(innerLength) == 0
+ case _ => true
})
(0 until innerLength).map(i => nestedSeq.map{
case null => null
- case innerSeq => innerSeq(i)
+ case innerSeq : Seq[Any] => innerSeq(i)
+ case any => any
})
}
@@ -215,8 +218,17 @@ class FlattenBatch(val uid: String)
it.flatMap { rowOfLists =>
val transposed: Seq[Seq[Any]] = transpose(
(0 until rowOfLists.length)
- .filterNot(rowOfLists.isNullAt)
- .map(rowOfLists.getSeq))
+ .map(i => {
+ if (rowOfLists.isNullAt(i)) {
+ null
+ } else {
+ val fieldSchema = rowOfLists.schema.fields(i)
+ fieldSchema.dataType match {
+ case _ : ArrayType => rowOfLists.getSeq(i)
+ case _ => rowOfLists.get(i)
+ }
+ }
+ }))
transposed.map {
values => new GenericRowWithSchema(values.toArray, outputSchema)
}
@@ -228,11 +240,11 @@ class FlattenBatch(val uid: String)
override def copy(extra: ParamMap): this.type = defaultCopy(extra)
override def transformSchema(schema: StructType): StructType = {
- assert(schema.fields.forall(sf => sf.dataType match {
- case _: ArrayType => true
- case _ => false
+ StructType(schema.map(f => {
+ f.dataType match {
+ case arrayField : ArrayType => StructField(f.name, f.dataType.asInstanceOf[ArrayType].elementType)
+ case nonArrayField => StructField(f.name, f.dataType)
+ }
}))
- StructType(schema.map(f => StructField(f.name, f.dataType.asInstanceOf[ArrayType].elementType)))
}
-
}
diff --git a/core/src/test/scala/com/microsoft/azure/synapse/ml/codegen/TestGen.scala b/core/src/test/scala/com/microsoft/azure/synapse/ml/codegen/TestGen.scala
index 6badac698c..3bb582e25d 100644
--- a/core/src/test/scala/com/microsoft/azure/synapse/ml/codegen/TestGen.scala
+++ b/core/src/test/scala/com/microsoft/azure/synapse/ml/codegen/TestGen.scala
@@ -8,6 +8,7 @@ import com.microsoft.azure.synapse.ml.core.test.fuzzing.PyTestFuzzing
import java.io.File
import CodegenConfigProtocol._
+import com.microsoft.azure.synapse.ml.build.BuildInfo
import com.microsoft.azure.synapse.ml.core.env.FileUtilities._
import com.microsoft.azure.synapse.ml.core.utils.JarLoadingUtils.instantiateServices
import org.apache.commons.io.FileUtils
@@ -30,8 +31,8 @@ object TestGen {
}
private def makeInitFiles(conf: CodegenConfig, packageFolder: String = ""): Unit = {
- val dir = new File(new File(conf.pyTestDir, "synapsemltest"), packageFolder)
- if (!dir.exists()){
+ val dir = new File(new File(conf.pyTestDir, "synapsemltest"), packageFolder)
+ if (!dir.exists()) {
dir.mkdirs()
}
writeFile(new File(dir, "__init__.py"), "")
@@ -46,7 +47,8 @@ object TestGen {
if (!conf.pySrcDir.exists()) {
conf.pySrcDir.mkdir()
}
- writeFile(join(conf.pyTestDir,"synapsemltest", "spark.py"),
+ val scalaVersion = BuildInfo.scalaVersion.split(".".toCharArray).dropRight(1).mkString(".")
+ writeFile(join(conf.pyTestDir, "synapsemltest", "spark.py"),
s"""
|# Copyright (C) Microsoft Corporation. All rights reserved.
|# Licensed under the MIT License. See LICENSE in project root for information.
@@ -59,7 +61,7 @@ object TestGen {
|spark = (SparkSession.builder
| .master("local[*]")
| .appName("PysparkTests")
- | .config("spark.jars.packages", "com.microsoft.azure:synapseml:" + __spark_package_version__)
+ | .config("spark.jars.packages", "com.microsoft.azure:synapseml_$scalaVersion:" + __spark_package_version__)
| .config("spark.jars.repositories", "https://mmlspark.azureedge.net/maven")
| .config("spark.executor.heartbeatInterval", "60s")
| .config("spark.sql.shuffle.partitions", 10)
@@ -79,7 +81,7 @@ object TestGen {
generatePythonTests(conf)
TestBase.stopSparkSession()
generatePyPackageData(conf)
- if (toDir(conf.pyTestOverrideDir).exists()){
+ if (toDir(conf.pyTestOverrideDir).exists()) {
FileUtils.copyDirectoryToDirectory(toDir(conf.pyTestOverrideDir), toDir(conf.pyTestDir))
}
makeInitFiles(conf)
diff --git a/core/src/test/scala/com/microsoft/azure/synapse/ml/core/test/fuzzing/Fuzzing.scala b/core/src/test/scala/com/microsoft/azure/synapse/ml/core/test/fuzzing/Fuzzing.scala
index ec8f1ed5cd..f561c2e207 100644
--- a/core/src/test/scala/com/microsoft/azure/synapse/ml/core/test/fuzzing/Fuzzing.scala
+++ b/core/src/test/scala/com/microsoft/azure/synapse/ml/core/test/fuzzing/Fuzzing.scala
@@ -213,9 +213,9 @@ trait ExperimentFuzzing[S <: PipelineStage] extends TestBase with DataFrameEqual
}
}
- test("Experiment Fuzzing") {
- testExperiments()
- }
+ // test("Experiment Fuzzing") {
+ // testExperiments()
+ // }
}
@@ -291,9 +291,9 @@ trait SerializationFuzzing[S <: PipelineStage with MLWritable] extends TestBase
}
}
- test("Serialization Fuzzing") {
- testSerialization()
- }
+ // test("Serialization Fuzzing") {
+ // testSerialization()
+ // }
}
diff --git a/core/src/test/scala/com/microsoft/azure/synapse/ml/io/split2/DistributedHTTPSuite.scala b/core/src/test/scala/com/microsoft/azure/synapse/ml/io/split2/DistributedHTTPSuite.scala
index a8526841b8..8065137e80 100644
--- a/core/src/test/scala/com/microsoft/azure/synapse/ml/io/split2/DistributedHTTPSuite.scala
+++ b/core/src/test/scala/com/microsoft/azure/synapse/ml/io/split2/DistributedHTTPSuite.scala
@@ -74,7 +74,7 @@ trait HTTPTestUtils extends TestBase with WithFreeUrl with HasHttpClient {
new BasicResponseHandler().handleResponse(res)
}
res.close()
- //println("request suceeded")
+ //println("request succeeded")
(out, (t1 - t0).toDouble / 1e6)
}
diff --git a/core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/DatabricksUtilities.scala b/core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/DatabricksUtilities.scala
index 119b7a581f..51576096aa 100644
--- a/core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/DatabricksUtilities.scala
+++ b/core/src/test/scala/com/microsoft/azure/synapse/ml/nbtest/DatabricksUtilities.scala
@@ -40,9 +40,10 @@ object DatabricksUtilities extends HasHttpClient {
lazy val ClusterName = s"mmlspark-build-${LocalDateTime.now()}"
val Folder = s"/SynapseMLBuild/build_${BuildInfo.version}"
+ val ScalaVersion: String = BuildInfo.scalaVersion.split(".".toCharArray).dropRight(1).mkString(".")
// SynapseML info
- val Version = s"com.microsoft.azure:synapseml:${BuildInfo.version}"
+ val Version = s"com.microsoft.azure:synapseml_$ScalaVersion:${BuildInfo.version}"
val Repository = "https://mmlspark.azureedge.net/maven"
val Libraries: String = List(
@@ -58,9 +59,13 @@ object DatabricksUtilities extends HasHttpClient {
// Execution Params
val TimeoutInMillis: Int = 40 * 60 * 1000
- val NotebookFiles: Array[File] = Option(
- FileUtilities.join(BuildInfo.baseDirectory.getParent, "notebooks").getCanonicalFile.listFiles()
- ).get
+ def recursiveListFiles(f: File): Array[File] = {
+ val files = f.listFiles()
+ files.filter(_.isFile) ++ files.filter(_.isDirectory).flatMap(recursiveListFiles)
+ }
+
+ val NotebookFiles: Array[File] = recursiveListFiles(FileUtilities.join(BuildInfo.baseDirectory.getParent,
+ "notebooks").getCanonicalFile)
val ParallizableNotebooks: Seq[File] = NotebookFiles
@@ -94,7 +99,7 @@ object DatabricksUtilities extends HasHttpClient {
}
//TODO convert all this to typed code
- def databricksPost(path: String, body: String, retries:List[Int]=List(100, 500, 1000)): JsValue = {
+ def databricksPost(path: String, body: String, retries: List[Int] = List(100, 500, 1000)): JsValue = {
retry(retries, { () =>
val request = new HttpPost(BaseURL + path)
request.addHeader("Authorization", AuthValue)
@@ -272,7 +277,7 @@ object DatabricksUtilities extends HasHttpClient {
val error = finalState match {
case Some("SUCCESS") =>
- if (logLevel >= 1) println(s"Notebook $nbName Suceeded")
+ if (logLevel >= 1) println(s"Notebook $nbName Succeeded")
None
case Some(state) =>
Some(new RuntimeException(s"Notebook $nbName failed with state $state. " +
diff --git a/core/src/test/scala/com/microsoft/azure/synapse/ml/stages/MiniBatchTransformerSuite.scala b/core/src/test/scala/com/microsoft/azure/synapse/ml/stages/MiniBatchTransformerSuite.scala
index f9f3d5b6b1..343e6eceeb 100644
--- a/core/src/test/scala/com/microsoft/azure/synapse/ml/stages/MiniBatchTransformerSuite.scala
+++ b/core/src/test/scala/com/microsoft/azure/synapse/ml/stages/MiniBatchTransformerSuite.scala
@@ -9,11 +9,12 @@ import org.apache.spark.injections.UDFUtils
import org.apache.spark.ml.param.DataFrameEquality
import org.apache.spark.ml.util.MLReadable
import org.apache.spark.sql.catalyst.encoders.RowEncoder
-import org.apache.spark.sql.functions.col
+import org.apache.spark.sql.functions.{col, lit}
import org.apache.spark.sql.types.{ArrayType, IntegerType, StringType, StructType}
import org.apache.spark.sql.{DataFrame, Dataset}
import org.scalactic.Equality
import org.scalatest.Assertion
+import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
trait MiniBatchTestUtils extends TestBase with DataFrameEquality {
import spark.implicits._
@@ -130,7 +131,39 @@ class FlattenBatchSuite extends TransformerFuzzing[FlattenBatch] {
val batchedDf = new FixedMiniBatchTransformer().setBatchSize(3).transform(df)
val nullifiedDf = batchedDf.withColumn(
"nullCol", UDFUtils.oldUdf(FlattenBatchUtils.nullify _, ArrayType(IntegerType))(col("in1")))
- assert(new FlattenBatch().transform(nullifiedDf).count() == 1000)
+
+ val result = new FlattenBatch().transform(nullifiedDf)
+ assert(result.count() == 1000)
+
+ val rows = result.collect()
+ val row1 = rows(0).asInstanceOf[GenericRowWithSchema]
+ assert(row1(0) == 1)
+ assert(row1(1) == "foo")
+ assert(row1(2) == 1)
+ val row7 = rows(6).asInstanceOf[GenericRowWithSchema]
+ assert(row7(0) == 7)
+ assert(row7(1) == "foo")
+ assert(row7(2) == null)
+
+ }
+
+ test("propagate non-array") {
+ val batchedDf = new FixedMiniBatchTransformer().setBatchSize(3).transform(df)
+ val fixedValueDf = batchedDf.withColumn("fixed_col", lit("fixed value"))
+
+ val result = new FlattenBatch().transform(fixedValueDf)
+ assert(result.count() == 1000)
+
+ val rows = result.collect()
+ val row1 = rows(0).asInstanceOf[GenericRowWithSchema]
+ assert(row1(0) == 1)
+ assert(row1(1) == "foo")
+ assert(row1(2) == "fixed value")
+ val row2 = rows(1).asInstanceOf[GenericRowWithSchema]
+ assert(row2(0) == 2)
+ assert(row2(1) == "foo")
+ assert(row2(2) == "fixed value")
+
}
override def testObjects(): Seq[TestObject[FlattenBatch]] = Seq(
diff --git a/environment.yaml b/environment.yaml
index 4495bcaaac..c19438bd05 100644
--- a/environment.yaml
+++ b/environment.yaml
@@ -9,11 +9,11 @@ dependencies:
- pip
- r-base
- r-dplyr
- - r-sparklyr
+ - r-sparklyr=1.7.2
- r-devtools
- pip:
- wheel
- - sphinx
+ - sphinx==4.2.0
- sphinx_rtd_theme
- coverage
- pytest
@@ -23,3 +23,4 @@ dependencies:
- pytest-codeblocks
- azure-storage-blob
- twine
+ - jupyter
diff --git a/lightgbm/src/main/python/synapse/ml/lightgbm/mixin.py b/lightgbm/src/main/python/synapse/ml/lightgbm/mixin.py
index c2b58e1a1f..84e9ff8463 100644
--- a/lightgbm/src/main/python/synapse/ml/lightgbm/mixin.py
+++ b/lightgbm/src/main/python/synapse/ml/lightgbm/mixin.py
@@ -64,3 +64,14 @@ def getBoosterNumFeatures(self):
The number of features.
"""
return self._java_obj.getBoosterNumFeatures()
+
+ def setPredictDisableShapeCheck(self, value=None):
+ """
+ Set shape check or not when predict.
+ """
+ if not value:
+ value = False
+ else:
+ value = True
+
+ self._java_obj.setPredictDisableShapeCheck(value)
diff --git a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMBase.scala b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMBase.scala
index f815c3f7e0..0ad6ac64de 100644
--- a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMBase.scala
+++ b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMBase.scala
@@ -247,8 +247,12 @@ trait LightGBMBase[TrainedModel <: Model[TrainedModel]] extends Estimator[Traine
*
* @return ExecutionParams object containing parameters related to LightGBM execution.
*/
- protected def getExecutionParams: ExecutionParams = {
- ExecutionParams(getChunkSize, getMatrixType, getNumThreads, getUseSingleDatasetMode)
+ protected def getExecutionParams(numTasksPerExec: Int): ExecutionParams = {
+ val execNumThreads =
+ if (getUseSingleDatasetMode) get(numThreads).getOrElse(numTasksPerExec - 1)
+ else getNumThreads
+
+ ExecutionParams(getChunkSize, getMatrixType, execNumThreads, getUseSingleDatasetMode)
}
protected def getColumnParams: ColumnParams = {
@@ -267,6 +271,7 @@ trait LightGBMBase[TrainedModel <: Model[TrainedModel]] extends Estimator[Traine
def getDatasetParams(categoricalIndexes: Array[Int], numThreads: Int): String = {
val datasetParams = s"max_bin=$getMaxBin is_pre_partition=True " +
s"bin_construct_sample_cnt=$getBinSampleCount " +
+ s"min_data_in_leaf=$getMinDataInLeaf " +
s"num_threads=$numThreads " +
(if (categoricalIndexes.isEmpty) ""
else s"categorical_feature=${categoricalIndexes.mkString(",")}")
diff --git a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMClassifier.scala b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMClassifier.scala
index 5ede62a535..1bbc397849 100644
--- a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMClassifier.scala
+++ b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMClassifier.scala
@@ -54,7 +54,7 @@ class LightGBMClassifier(override val uid: String)
getIsUnbalance, getVerbosity, categoricalIndexes, actualNumClasses, getBoostFromAverage,
getBoostingType, get(lambdaL1), get(lambdaL2), get(isProvideTrainingMetric),
get(metric), get(minGainToSplit), get(maxDeltaStep), getMaxBinByFeature, get(minDataInLeaf), getSlotNames,
- getDelegate, getDartParams, getExecutionParams, getObjectiveParams)
+ getDelegate, getDartParams, getExecutionParams(numTasksPerExec), getObjectiveParams)
}
def getModel(trainParams: TrainParams, lightGBMBooster: LightGBMBooster): LightGBMClassificationModel = {
@@ -163,11 +163,11 @@ class LightGBMClassificationModel(override val uid: String)
override def numClasses: Int = getActualNumClasses
override def predictRaw(features: Vector): Vector = {
- Vectors.dense(getModel.score(features, true, true))
+ Vectors.dense(getModel.score(features, true, true, getPredictDisableShapeCheck))
}
override def predictProbability(features: Vector): Vector = {
- Vectors.dense(getModel.score(features, false, true))
+ Vectors.dense(getModel.score(features, false, true, getPredictDisableShapeCheck))
}
override def copy(extra: ParamMap): LightGBMClassificationModel = defaultCopy(extra)
diff --git a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMRanker.scala b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMRanker.scala
index 92e897b27e..3a18d26bdf 100644
--- a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMRanker.scala
+++ b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMRanker.scala
@@ -60,7 +60,7 @@ class LightGBMRanker(override val uid: String)
getVerbosity, categoricalIndexes, getBoostingType, get(lambdaL1), get(lambdaL2), getMaxPosition, getLabelGain,
get(isProvideTrainingMetric), get(metric), getEvalAt, get(minGainToSplit), get(maxDeltaStep),
getMaxBinByFeature, get(minDataInLeaf), getSlotNames, getDelegate, getDartParams,
- getExecutionParams, getObjectiveParams)
+ getExecutionParams(numTasksPerExec), getObjectiveParams)
}
def getModel(trainParams: TrainParams, lightGBMBooster: LightGBMBooster): LightGBMRankerModel = {
@@ -144,7 +144,7 @@ class LightGBMRankerModel(override val uid: String)
override def predict(features: Vector): Double = {
logPredict(
- getModel.score(features, false, false)(0)
+ getModel.score(features, false, false, getPredictDisableShapeCheck)(0)
)
}
diff --git a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMRegressor.scala b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMRegressor.scala
index 359cf330f6..6b070ccfe9 100644
--- a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMRegressor.scala
+++ b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/LightGBMRegressor.scala
@@ -70,7 +70,7 @@ class LightGBMRegressor(override val uid: String)
getBoostFromAverage, getBoostingType, get(lambdaL1), get(lambdaL2), get(isProvideTrainingMetric),
get(metric), get(minGainToSplit), get(maxDeltaStep),
getMaxBinByFeature, get(minDataInLeaf), getSlotNames, getDelegate,
- getDartParams, getExecutionParams, getObjectiveParams)
+ getDartParams, getExecutionParams(numTasksPerExec), getObjectiveParams)
}
def getModel(trainParams: TrainParams, lightGBMBooster: LightGBMBooster): LightGBMRegressionModel = {
@@ -127,7 +127,7 @@ class LightGBMRegressionModel(override val uid: String)
override def predict(features: Vector): Double = {
logPredict(
- getModel.score(features, false, false)(0)
+ getModel.score(features, false, false, getPredictDisableShapeCheck)(0)
)
}
diff --git a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/SharedState.scala b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/SharedState.scala
index 45d103730b..9dc200754d 100644
--- a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/SharedState.scala
+++ b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/SharedState.scala
@@ -12,37 +12,25 @@ import org.slf4j.Logger
import java.util.concurrent.CountDownLatch
-class SharedState(columnParams: ColumnParams,
- schema: StructType,
- trainParams: TrainParams) {
- val mainExecutorWorker: Long = LightGBMUtils.getTaskId
- val useSingleDataset: Boolean = trainParams.executionParams.useSingleDatasetMode
+class SharedDatasetState(columnParams: ColumnParams,
+ schema: StructType,
+ trainParams: TrainParams,
+ sharedState: SharedState) {
val chunkSize: Int = trainParams.executionParams.chunkSize
+ val useSingleDataset: Boolean = trainParams.executionParams.useSingleDatasetMode
val matrixType: String = trainParams.executionParams.matrixType
lazy val denseAggregatedColumns: BaseDenseAggregatedColumns = new DenseSyncAggregatedColumns(chunkSize)
lazy val sparseAggregatedColumns: BaseSparseAggregatedColumns = new SparseSyncAggregatedColumns(chunkSize)
- def getArrayType(rowsIter: Iterator[Row], matrixType: String): (Iterator[Row], Boolean) = {
- if (matrixType == "auto") {
- sampleRowsForArrayType(rowsIter, columnParams)
- } else if (matrixType == "sparse") {
- (rowsIter: Iterator[Row], true)
- } else if (matrixType == "dense") {
- (rowsIter: Iterator[Row], false)
- } else {
- throw new Exception(s"Invalid parameter matrix type specified: ${matrixType}")
- }
- }
-
def prep(iter: Iterator[Row]): BaseChunkedColumns = {
val (concatRowsIter: Iterator[Row], isSparseHere: Boolean) = getArrayType(iter, matrixType)
val peekableIter = new PeekingIterator(concatRowsIter)
// Note: the first worker sets "is sparse", other workers read it
- linkIsSparse(isSparseHere)
+ sharedState.linkIsSparse(isSparseHere)
- if (!isSparse.get) {
+ if (!sharedState.isSparse.get) {
new DenseChunkedColumns(peekableIter, columnParams, schema, chunkSize)
} else {
new SparseChunkedColumns(peekableIter, columnParams, schema, chunkSize, useSingleDataset)
@@ -50,7 +38,7 @@ class SharedState(columnParams: ColumnParams,
}
def merge(ts: BaseChunkedColumns): BaseAggregatedColumns = {
- val isSparseVal = isSparse.get
+ val isSparseVal = sharedState.isSparse.get
val aggregatedColumns = if (!isSparseVal) {
if (useSingleDataset) denseAggregatedColumns
else new DenseAggregatedColumns(chunkSize)
@@ -68,6 +56,41 @@ class SharedState(columnParams: ColumnParams,
aggregatedColumns
}
+ @volatile var arrayProcessedSignal: CountDownLatch = new CountDownLatch(0)
+
+ def incrementArrayProcessedSignal(log: Logger): Int = {
+ this.synchronized {
+ val count = arrayProcessedSignal.getCount.toInt + 1
+ arrayProcessedSignal = new CountDownLatch(count)
+ log.info(s"Task incrementing ArrayProcessedSignal to $count")
+ count
+ }
+ }
+
+ def getArrayType(rowsIter: Iterator[Row], matrixType: String): (Iterator[Row], Boolean) = {
+ if (matrixType == "auto") {
+ sampleRowsForArrayType(rowsIter, columnParams)
+ } else if (matrixType == "sparse") {
+ (rowsIter: Iterator[Row], true)
+ } else if (matrixType == "dense") {
+ (rowsIter: Iterator[Row], false)
+ } else {
+ throw new Exception(s"Invalid parameter matrix type specified: ${matrixType}")
+ }
+ }
+}
+
+class SharedState(columnParams: ColumnParams,
+ schema: StructType,
+ trainParams: TrainParams) {
+ val mainExecutorWorker: Long = LightGBMUtils.getTaskId
+ val useSingleDataset: Boolean = trainParams.executionParams.useSingleDatasetMode
+ val chunkSize: Int = trainParams.executionParams.chunkSize
+ val matrixType: String = trainParams.executionParams.matrixType
+
+ val datasetState: SharedDatasetState = new SharedDatasetState(columnParams, schema, trainParams, this)
+ val validationDatasetState: SharedDatasetState = new SharedDatasetState(columnParams, schema, trainParams, this)
+
@volatile var isSparse: Option[Boolean] = None
def linkIsSparse(isSparse: Boolean): Unit = {
@@ -80,15 +103,9 @@ class SharedState(columnParams: ColumnParams,
}
}
- @volatile var arrayProcessedSignal: CountDownLatch = new CountDownLatch(0)
-
def incrementArrayProcessedSignal(log: Logger): Int = {
- this.synchronized {
- val count = arrayProcessedSignal.getCount.toInt + 1
- arrayProcessedSignal = new CountDownLatch(count)
- log.info(s"Task incrementing ArrayProcessedSignal to $count")
- count
- }
+ datasetState.incrementArrayProcessedSignal(log)
+ validationDatasetState.incrementArrayProcessedSignal(log)
}
@volatile var doneSignal: CountDownLatch = new CountDownLatch(0)
diff --git a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/TaskTrainingMethods.scala b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/TaskTrainingMethods.scala
index bd09b4d784..4161586349 100644
--- a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/TaskTrainingMethods.scala
+++ b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/TaskTrainingMethods.scala
@@ -38,13 +38,12 @@ object TaskTrainingMethods {
validationData: Option[Broadcast[Array[Row]]],
sharedState: SharedState): (BaseAggregatedColumns, Option[BaseAggregatedColumns]) = {
val aggregatedColumns = {
- val prepAggregatedColumns = sharedState.prep(inputRows)
- sharedState.merge(prepAggregatedColumns)
+ val prepAggregatedColumns = sharedState.datasetState.prep(inputRows)
+ sharedState.datasetState.merge(prepAggregatedColumns)
}
-
val aggregatedValidationColumns = validationData.map { data =>
- val prepAggregatedColumns = sharedState.prep(data.value.toIterator)
- sharedState.merge(prepAggregatedColumns)
+ val prepAggregatedColumns = sharedState.validationDatasetState.prep(data.value.toIterator)
+ sharedState.validationDatasetState.merge(prepAggregatedColumns)
}
(aggregatedColumns, aggregatedValidationColumns)
}
diff --git a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/booster/LightGBMBooster.scala b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/booster/LightGBMBooster.scala
index c19f7c9020..def6d9350c 100644
--- a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/booster/LightGBMBooster.scala
+++ b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/booster/LightGBMBooster.scala
@@ -387,14 +387,14 @@ class LightGBMBooster(val trainDataset: Option[LightGBMDataset] = None, val para
}
}
- def score(features: Vector, raw: Boolean, classification: Boolean): Array[Double] = {
+ def score(features: Vector, raw: Boolean, classification: Boolean, disableShapeCheck: Boolean): Array[Double] = {
val kind =
if (raw) boosterHandler.rawScoreConstant
else boosterHandler.normalScoreConstant
features match {
- case dense: DenseVector => predictForMat(dense.toArray, kind,
+ case dense: DenseVector => predictForMat(dense.toArray, kind, disableShapeCheck,
boosterHandler.scoredDataLengthLongPtr.get().ptr, boosterHandler.scoredDataOutPtr.get().ptr)
- case sparse: SparseVector => predictForCSR(sparse, kind,
+ case sparse: SparseVector => predictForCSR(sparse, kind, disableShapeCheck,
boosterHandler.scoredDataLengthLongPtr.get().ptr, boosterHandler.scoredDataOutPtr.get().ptr)
}
predScoreToArray(classification, boosterHandler.scoredDataOutPtr.get().ptr, kind)
@@ -403,9 +403,9 @@ class LightGBMBooster(val trainDataset: Option[LightGBMDataset] = None, val para
def predictLeaf(features: Vector): Array[Double] = {
val kind = boosterHandler.leafIndexPredictConstant
features match {
- case dense: DenseVector => predictForMat(dense.toArray, kind,
+ case dense: DenseVector => predictForMat(dense.toArray, kind, false,
boosterHandler.leafIndexDataLengthLongPtr.get().ptr, boosterHandler.leafIndexDataOutPtr.get().ptr)
- case sparse: SparseVector => predictForCSR(sparse, kind,
+ case sparse: SparseVector => predictForCSR(sparse, kind, false,
boosterHandler.leafIndexDataLengthLongPtr.get().ptr, boosterHandler.leafIndexDataOutPtr.get().ptr)
}
predLeafToArray(boosterHandler.leafIndexDataOutPtr.get().ptr)
@@ -414,9 +414,9 @@ class LightGBMBooster(val trainDataset: Option[LightGBMDataset] = None, val para
def featuresShap(features: Vector): Array[Double] = {
val kind = boosterHandler.contribPredictConstant
features match {
- case dense: DenseVector => predictForMat(dense.toArray, kind,
+ case dense: DenseVector => predictForMat(dense.toArray, kind, false,
boosterHandler.shapDataLengthLongPtr.get().ptr, boosterHandler.shapDataOutPtr.get().ptr)
- case sparse: SparseVector => predictForCSR(sparse, kind,
+ case sparse: SparseVector => predictForCSR(sparse, kind, false,
boosterHandler.shapDataLengthLongPtr.get().ptr, boosterHandler.shapDataOutPtr.get().ptr)
}
shapToArray(boosterHandler.shapDataOutPtr.get().ptr)
@@ -508,11 +508,12 @@ class LightGBMBooster(val trainDataset: Option[LightGBMDataset] = None, val para
lazy val numTotalIterations: Int = numTotalModel / numModelPerIteration
protected def predictForCSR(sparseVector: SparseVector, kind: Int,
+ disableShapeCheck: Boolean,
dataLengthLongPtr: SWIGTYPE_p_long_long,
dataOutPtr: SWIGTYPE_p_double): Unit = {
val numCols = sparseVector.size
- val datasetParams = "max_bin=255"
+ val datasetParams = s"max_bin=255 predict_disable_shape_check=${disableShapeCheck.toString}"
val dataInt32bitType = boosterHandler.dataInt32bitType
val data64bitType = boosterHandler.data64bitType
@@ -526,6 +527,7 @@ class LightGBMBooster(val trainDataset: Option[LightGBMDataset] = None, val para
}
protected def predictForMat(row: Array[Double], kind: Int,
+ disableShapeCheck: Boolean,
dataLengthLongPtr: SWIGTYPE_p_long_long,
dataOutPtr: SWIGTYPE_p_double): Unit = {
val data64bitType = boosterHandler.data64bitType
@@ -533,7 +535,7 @@ class LightGBMBooster(val trainDataset: Option[LightGBMDataset] = None, val para
val numCols = row.length
val isRowMajor = 1
- val datasetParams = "max_bin=255"
+ val datasetParams = s"max_bin=255 predict_disable_shape_check=${disableShapeCheck.toString}"
LightGBMUtils.validate(
lightgbmlib.LGBM_BoosterPredictForMatSingle(
diff --git a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/dataset/DatasetAggregator.scala b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/dataset/DatasetAggregator.scala
index 33ba5cda66..b5abca6a8f 100644
--- a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/dataset/DatasetAggregator.scala
+++ b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/dataset/DatasetAggregator.scala
@@ -7,6 +7,7 @@ import com.microsoft.azure.synapse.ml.lightgbm.dataset.DatasetUtils.getRowAsDoub
import com.microsoft.azure.synapse.ml.lightgbm.swig._
import com.microsoft.azure.synapse.ml.lightgbm.{ColumnParams, LightGBMUtils}
import com.microsoft.ml.lightgbm.{SWIGTYPE_p_int, lightgbmlib, lightgbmlibConstants}
+import org.apache.spark.internal.Logging
import org.apache.spark.ml.linalg.SQLDataTypes.VectorType
import org.apache.spark.ml.linalg.{DenseVector, SparseVector}
import org.apache.spark.sql.Row
@@ -181,7 +182,7 @@ private[lightgbm] final class DenseChunkedColumns(rowsIter: PeekingIterator[Row]
}
-private[lightgbm] abstract class BaseAggregatedColumns(val chunkSize: Int) {
+private[lightgbm] abstract class BaseAggregatedColumns(val chunkSize: Int) extends Logging {
protected var labels: FloatSwigArray = _
protected var weights: Option[FloatSwigArray] = None
protected var initScores: Option[DoubleSwigArray] = None
@@ -327,14 +328,17 @@ private[lightgbm] abstract class BaseDenseAggregatedColumns(chunkSize: Int) exte
def getFeatures: DoubleSwigArray = features
- def generateDataset(referenceDataset: Option[LightGBMDataset], datasetParams: String): LightGBMDataset = {
+ def generateDataset(referenceDataset: Option[LightGBMDataset],
+ datasetParams: String): LightGBMDataset = {
val pointer = lightgbmlib.voidpp_handle()
try {
+ val numRows = rowCount.get().toInt
+ logInfo(s"LightGBM task generating dense dataset with $numRows rows and $numCols columns")
// Generate the dataset for features
LightGBMUtils.validate(lightgbmlib.LGBM_DatasetCreateFromMat(
lightgbmlib.double_to_voidp_ptr(features.array),
lightgbmlibConstants.C_API_DTYPE_FLOAT64,
- rowCount.get().toInt,
+ numRows,
numCols,
1,
datasetParams,
@@ -434,9 +438,12 @@ private[lightgbm] abstract class BaseSparseAggregatedColumns(chunkSize: Int)
}
}
- def generateDataset(referenceDataset: Option[LightGBMDataset], datasetParams: String): LightGBMDataset = {
+ def generateDataset(referenceDataset: Option[LightGBMDataset],
+ datasetParams: String): LightGBMDataset = {
indexPointerArrayIncrement(getIndexPointers.array)
val pointer = lightgbmlib.voidpp_handle()
+ val numRows = indptrCount.get() - 1
+ logInfo(s"LightGBM task generating sparse dataset with $numRows rows and $numCols columns")
// Generate the dataset for features
LightGBMUtils.validate(lightgbmlib.LGBM_DatasetCreateFromCSR(
lightgbmlib.int_to_voidp_ptr(indexPointers.array),
diff --git a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/params/LightGBMParams.scala b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/params/LightGBMParams.scala
index b7c27f8821..dc092aa0c9 100644
--- a/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/params/LightGBMParams.scala
+++ b/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm/params/LightGBMParams.scala
@@ -61,7 +61,7 @@ trait LightGBMExecutionParams extends Wrappable {
val useSingleDatasetMode = new BooleanParam(this, "useSingleDatasetMode",
"Use single dataset execution mode to create a single native dataset per executor (singleton) " +
"to reduce memory and communication overhead. Note this is disabled when running spark in local mode.")
- setDefault(useSingleDatasetMode -> false)
+ setDefault(useSingleDatasetMode -> true)
def getUseSingleDatasetMode: Boolean = $(useSingleDatasetMode)
def setUseSingleDatasetMode(value: Boolean): this.type = set(useSingleDatasetMode, value)
@@ -257,6 +257,14 @@ trait LightGBMPredictionParams extends Wrappable {
def getFeaturesShapCol: String = $(featuresShapCol)
def setFeaturesShapCol(value: String): this.type = set(featuresShapCol, value)
+
+ val predictDisableShapeCheck = new BooleanParam(this, "predictDisableShapeCheck",
+ "control whether or not LightGBM raises an error " +
+ "when you try to predict on data with a different number of features than the training data")
+ setDefault(predictDisableShapeCheck -> false)
+
+ def getPredictDisableShapeCheck: Boolean = $(predictDisableShapeCheck)
+ def setPredictDisableShapeCheck(value: Boolean): this.type = set(predictDisableShapeCheck, value)
}
/** Defines parameters for LightGBM models
diff --git a/lightgbm/src/test/scala/com/microsoft/azure/synapse/ml/lightgbm/split1/VerifyLightGBMClassifier.scala b/lightgbm/src/test/scala/com/microsoft/azure/synapse/ml/lightgbm/split1/VerifyLightGBMClassifier.scala
index 8c52c73951..338bb424f0 100644
--- a/lightgbm/src/test/scala/com/microsoft/azure/synapse/ml/lightgbm/split1/VerifyLightGBMClassifier.scala
+++ b/lightgbm/src/test/scala/com/microsoft/azure/synapse/ml/lightgbm/split1/VerifyLightGBMClassifier.scala
@@ -14,8 +14,8 @@ import com.microsoft.azure.synapse.ml.stages.MultiColumnAdapter
import org.apache.commons.io.FileUtils
import org.apache.spark.TaskContext
import org.apache.spark.ml.evaluation.{BinaryClassificationEvaluator, MulticlassClassificationEvaluator}
-import org.apache.spark.ml.feature.{StringIndexer, VectorAssembler}
-import org.apache.spark.ml.linalg.{DenseVector, Vector}
+import org.apache.spark.ml.feature.{LabeledPoint, StringIndexer, VectorAssembler}
+import org.apache.spark.ml.linalg.{DenseVector, Vector, Vectors}
import org.apache.spark.ml.tuning.{ParamGridBuilder, TrainValidationSplit}
import org.apache.spark.ml.util.MLReadable
import org.apache.spark.ml.{Estimator, Model}
@@ -329,11 +329,14 @@ class VerifyLightGBMClassifier extends Benchmarks with EstimatorFuzzing[LightGBM
}
}
val scoredDF1 = baseModel
+ .setUseSingleDatasetMode(false)
.fit(pimaDF)
.transform(pimaDF)
+
// Note: run for more iterations than non-custom objective to prevent flakiness
// Note we intentionally overfit here on the training data and don't do a split
val scoredDF2 = baseModel
+ .setUseSingleDatasetMode(false)
.setFObj(new LogLikelihood())
.setNumIterations(300)
.fit(pimaDF)
@@ -641,6 +644,31 @@ class VerifyLightGBMClassifier extends Benchmarks with EstimatorFuzzing[LightGBM
assertFitWithoutErrors(baseModel, df)
}
+ test("Verify LightGBM Classifier won't get stuck on " +
+ "number of features in data is not the same as it was in training data") {
+ val inputData = Seq(
+ LabeledPoint(1.0, Vectors.dense(1.0, 0.0, 3.0)),
+ LabeledPoint(0.0, Vectors.sparse(3, Array(0, 2), Array(1.0, 3.0)))
+ )
+ val testData = Seq(
+ (
+ "uuid-value",
+ Vectors.sparse(15, Array(2, 6, 8, 14), Array(1.0, 1.0, 1.0, 2.0))
+ )
+ )
+
+ val modelDF = spark.createDataFrame(inputData).toDF("labels", "features")
+ val testDF = spark.createDataFrame(testData).toDF("uuid", "features")
+
+ val fitModel = baseModel.fit(modelDF)
+ val oldModelString = fitModel.getModel.modelStr.get
+
+ val testModel = LightGBMClassificationModel.loadNativeModelFromString(oldModelString)
+ testModel.setPredictDisableShapeCheck(true)
+
+ assert(testModel.transform(testDF).collect().length > 0)
+ }
+
def verifyLearnerOnBinaryCsvFile(fileName: String,
labelColumnName: String,
decimals: Int): Unit = {
diff --git a/lightgbm/src/test/scala/com/microsoft/azure/synapse/ml/lightgbm/split2/VerifyLightGBMRegressor.scala b/lightgbm/src/test/scala/com/microsoft/azure/synapse/ml/lightgbm/split2/VerifyLightGBMRegressor.scala
index cbca56fc8a..5317701127 100644
--- a/lightgbm/src/test/scala/com/microsoft/azure/synapse/ml/lightgbm/split2/VerifyLightGBMRegressor.scala
+++ b/lightgbm/src/test/scala/com/microsoft/azure/synapse/ml/lightgbm/split2/VerifyLightGBMRegressor.scala
@@ -27,7 +27,7 @@ class VerifyLightGBMRegressor extends Benchmarks
verifyLearnerOnRegressionCsvFile("energyefficiency2012_data.train.csv", "Y1", 0,
Some(Seq("X1", "X2", "X3", "X4", "X5", "X6", "X7", "X8", "Y2")))
verifyLearnerOnRegressionCsvFile("airfoil_self_noise.train.csv", "Scaled sound pressure level", 1)
- verifyLearnerOnRegressionCsvFile("Buzz.TomsHardware.train.csv", "Mean Number of display (ND)", -3)
+ verifyLearnerOnRegressionCsvFile("Buzz.TomsHardware.train.csv", "Mean Number of display (ND)", -4)
verifyLearnerOnRegressionCsvFile("machine.train.csv", "ERP", -2)
// TODO: Spark doesn't seem to like the column names here because of '.', figure out how to read in the data
// verifyLearnerOnRegressionCsvFile("slump_test.train.csv", "Compressive Strength (28-day)(Mpa)", 2)
diff --git a/notebooks/HttpOnSpark - Working with Arbitrary Web APIs.ipynb b/notebooks/HttpOnSpark - Working with Arbitrary Web APIs.ipynb
deleted file mode 100644
index c3825ee9e2..0000000000
--- a/notebooks/HttpOnSpark - Working with Arbitrary Web APIs.ipynb
+++ /dev/null
@@ -1,81 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "collapsed": true
- },
- "source": [
- "### Use \"dogs as a service\" in a distributed fashion with HTTP on Spark\n",
- "\n",
- "In this example we will use the simple HTTP Transformer to call a public webAPI that returns random images of dogs. The service does not use the json payload, but this is for example purposes. \n",
- "\n",
- "A call to the dog service returns json objects structured like:\n",
- "\n",
- "`{\"status\":\"success\",\"message\":\"https:\\/\\/images.dog.ceo\\/breeds\\/lhasa\\/n02098413_2536.jpg\"}`\n",
- "\n",
- "If you visit the link you can download the image:\n",
- "\n",
- "![](\"https://images.dog.ceo//breeds//lhasa//n02098413_2536.jpg\"\n",)
\n",
- "\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "\n",
- "if os.environ.get(\"AZURE_SERVICE\", None) == \"Microsoft.ProjectArcadia\":\n",
- " from pyspark.sql import SparkSession\n",
- " spark = SparkSession.builder.getOrCreate()\n",
- "\n",
- "from pyspark.sql.functions import struct\n",
- "from pyspark.sql.types import *\n",
- "from synapse.ml.io.http import *\n",
- "\n",
- "df = spark.createDataFrame([(\"foo\",) for x in range(20)], [\"data\"]) \\\n",
- " .withColumn(\"inputs\", struct(\"data\"))\n",
- "\n",
- "response_schema = StructType().add(\"status\", StringType()).add(\"message\", StringType())\n",
- "\n",
- "client = SimpleHTTPTransformer() \\\n",
- " .setInputCol(\"inputs\") \\\n",
- " .setInputParser(JSONInputParser()) \\\n",
- " .setOutputParser(JSONOutputParser().setDataType(response_schema)) \\\n",
- " .setOutputCol(\"results\") \\\n",
- " .setUrl(\"https://dog.ceo/api/breeds/image/random\")\n",
- "\n",
- "responses = client.transform(df)\n",
- "responses.select(\"results\").show(truncate = False)"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.6.3"
- },
- "name": "HttpOnSpark - Parallelizing a Custom Web Service",
- "notebookId": 3187910992870443
- },
- "nbformat": 4,
- "nbformat_minor": 1
-}
\ No newline at end of file
diff --git a/notebooks/Classification - Adult Census with Vowpal Wabbit.ipynb b/notebooks/features/classification/Classification - Adult Census with Vowpal Wabbit.ipynb
similarity index 100%
rename from notebooks/Classification - Adult Census with Vowpal Wabbit.ipynb
rename to notebooks/features/classification/Classification - Adult Census with Vowpal Wabbit.ipynb
diff --git a/notebooks/Classification - Adult Census.ipynb b/notebooks/features/classification/Classification - Adult Census.ipynb
similarity index 96%
rename from notebooks/Classification - Adult Census.ipynb
rename to notebooks/features/classification/Classification - Adult Census.ipynb
index af9d521971..e1216289f3 100644
--- a/notebooks/Classification - Adult Census.ipynb
+++ b/notebooks/features/classification/Classification - Adult Census.ipynb
@@ -68,8 +68,8 @@
"cell_type": "code",
"execution_count": null,
"source": [
- "from synapse.ml.train import TrainClassifier\r\n",
- "from pyspark.ml.classification import LogisticRegression\r\n",
+ "from synapse.ml.train import TrainClassifier\n",
+ "from pyspark.ml.classification import LogisticRegression\n",
"model = TrainClassifier(model=LogisticRegression(), labelCol=\"income\", numFeatures=256).fit(train)"
],
"outputs": [],
diff --git a/notebooks/Classification - Before and After SynapseML.ipynb b/notebooks/features/classification/Classification - Before and After SynapseML.ipynb
similarity index 100%
rename from notebooks/Classification - Before and After SynapseML.ipynb
rename to notebooks/features/classification/Classification - Before and After SynapseML.ipynb
diff --git a/notebooks/Classification - Twitter Sentiment with Vowpal Wabbit.ipynb b/notebooks/features/classification/Classification - Twitter Sentiment with Vowpal Wabbit.ipynb
similarity index 100%
rename from notebooks/Classification - Twitter Sentiment with Vowpal Wabbit.ipynb
rename to notebooks/features/classification/Classification - Twitter Sentiment with Vowpal Wabbit.ipynb
diff --git a/notebooks/features/cognitive_services/CognitiveServices - Analyze Text.ipynb b/notebooks/features/cognitive_services/CognitiveServices - Analyze Text.ipynb
new file mode 100644
index 0000000000..db990c490d
--- /dev/null
+++ b/notebooks/features/cognitive_services/CognitiveServices - Analyze Text.ipynb
@@ -0,0 +1,118 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Cognitive Services - Analyze Text\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "\n",
+ "if os.environ.get(\"AZURE_SERVICE\", None) == \"Microsoft.ProjectArcadia\":\n",
+ " from pyspark.sql import SparkSession\n",
+ " spark = SparkSession.builder.getOrCreate()\n",
+ " from notebookutils.mssparkutils.credentials import getSecret\n",
+ " os.environ['TEXT_API_KEY'] = getSecret(\"mmlspark-keys\", \"mmlspark-cs-key\")\n",
+ "\n",
+ "#put your service keys here\n",
+ "key = os.environ['TEXT_API_KEY']\n",
+ "location = 'eastus'"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "\n",
+ "df = spark.createDataFrame(data=[\n",
+ " [\"en\", \"Hello Seattle\"],\n",
+ " [\"en\", \"There once was a dog who lived in London and thought she was a human\"]\n",
+ " ], \n",
+ " schema=[\"language\",\"text\"])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display(df)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from synapse.ml.cognitive import *\n",
+ "\n",
+ "text_analyze = (TextAnalyze()\n",
+ " .setLocation(location)\n",
+ " .setSubscriptionKey(key)\n",
+ " .setTextCol(\"text\")\n",
+ " .setOutputCol(\"textAnalysis\")\n",
+ " .setErrorCol(\"error\")\n",
+ " .setLanguageCol(\"language\")\n",
+ " # set the tasks to perform\n",
+ " .setEntityRecognitionTasks([{\"parameters\": { \"model-version\": \"latest\"}}])\n",
+ " .setKeyPhraseExtractionTasks([{\"parameters\": { \"model-version\": \"latest\"}}])\n",
+ " # Uncomment these lines to add more tasks\n",
+ " # .setEntityRecognitionPiiTasks([{\"parameters\": { \"model-version\": \"latest\"}}])\n",
+ " # .setEntityLinkingTasks([{\"parameters\": { \"model-version\": \"latest\"}}])\n",
+ " # .setSentimentAnalysisTasks([{\"parameters\": { \"model-version\": \"latest\"}}])\n",
+ " )\n",
+ "\n",
+ "df_results = text_analyze.transform(df)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display(df_results)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from pyspark.sql.functions import col\n",
+ "\n",
+ "# reformat and display for easier viewing\n",
+ "display(\n",
+ " df_results.select(\"language\", \"text\", \"error\", col(\"textAnalysis\").getItem(0)) # we are not batching so only have a single result\n",
+ " .select(\"language\", \"text\", \"error\", \"textAnalysis[0].*\") # explode the Text Analytics tasks into columns\n",
+ ")"
+ ]
+ }
+ ],
+ "metadata": {
+ "description": null,
+ "kernelspec": {
+ "display_name": "Synapse PySpark",
+ "name": "synapse_pyspark"
+ },
+ "language_info": {
+ "name": "python"
+ },
+ "save_output": true
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file
diff --git a/notebooks/CognitiveServices - Celebrity Quote Analysis.ipynb b/notebooks/features/cognitive_services/CognitiveServices - Celebrity Quote Analysis.ipynb
similarity index 100%
rename from notebooks/CognitiveServices - Celebrity Quote Analysis.ipynb
rename to notebooks/features/cognitive_services/CognitiveServices - Celebrity Quote Analysis.ipynb
diff --git a/notebooks/features/cognitive_services/CognitiveServices - Create a Multilingual Search Engine from Forms.ipynb b/notebooks/features/cognitive_services/CognitiveServices - Create a Multilingual Search Engine from Forms.ipynb
new file mode 100644
index 0000000000..f6b897b1f0
--- /dev/null
+++ b/notebooks/features/cognitive_services/CognitiveServices - Create a Multilingual Search Engine from Forms.ipynb
@@ -0,0 +1,301 @@
+{
+ "cells":[
+ {
+ "cell_type":"code",
+ "execution_count":null,
+ "metadata":{
+
+ },
+ "outputs":[
+
+ ],
+ "source":[
+ "import os\n",
+ "key = os.environ['VISION_API_KEY']\n",
+ "search_key = os.environ['AZURE_SEARCH_KEY']\n",
+ "translator_key = os.environ['TRANSLATOR_KEY']\n",
+ "\n",
+ "search_service = \"mmlspark-azure-search\"\n",
+ "search_index = \"form-demo-index\""
+ ]
+ },
+ {
+ "cell_type":"code",
+ "execution_count":null,
+ "metadata":{
+
+ },
+ "outputs":[
+
+ ],
+ "source":[
+ "from pyspark.sql.functions import udf\n",
+ "from pyspark.sql.types import StringType\n",
+ "\n",
+ "def blob_to_url(blob):\n",
+ " [prefix, postfix] = blob.split(\"@\")\n",
+ " container = prefix.split(\"/\")[-1]\n",
+ " split_postfix = postfix.split(\"/\")\n",
+ " account = split_postfix[0]\n",
+ " filepath = \"/\".join(split_postfix[1:])\n",
+ " return \"https://{}/{}/{}\".format(account, container, filepath)\n",
+ "\n",
+ "\n",
+ "df2 = (spark.read.format(\"binaryFile\")\n",
+ " .load(\"wasbs://ignite2021@mmlsparkdemo.blob.core.windows.net/form_subset/*\")\n",
+ " .select(\"path\")\n",
+ " .limit(10)\n",
+ " .select(udf(blob_to_url, StringType())(\"path\").alias(\"url\"))\n",
+ " .cache()\n",
+ " )\n"
+ ]
+ },
+ {
+ "cell_type":"code",
+ "execution_count":null,
+ "metadata":{
+
+ },
+ "outputs":[
+
+ ],
+ "source":[
+ "display(df2)"
+ ]
+ },
+ {
+ "cell_type":"code",
+ "execution_count":null,
+ "metadata":{
+
+ },
+ "outputs":[
+
+ ],
+ "source":[
+ "displayHTML(\"\"\"\n",
+ "