From 02da12c3aacf2904a4e3caef85cede273b525dc8 Mon Sep 17 00:00:00 2001 From: elvaliuliuliu <47404285+elvaliuliuliu@users.noreply.github.com> Date: Fri, 24 Jan 2020 17:45:58 -0800 Subject: [PATCH 01/10] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f3d31f4e5..185e71aec 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ 2.3.* - v0.8.0 + v0.9.0 2.4.0 From 40f51150d8996e1aad6e6667507405ab9e172f2b Mon Sep 17 00:00:00 2001 From: elvaliuliuliu <47404285+elvaliuliuliu@users.noreply.github.com> Date: Fri, 24 Jan 2020 17:47:03 -0800 Subject: [PATCH 02/10] Update pom.xml --- benchmark/scala/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/benchmark/scala/pom.xml b/benchmark/scala/pom.xml index c21b78ee5..8d26879a9 100644 --- a/benchmark/scala/pom.xml +++ b/benchmark/scala/pom.xml @@ -3,7 +3,7 @@ 4.0.0 com.microsoft.spark microsoft-spark-benchmark - 0.8.0 + 0.9.0 2019 UTF-8 From 06ff09740225b26751207f5e3cf57f368f0314a6 Mon Sep 17 00:00:00 2001 From: elvaliuliuliu <47404285+elvaliuliuliu@users.noreply.github.com> Date: Fri, 24 Jan 2020 17:54:42 -0800 Subject: [PATCH 03/10] Create release-0.9.md --- docs/release-notes/0.9/release-0.9.md | 45 +++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 docs/release-notes/0.9/release-0.9.md diff --git a/docs/release-notes/0.9/release-0.9.md b/docs/release-notes/0.9/release-0.9.md new file mode 100644 index 000000000..a94011fe8 --- /dev/null +++ b/docs/release-notes/0.9/release-0.9.md @@ -0,0 +1,45 @@ +# .NET for Apache Spark 0.9 Release Notes + +### New Features and Improvements + +* Support UDF that returns RowType ([#376](https://github.com/dotnet/spark/pull/376)) +* Support for Bucketizer ([#378](https://github.com/dotnet/spark/pull/378)) + +### Breaking Changes +* None + +### Supported Spark Versions + +The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Spark Versionmicrosoft-spark JAR
2.3.*microsoft-spark-2.3.x-0.9.0.jar
2.4.0microsoft-spark-2.4.x-0.9.0.jar
2.4.1
2.4.3
2.4.4
2.4.2Not supported
From b638ac65673a7f85a9072364523a951aafd6ac90 Mon Sep 17 00:00:00 2001 From: elvaliuliuliu <47404285+elvaliuliuliu@users.noreply.github.com> Date: Fri, 24 Jan 2020 17:55:31 -0800 Subject: [PATCH 04/10] Update Versions.props --- eng/Versions.props | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/eng/Versions.props b/eng/Versions.props index 408638d6a..9459b5a72 100644 --- a/eng/Versions.props +++ b/eng/Versions.props @@ -1,7 +1,7 @@ - 0.8.0 + 0.9.0 prerelease $(RestoreSources); From 49551ab1b01d776d5149a5d78e97aa4f71be3a97 Mon Sep 17 00:00:00 2001 From: elvaliuliuliu <47404285+elvaliuliuliu@users.noreply.github.com> Date: Fri, 24 Jan 2020 17:56:13 -0800 Subject: [PATCH 05/10] Update pom.xml --- src/scala/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scala/pom.xml b/src/scala/pom.xml index 9795a4e3e..2a8785da1 100644 --- a/src/scala/pom.xml +++ b/src/scala/pom.xml @@ -7,7 +7,7 @@ ${microsoft-spark.version} UTF-8 - 0.8.0 + 0.9.0 From b728c23ed534e14e1511b13513fbc18e92cde082 Mon Sep 17 00:00:00 2001 From: elvaliuliuliu <47404285+elvaliuliuliu@users.noreply.github.com> Date: Wed, 29 Jan 2020 22:41:29 -0800 Subject: [PATCH 06/10] update release note --- docs/release-notes/0.9/release-0.9.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/release-notes/0.9/release-0.9.md b/docs/release-notes/0.9/release-0.9.md index a94011fe8..fce87067a 100644 --- a/docs/release-notes/0.9/release-0.9.md +++ b/docs/release-notes/0.9/release-0.9.md @@ -2,11 +2,12 @@ ### New Features and Improvements -* Support UDF that returns RowType ([#376](https://github.com/dotnet/spark/pull/376)) +* Expose `DataStreamWriter.Foreach` API ([#387](https://github.com/dotnet/spark/pull/387)) +* Support UDF that returns `Row` object ([#376](https://github.com/dotnet/spark/pull/376), [#406](https://github.com/dotnet/spark/pull/406)) * Support for Bucketizer ([#378](https://github.com/dotnet/spark/pull/378)) ### Breaking Changes -* None +* The new `Microsoft.Spark.Worker` needs to be used to enable UDF that returns `Row` object ([#376](https://github.com/dotnet/spark/pull/376)) and `DataStreamWriter.Foreach` ([#387](https://github.com/dotnet/spark/pull/387)). ### Supported Spark Versions From 1c4491d7e20285f4015a17c23c9c6c6bd4366a4f Mon Sep 17 00:00:00 2001 From: elvaliuliuliu <47404285+elvaliuliuliu@users.noreply.github.com> Date: Fri, 7 Feb 2020 19:18:19 -0800 Subject: [PATCH 07/10] Add chained UDF PR --- docs/release-notes/0.9/release-0.9.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-notes/0.9/release-0.9.md b/docs/release-notes/0.9/release-0.9.md index fce87067a..5b99187b1 100644 --- a/docs/release-notes/0.9/release-0.9.md +++ b/docs/release-notes/0.9/release-0.9.md @@ -3,7 +3,7 @@ ### New Features and Improvements * Expose `DataStreamWriter.Foreach` API ([#387](https://github.com/dotnet/spark/pull/387)) -* Support UDF that returns `Row` object ([#376](https://github.com/dotnet/spark/pull/376), [#406](https://github.com/dotnet/spark/pull/406)) +* Support UDF that returns `Row` object ([#376](https://github.com/dotnet/spark/pull/376), [#406](https://github.com/dotnet/spark/pull/406), [#411](https://github.com/dotnet/spark/pull/411)) * Support for Bucketizer ([#378](https://github.com/dotnet/spark/pull/378)) ### Breaking Changes From e38bda0618be35f3037e6d721f594d454b138269 Mon Sep 17 00:00:00 2001 From: elvaliuliuliu <47404285+elvaliuliuliu@users.noreply.github.com> Date: Mon, 10 Feb 2020 17:23:31 -0800 Subject: [PATCH 08/10] Update release-0.9.md --- docs/release-notes/0.9/release-0.9.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/release-notes/0.9/release-0.9.md b/docs/release-notes/0.9/release-0.9.md index 5b99187b1..7e2e7afa1 100644 --- a/docs/release-notes/0.9/release-0.9.md +++ b/docs/release-notes/0.9/release-0.9.md @@ -7,7 +7,8 @@ * Support for Bucketizer ([#378](https://github.com/dotnet/spark/pull/378)) ### Breaking Changes -* The new `Microsoft.Spark.Worker` needs to be used to enable UDF that returns `Row` object ([#376](https://github.com/dotnet/spark/pull/376)) and `DataStreamWriter.Foreach` ([#387](https://github.com/dotnet/spark/pull/387)). + +* The prior versions (<0.9) of `Microsoft.Spark.Worker` **are not compatible** with this release due to the internal changes related to UDFs. ### Supported Spark Versions @@ -27,7 +28,7 @@ The following table outlines the supported Spark versions along with the microso 2.4.0 - microsoft-spark-2.4.x-0.9.0.jar + microsoft-spark-2.4.x-0.9.0.jar 2.4.1 @@ -38,6 +39,9 @@ The following table outlines the supported Spark versions along with the microso 2.4.4 + + 2.4.5 + 2.4.2 Not supported From 872a6fa800a79e057951ab01637f6d136f91c9af Mon Sep 17 00:00:00 2001 From: elvaliuliuliu <47404285+elvaliuliuliu@users.noreply.github.com> Date: Tue, 18 Feb 2020 13:40:13 -0800 Subject: [PATCH 09/10] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 64c8d40ff..bb1c03b01 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ 2.3.* - v0.9.0 + v0.9.0 2.4.0 From ac96bb2c7c53a2235923e22bab88d414893a6444 Mon Sep 17 00:00:00 2001 From: elvaliuliuliu <47404285+elvaliuliuliu@users.noreply.github.com> Date: Tue, 18 Feb 2020 22:10:48 -0800 Subject: [PATCH 10/10] Update release-0.9.md --- docs/release-notes/0.9/release-0.9.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/release-notes/0.9/release-0.9.md b/docs/release-notes/0.9/release-0.9.md index 7e2e7afa1..ae0cf16af 100644 --- a/docs/release-notes/0.9/release-0.9.md +++ b/docs/release-notes/0.9/release-0.9.md @@ -5,6 +5,7 @@ * Expose `DataStreamWriter.Foreach` API ([#387](https://github.com/dotnet/spark/pull/387)) * Support UDF that returns `Row` object ([#376](https://github.com/dotnet/spark/pull/376), [#406](https://github.com/dotnet/spark/pull/406), [#411](https://github.com/dotnet/spark/pull/411)) * Support for Bucketizer ([#378](https://github.com/dotnet/spark/pull/378)) +* Support `DateType` ([#420](https://github.com/dotnet/spark/pull/420)) ### Breaking Changes