Skip to content

Latest commit

 

History

History
143 lines (123 loc) · 3.85 KB

release-1.0.1.md

File metadata and controls

143 lines (123 loc) · 3.85 KB
Raw

.NET for Apache Spark 1.0.1 Release Notes

This is a maintenance release which contains backported PRs to support Spark 2.4.8.

New Features/Improvements

  • Add spark.dotnet.ignoreSparkPatchVersionCheck conf to ignore patch version in DotnetRunner (#990)

Bug Fixes

  • Fix signer information mismatch issue (#989)

Infrastructure / Documentation / Etc.

  • Update dotnet-interactive deprecated feed (#991, #995)

Breaking Changes

  • None

Known Issues

  • Broadcast variables do not work with dotnet-interactive (#561)
  • UDFs defined using class objects with closures does not work with dotnet-interactive (#619)
  • In dotnet-interactive blocking Spark methods that require external threads to unblock them does not work. ie StreamingQuery.AwaitTermination requires StreamingQuery.Stop to unblock (#736)
  • GroupedMap does not work on Spark 3.0.0 (#654)

Supported Spark Versions

The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with:

Spark Version microsoft-spark JAR
2.3.* microsoft-spark-2-3_2.11-1.0.1.jar
2.4.0 microsoft-spark-2-4_2.11-1.0.1.jar
2.4.1
2.4.3
2.4.4
2.4.5
2.4.6
2.4.7
2.4.8
2.4.2 Not supported
3.0.0 microsoft-spark-3-0_2.12-1.0.1.jar
3.0.1

Supported Delta Versions

The following table outlines the supported Delta versions along with the Microsoft.Spark.Extensions version to use with:

Delta Version Microsoft.Spark.Extensions.Delta
0.1.0 1.0.1
0.2.0
0.3.0
0.4.0
0.5.0
0.6.0
0.6.1
0.7.0

Supported Hyperspace Versions

The following table outlines the supported Hyperspace versions along with the Microsoft.Spark.Extensions version to use with:

Hyperspace Version Microsoft.Spark.Extensions.Hyperspace
0.1.0 1.0.1
0.2.0