1.2.4 (Spark up to 3.3.2, Java 8 in Jupyter notebooks)

What's Changed

• Replace type check by pattern match by @cutiechi in #193
• Set Jupyter version to Java 8 by @Jolanrensen in #200
• Update spark to 3.3.2 by @devongleeson in #198
• Spark yarn in Jupyter notebooks by @Jolanrensen in #201
• Version bumps by @Jolanrensen in #202

New Contributors

• @cutiechi made their first contribution in #193
• @devongleeson made their first contribution in #198

Full Changelog: 1.2.3...v1.2.4

1.2.3 (Java 8+, Spark 3.0.0 up to 3.3.1)

Adds Java 8 support #189 requested by #186, #183, #190.
All modules except jupyter now target Java (1.)8 and should be compatible with Java (1.)8 servers.
Let us know if you have any issues with it!

1.2.2 (Spark 3.0.0 up to 3.3.1)

• Added BigInteger support in #182 thanks to #181
• New Spark versions: 3.2.3, 3.3.1, 3.2.2
• New Scala versions: 2.12.17, 2.13.10
• Updated Kotlin to 1.7.20
• Small bugfix regarding Map encoding

1.2.1 (Spark 3.0.0 up to 3.3.0)

What's Changed

• Reorganized project: Gradle by @Jolanrensen in #168.

• Reorganized project: Now supporting builds for each patch-version of Spark from 3.0.0 to 3.3.0 and Scala 2.12 / 2.13 by @Jolanrensen in #168.
  NOTE: The new naming scheme.

• Added: UDT Support for MLlib etc. by @Jolanrensen in #162. Example.

• Added: Better UD(A)F support by @Jolanrensen in #152. Example

• Added: seqOf() function by @Jolanrensen in #172.

• Added: dfOf() function by @Jolanrensen in #162.

• Added: JavaRDD extension functions by @Jolanrensen in #174. Example / Use case.

• Added: Spark properties now in use-magic in Jupyter in #176.

• Fixed: Reading files from Datalore by @Jolanrensen in #165.

• Fixed: Encoding "isSomething" names in data classes by @Jolanrensen in #171.

• Fixed: Conversion methods for Scala 2.12 / 2.13 by @Jolanrensen in #175.

• Updated wiki.

Full Changelog: https://github.com/Kotlin/kotlin-spark-api/milestone/10?closed=1.

Get it

Maven Central: https://search.maven.org/artifact/org.jetbrains.kotlinx.spark/kotlin-spark-api_3.3.0_2.13/1.2.1/jar.

For other Spark- and Scala versions, follow the naming scheme.

Read more

Blog post: Coming soon!

1.1.0 (Spark 3.2)

What's Changed

• Default master value to first check the system variables. by @Jolanrensen in #127
• More documentation by @Jolanrensen in #128, #130
• Rdd related helpers by @Jolanrensen in #132
• 3.2 Encoders, supporting scala case classes by @Jolanrensen in #134, #147
• Reorganizing API and adding conversions by @Jolanrensen in #136
• Sorting functions for key/value Datasets by @Jolanrensen in #138
• Tuple first by @Jolanrensen in #144
• Streaming support by @Jolanrensen in #133, #157
• Kotlin Jupyter notebook support by @Jolanrensen in #137, #155, #158
• Created and updated wiki

Full Changelog: kotlin-spark-3.2-1.0.3...3.2-1.1.0

Get it

Maven Central: https://search.maven.org/artifact/org.jetbrains.kotlinx.spark/kotlin-spark-api-3.2/1.1.0/jar

Read more

Blog post: https://blog.jetbrains.com/big-data-tools/2022/05/26/kotlin-api-for-apache-spark-streaming-jupyter-and-more/

1.1.0 (Spark 3.1)

What's Changed

• Default master value to first check the system variables. by @Jolanrensen in #127
• More documentation by @Jolanrensen in #128, #130
• Rdd related helpers by @Jolanrensen in #132
• 3.2 Encoders, supporting scala case classes by @Jolanrensen in #134, #147
• Reorganizing API and adding conversions by @Jolanrensen in #136
• Sorting functions for key/value Datasets by @Jolanrensen in #138
• Tuple first by @Jolanrensen in #144
• Streaming support by @Jolanrensen in #133, #157
• Kotlin Jupyter notebook support by @Jolanrensen in #137, #155, #158
• Created and updated wiki

Full Changelog: kotlin-spark-3.2-1.0.3...3.1-1.1.0

Get it

Maven Central: https://search.maven.org/artifact/org.jetbrains.kotlinx.spark/kotlin-spark-api-3.1/1.1.0/jar

Read more

Blog post: https://blog.jetbrains.com/big-data-tools/2022/05/26/kotlin-api-for-apache-spark-streaming-jupyter-and-more/

1.1.0 (Spark 3.0)

What's Changed

• Default master value to first check the system variables. by @Jolanrensen in #127
• More documentation by @Jolanrensen in #128, #130
• Rdd related helpers by @Jolanrensen in #132
• 3.2 Encoders, supporting scala case classes by @Jolanrensen in #134, #147
• Reorganizing API and adding conversions by @Jolanrensen in #136
• Sorting functions for key/value Datasets by @Jolanrensen in #138
• Tuple first by @Jolanrensen in #144
• Streaming support by @Jolanrensen in #133, #157
• Kotlin Jupyter notebook support by @Jolanrensen in #137, #155, #158
• Created and updated wiki

Full Changelog: kotlin-spark-3.2-1.0.3...3.0-1.1.0

Get it

Maven Central: https://search.maven.org/artifact/org.jetbrains.kotlinx.spark/kotlin-spark-api-3.0/1.1.0/jar

Read more

Blog post: https://blog.jetbrains.com/big-data-tools/2022/05/26/kotlin-api-for-apache-spark-streaming-jupyter-and-more/

1.0.3 (Spark 3.2 support)

Bug Fixes

• Adds test for nullable properties in data classes
• Start to fix array of generics support
• Adds support for List and List and relative structures
• Struct type for nullable field
• Perform fixes in README.md (#113)

Build

• Updates build.yml

Documentation

• Fixes name of spark-3 API in README

Features

• UDF support
• Better selectTyped behavior
• Spark 3.2 support

Miscellaneous Tasks

• Updates Kotlin to 1.5.30

1.0.2

Features

• Added support for enums (a888ea4)

Bug

• adds reduceK function to avoid resolution ambiguity for reduce (72fb5ea)
• fixes invalid package name kotlinx → kotlinx (8e7523a)

CI

• Adds maven wrapper (e951424)

1.0.1

Accidentally 1.0.0 release had no content, this release contains content, release to maven central with version 1.0.1

No other changes