1.0.0-rc01

[koperagen]

Kotlin DataFrame 1.0.0-rc01 is released!

What's new?

• All our example projects now use the compiler plugin.
• requireColumn - addition to cast, convertTo family of functions. See requireColumn documentation for detailed example.
• Documentation pages for pivot and groupBy provide code examples for all possible aggregations - 170 in total and all with rich visualization.
• Updated quickstart guide shows a few simple steps: loading data, transforming it, and visualizing it in IntelliJ IDEA project.
• Changes in DataFrame.print to make it more comprehensive: rendering column types and total col x row by default; its rowsLimit and valuesLimit parameters now accept null value to disable truncation.

And many other improvements and fixes! See below for a complete list of changes in this release grouped by category.

We invite you to try this release:

• Use the 1.0.0-rc01 version in your Gradle or Maven projects:
  • See setup instructions for Gradle, Maven and Android projects!
  • We also recommend trying the compiler plugin!
• Use the special 1.0.0-rc01n version in Kotlin Notebook. The n version includes a patch that resolves notebook specific issue with statistical functions.
  • You can specify the version in the descriptor or use %useLatestDescriptors.
    If you want to include kandy, list it after dataframe:

    %useLatestDescriptors
    %use dataframe, kandy
    // or
    %use dataframe(1.0.0-rc01n), kandy(0.8.5, 0.8.5)

  • Use kernel version 0.16.0-736 or newer.

Features

• Added requireColumn operation by @koperagen in Added requireColumn operation #1715
• Added DataFrame.renderToMarkdown(): String function by @koperagen in Added DataFrame.renderToMarkdown(): String function; Improved type rendering in DataFrame.print() and enabled it by default #1760
• Made emptyDataFrame reified so resulting "null object" dataframes are suitable for aggregation and other column access operations like select by @koperagen in Made emptyDataFrame reified so resulting "null object" dataframes are suitable for aggregation and other column access operations like select #1914
• Added countDistinct overloads on GroupBy by @Allex-Nik in countDistinct overloads on GroupBy #1875
• Added Json instance overload to readJson(inputStream, ...) by @Jolanrensen in Added Json instance overload to readJson(inputStream, ...) #1869
• Improved rendering in DataFrame.print() by @koperagen in Added DataFrame.renderToMarkdown(): String function; Improved type rendering in DataFrame.print() and enabled it by default #1760
  rowsLimit , valuesLimit parameters now accept null to display all rows or complete column values.
  New borderStyle parameter to choose what ASCII symbols will be used for border
  Types are now rendered on a separate row, and rendering is enabled by default
• Aligned column name clashing handling across different IO by @AndreiKingsley in Aligned column name clashing handling across different IO #1904
• Supported reading new GEOMETRY, TIME_NS, VARIANT types from DuckDB by @Jolanrensen in Updated dependencies and fixed compatibility issues #1924

Removals and Deprecations

• Removed remaining data modifiers from intermediate operations classes: Corr, SplitWithTransform for cleaner public API by @koperagen in Removed remaining data modifiers from intermediate operations classes: Corr, SplitWithTransform for cleaner public API #1826
• rename into is deprecated in favor of rename to by @Jolanrensen in API change: Rename to #1837
• Deprecated format guessing DataFrame.read() and SupportedDataFrameFormat in favor of specific DataFrame.readCsv/readJson/... functions by @Jolanrensen in Deprecating DataFrame.read() and SupportedDataFrameFormat #1916

Compiler plugin

• Annotated DataColumn.map* for compiler plugin support by @koperagen in Annotated DataColumn.map* for compiler plugin support #1825
• Annotated DataFrame.get(String columns) for compiler plugin support by @koperagen in Annotated DataFrame.get(String columns) for compiler plugin support #1853
• Annotated allExcept CS DSL operations for compiler plugin support by @Jolanrensen in allExcept compiler plugin annotations #1897
• Updated DataFrame.concat signature for better plugin support
• Annotated DataFrameBuilder APIs for plugin support by @koperagen in Added kdocs for DataFrame.concat, updated its signature for better plugin support, annotated DataFrameBuilder APIs for plugin support #1828

Fixes and other changes

• Fixed double opt-in in notebooks by @Jolanrensen in Fixed double opt-in in notebooks #1822
• Set the dataframe-compiler-plugin-core module back to java 8 by @Jolanrensen in set the dataframe-compiler-plugin-core module back to java 8 #1824
• Refactored the expressions-converter plugin for docs samples with easier API by @koperagen in Refactoring the expressions-converter plugin for docs samples with easier API #1816
• Created dfbuild.dependencyUpdates convention plugin by @Jolanrensen in created dfbuild.dependencyUpdates convention plugin #1827
• Post Beta5 release project update by @AndreiKingsley in Post Beta5 release project update #1831
• Infrastructure version bumps by @Jolanrensen in Infrastructure version bumps #1834
• Added column kind and type checks in explode and ungroup by @AndreiKingsley in Added column kind and type checks in explode and ungroup #1847
• relaxing error for unexpected folder in examples/projects to warning by @Jolanrensen in relaxing error for unexpected folder in examples/projects to warning #1849
• Disallowing ValueColumn<AnyFrame> because FrameColumn should be used by @Jolanrensen in Disallowing ValueColumn<AnyFrame> because FrameColumn should be used #1856
• Migrated to Korro 0.2.2 by @AndreiKingsley in Migrated to Korro 0.2.2 #1846
• Fixed formatted frame toHtml methods by @AndreiKingsley in Fixed formatted frame toHtml methods #1892
• Preserve resolved path from ColumnsSelector in moveToImpl so that it can be correctly used from compiler plugin by @koperagen in Preserve resolved path from ColumnsSelector in moveToImpl so that it can be correctly used from compiler plugin #1895
• Added a method for handling extension property exceptions by @AndreiKingsley in Added a method for handling extension property exceptions #1896
• Re-enabled incremental compilation by @Jolanrensen in Re-enabled incremental compilation #1899
• Fixed format for column group by @AndreiKingsley in Fixed format for column group #1898
• Disabled renaming to camelcase for generateInterfaces/DataClasses functions by @Jolanrensen in Disabled renaming to camelcase for generateInterfaces/DataClasses functions #1922
• Inlined AnyFrame, AnyRow, and AnyBaseCol type aliases in public API by @Allex-Nik in Inline AnyFrame, AnyRow, and AnyBaseCol type aliases in public API #1863
• Ensured scoped table retrieval for MySQL/MariaDB connections by @zaleslaw in Ensure scoped table retrieval for MySQL/MariaDB connections #1901
• Fixed SQL query validation: replace strictValidation with SqlValidation enum by @zaleslaw in ## Fix SQL query validation: replace strictValidation with SqlValidation enum #1881
• To type overloads with diff nullability by @Allex-Nik in To type overloads with diff nullability #1839
• Enable Module dependencies in KoDEx context by @Jolanrensen in Enable Module dependencies in KoDEx context  #1910
• Bumping project to Kotlin 2.4 by @Jolanrensen in Bumping project to Kotlin 2.4 #1891
• dependencyUpdates for example projects by @Jolanrensen in dependencyUpdates for example projects #1893
• Typesafe conventions bump and keywordsGenerator convention plugin by @Jolanrensen in Typesafe conventions bump and keywordsGenerator convention plugin #1915
• Updated dependencies and fixed compatibility issues by @Jolanrensen in Updated dependencies and fixed compatibility issues #1924

Documentation and Examples

• Expanding and updating custom SQL DB docs by @Jolanrensen in Expanding and updating custom SQL DB docs  #1710
• Parsing/convert docs update by @Jolanrensen in Parsing/convert docs update #1817
• Added parsing date-time migration docs for 1.0 by @Jolanrensen in Added parsing date-time migration docs for 1.0 #1830
• moving titanic example to examples/projects and enabling compiler plugin by @Jolanrensen in moving titanic example to examples/projects and enabling compiler plugin #1832
• Added kdocs for DataFrame.concat by @koperagen in Added kdocs for DataFrame.concat, updated its signature for better plugin support, annotated DataFrameBuilder APIs for plugin support #1828
• fixed ExcludeFromSources in DslGrammarTemplateColumnsSelectionDsl.kt to exclude only the right elements by @Jolanrensen in fixed ExcludeFromSources in DslGrammarTemplateColumnsSelectionDsl.kt to exclude only the right elements #1838
• Docs warnings update by @Jolanrensen in Docs warnings update #1843
• Improved error message in readCsv when colTypes causes parsing exception by @Jolanrensen in Improved error message in readCsv when colTypes causes parsing exception #1848
• Update website documentation for groupBy and pivot by @Allex-Nik in Update website documentation for groupBy and pivot #1800
• Moved JSON-OpenAPI example and updated docs by @Jolanrensen in Moved JSON-OpenAPI example and updated docs #1845
• Added missing samples changes by @AndreiKingsley in Added missing samples changes #1861
• Added texts samples saving by @AndreiKingsley in Added texts samples saving #1862
• Apply korro to samples in data sources by @Allex-Nik in Apply korro to samples in data sources #1833
• Updated data schema documentation by @AndreiKingsley in Updated data schema documentation #1859
• Llms txt for website by @Jolanrensen in Llms txt for website #1874
• Moving (and fixing) kotlin-spark example to /projects by @Jolanrensen in Moving (and fixing) kotlin-spark example to /projects #1882
• Moving final examples to /projects by @Jolanrensen in Moving final examples to /projects #1886
• Clarify the descriptions of some functions in the pivot docs by @Allex-Nik in Clarify the descriptions of some functions in the pivot docs #1860
• Add medianBy and percentileBy to the groupBy grammar by @Allex-Nik in Add medianBy and percentileBy to the groupBy grammar #1878
• Configure KorroGenerateTask to run after test tasks by @AndreiKingsley in Configure KorroGenerateTask to run after test tasks #1890
• Added troubleshooting guide for Data Schemas and Extension Properties by @AndreiKingsley in Added troubleshooting guide for Data Schemas and Extension Properties #1870
• Update the documentation of Row condition by @Allex-Nik in Update the documentation of Row condition  #1889
• Group by values exceptions by @Jolanrensen in Group by values exceptions #1905
• Updated quickstart guide by @AndreiKingsley in Updated quickstart guide #1906

Full Changelog: v1.0.0-Beta5...1.0.0-rc01

---

This discussion was created from the release 1.0.0-rc01.