Version 3.4.2

What's Changed

3.4.2 (2024-08-08)

• [ADD] Annotator: New transformer to load and apply comments to dataframes
• [MOD] Mapper: Change mode and missing_column_handling from strings to Enums
• [MOD] Mapper: Add support for column comments (via annotator)

Full Changelog: v3.4.1...v3.4.2

v3.4.1

What's Changed

• Add validation mode for mapper by @Breaka84 in #59
• Fix mapper tests for older pyspark version by @rt-phb in #62

Full Changelog: v3.4.0...v3.4.1

Version 3.4.0

What's Changed

• Parameterizable Spooq Mapping Transformations by @Breaka84 in #39
• Bump werkzeug from 2.0.1 to 2.2.3 by @dependabot in #45
• Bump pyspark from 3.0.1 to 3.2.2 in /docs/source by @dependabot in #41
• Bump pyspark from 3.0.1 to 3.2.2 by @dependabot in #42
• Bump future from 0.18.2 to 0.18.3 by @dependabot in #46
• Fix Tests on Github Actions by @Selbstsesslingen in #53
• Fix readthedocs by @Breaka84 in #54
• Fix release action by @Breaka84 in #55
• Fix read-the-docs & update readme by @Selbstsesslingen in #56
• remove long_description_content_type from setup.py by @Breaka84 in #57
• drop changelog from pypi description (cannot render rst documentation) by @Breaka84 in #58

Full Changelog: v3.3.9...v3.4.0

Version 3.3.9

What's Changed

• Single parameter for handling missing columns by @rt-phb in #40

Full Changelog: v3.3.8...v3.3.9

Version 3.3.8

What's Changed

• Handle missing input columns by @rt-phb in #38

Full Changelog: v3.3.7...v3.3.8

Version 3.3.7

What's Changed

• Fix long overflow in extended_string_to_timestamp by @emavgl in #36

Full Changelog: v3.3.6...v3.3.7

Version 3.3.6

[FIX] Fix Cleaners logs in case of field type different than string

Version 3.3.5

[ADD] Add Null Cleaner spooq.transformer.NullCleaner

Version 3.3.4

• [MOD] Store null value instead of an empty Map in case no cleansing was necessary

Version 3.3.3

• [MOD] Change logic for storing cleansed values as MapType Column to not break Spark (logical plan got to big)
• [MOD] Add streaming tests (parquet & delta) for EnumCleaner unit tests.