Parameterizable Spooq Mapping Transformations #39

Breaka84 · 2022-08-12T14:52:20Z

3.4.0 (2022-08-12)

[MOD] Mapper: Custom transformations can now also be used with select, withColumn or where clauses
[MOD] Mapper: Custom transformations can now be passed as python objects with or without parameters
[MOD] Mapper: Spark's built-in data types can now be passed as simple strings (f.e. "string")
[MOD] Mapper: Renaming (shortening) of most custom Mapper transformations
[ADD] Mapper: str_to_array transformation
[ADD] Mapper: map_values transformation
[ADD] Mapper: apply transformation
[MOD] Tests use now Python 3.8
[MOD] Spark 3.3.0 compatibility (Tested for all Spark version from 3.0 to 3.3)
[MOD] Clean up documentation

New documentation should be online in a few minutes
https://spooq.readthedocs.io/en/branch-3.4.0/

…ta type in mapper

…q_mapping_transformations

…revity reasons

…st with dynamic min/max expressions

…ctions

…till need to figure out why pycharm does not show the docstring as expected (numpy style)

… spark versions for the master merge checks

…nsformations

…ns); => move string function fixtures to own file

… set delta packages manually for github action tests

Breaka84 and others added 30 commits October 1, 2021 16:54

add functionality to use functions (with or without parameters) as da…

2a32eb4

…ta type in mapper

add remaining transformations

ab7fb86

fix string transformation tests

01f3862

add tests for as_is & to_json_string

858dc58

add some more tests

89795b8

Merge branch 'master' of github.com:Breaka84/Spooq into flexible_spoo…

e200cf5

…q_mapping_transformations

add tests for alternative_source_columns + unix_ts_to_unix_ts

f2d71c3

add tests for different syntax; add tests for output_type casting

e808a60

add more tests, rename "extended_string_to_xxx" => "str_to_xxx" for b…

7e094c0

…revity reasons

add option to directly call transformations; add threshold cleaner te…

5b647e8

…st with dynamic min/max expressions

add tests and code for map_values

16d69ff

drop duplicate test fixture

10748e9

finish tests

9691bc5

add more pyspark versions for push_to_master tests; clean up github a…

14b306b

…ctions

update map_values; add tests; start to update docstrings (as_is) -> s…

4cb4ce0

…till need to figure out why pycharm does not show the docstring as expected (numpy style)

add some docstrings

55a4090

update docstring

c61627c

add docstrings

1fbc86d

add docstring for map_values

1079317

add docstrings; remove first_of_month function

4cda57f

fix sphinx; bump version; update requirements (spark 3.2.1); add more…

c240240

… spark versions for the master merge checks

Merge remote-tracking branch 'origin' into flexible_spooq_mapping_tra…

c05ccd5

…nsformations

update documentation

53efa9a

update requirements.txt for readthedocs

f07ad40

update changelog

2326c92

fix spark version dependencies (different output for different versio…

a0b1e7d

…ns); => move string function fixtures to own file

remove ipython imports

8d42983

fix some str_to_timestamp global limit tests

41e2534

add convenience methods like to_str and to_int

ca5ce5d

add todos

dba30b8

dae added 2 commits February 17, 2023 16:57

tidy up documentation

87e3589

solve merge conflict

66d3b90

Breaka84 added test-it Triggers github test action and removed test-it Triggers github test action labels Feb 17, 2023

Breaka84 marked this pull request as ready for review February 17, 2023 16:25

Breaka84 requested review from rt-phb and emavgl February 17, 2023 16:25

fix github actions

a0fcda7

Breaka84 added test-it Triggers github test action and removed test-it Triggers github test action labels Feb 17, 2023

switch from pytest-spark to manual spark fixture; add delta packages;…

c5c17f9

… set delta packages manually for github action tests