Skip to content

Move Spark 4 usage to shim 0.3.0 and frameless with agnostic encoders #90

@chris-twiner

Description

@chris-twiner

unlike #88 this only applies to Spark 4 / DBR 17, the rest remains.

This largely applies to the SparkHelpers.scala as the implicits from frameless to auto lift codecs should be enough.

  • create new testing project, move testutils and ClassicOnly to it
  • testutils to self type as suite and provide withFixture to handle ClassicOnly vs swapping out "sparkSession" and running the test twice
  • introduce override for conf / gencode handling on connect tests (current behaviour on ClassicOnly tagged tests)
  • testShades test runner needs to move to scalatest, move testless' batch runner to the testing, base trait with package name abstract
  • pom move from surefire to scalatest plugin (check reports and coverage work)
    migrate other projects to the testing package
  • non qualitysparkutils version workarounds split out to specific directories:
    • classicEncoding (pre 4) / agnosticEncoding (4 and above)
    • threeOnly / notThree (ExpressionProxy and Unevaluable foldable = false handling)
    • pre32 / post32 (correct currentSubExprState for FunReWrites)
    • pre33_sparkless / post33_sparkless (collect vs exists on subexpr checks) / no_sparkless (DBRs)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions