Boosting Spark to 2.1.0. Adding tests that include Java types encoded … #105
Conversation
imarios
changed the title
Codecov Report
@@ Coverage Diff @@
## master #105 +/- ##
==========================================
+ Coverage 90.72% 90.82% +0.09%
==========================================
Files 25 25
Lines 507 512 +5
Branches 7 7
==========================================
+ Hits 460 465 +5
Misses 47 47
Continue to review full report at Codecov.
|
|
Apparently Spark's 2.10 jar contains an old version of shapeless, introduced by this PR apache/spark#14150. I hope this won't cause binary incompatibilities... |
| @@ -32,4 +32,12 @@ object implicits { | |||
| // implicit def floatToDouble[T](col: TypedColumn[T, Float]): TypedColumn[T, Double] = col.cast[Double] | |||
| // implicit def floatToBigDecimal[T](col: TypedColumn[T, Float]): TypedColumn[T, BigDecimal] = col.cast[BigDecimal] | |||
| } | |||
|
|
|||
| object injections { | |||
| implicit val javaBoolean: Injection[java.lang.Boolean, Boolean] = Injection(a => a, b => b) | |||
There was a problem hiding this comment.
DId you consider Injection[java.lang.Boolean, Option[Boolean]]? Often when I see Java boxed primitives, it's Java-style Scala code using nulls.
There was a problem hiding this comment.
Hmm I didn't think about this. From what I tested, using Optional here doesn't come for free in terms of extra code being generated. My main goal with the Java types here was for completeness and and enrich some of the tests.
|
@OlivierBlanvillain looks to be ok for now! I didn't see any serious issue with 2.1.0. |
|
@OlivierBlanvillain I didn't really see any problems in our code. We can try to add an exclusion rule in Spark to make sure that they are using our own shapeless. Else we will need to publish shapeless under a separate maven coordinate so the class-paths don't collide. Shading will not really work for us here. |
|
will there be a way to get support for several spark version? e.g. 2.0 and 2.1? |
|
@julien-truffaut I think the best is to release a new version of Frameless for every new version of Spark and have a table saying Spark 2.0.x use Frameless 0.2.0 for Spark 2.1.0 use Frameless 0.3.0, etc. Supporting multiple Spark version for every release will be a lot of extra work for us (testing and validating all new features across different spark version). We might want to do that later if we see people asking for it and us having more contributors to actually take on the challenge. |
|
Fixed problems with udfs #107 |
|
Hey @kanterov, let me know if you want any changes for this PR. thanks! |
|
@imarios hate to be the one to say it, but I really want to use frameless at $work (hence the sudden flurry of PRs) and we're stuck on 2.0 for the time being. If someone needs to maintain cross-building for 2.0 and 2.1 then I'll volunteer, as long as we can drop Scala 2.10 (hopefully nobody is using Spark 2 on Scala 2.10?) |
|
@jeremyrsmith so glad that you guys use it work! I think it's a good idea to move Frameless together with Spark. Maybe not same day releases but if we are having a new release it should be with latest Spark. The assumption here is that anyone using frameless is production is also brave enough to be updating their Spark dependencies as new version come out :D |
|
@imarios Could you explain the need for this |
|
@OlivierBlanvillain I will actually revisit this PR. Maybe close and open two separate. One for boosting Spark and one for the Injection tests. Right now I wouldn't consider this for 0.3.0. |
…using injection.