Datasets of lists of data classes / tuples don't work

[Jolanrensen]

While support for data classes in arrays works fine:

dsOf(
    arrayOf(1 to 2, 3 to 4)
).show()

This doesn't work

dsOf(
    listOf(1 to 2, 3 to 4)
).show()

The exception is:

Exception in thread "main" java.lang.UnsupportedOperationException: No Encoder found for kotlin.Pair[A,B]
- array element class: "kotlin.Pair"
- root class: "java.util.List"

I suspect this is because the encoding for lists in the schema() function of the API misses a recursive call on its subtypes somehow. Needs to be investigated though, as it would also fix Datasets of lists of Tuples I think.

[Jolanrensen]

So, I've narrowed down the issue to the deserializerFor() function where predefinedDt seems to be None where this isn't the case for serializerFor() while trying to find a serializer for a data class. Now I need to trace back how this method is called and where it differs from serializerFor()...

Edit: Found it!!!

[Jolanrensen]

Supplied the pull request: #89 :)
It was literally one .dt too much somewhere in the 1100+ lines of code haha.

[asm0dey]

So, where I can send a beer?

[asm0dey]

But yes, this was the hell to write this code: it's barely debuggable and understandable. And I don't have any idea on how to do it better

[Jolanrensen]

@asm0dey
Hahaha, well I do have a Paypal somewhere :P, but a mention on the next blog would mean much more to me personally :).

But you're right! I think it's partly because of Scala. It's really easy to write difficult code in it (from what I have seen, that is. Scala is still very new for me). That's why I think the Spark API for Kotlin is such a good idea, because Kotlin invites devs to code better.

[asm0dey]

Currently, the plan is to release version 1.0 after Kotlin 1.5.20 release because a couple of things is fixed there:

1. In 1.5.0, lambda serialization is broken and requires an additional, potentially dangerous compiler flag to WA
2. In 1.5.20, interop with Scala may be improved, which will allow upgrading Scala to the latest minor

Thank you very much for your effort; you will be most definitely mentioned (ping me if you want something specific to be written, please) in the next post :)

[Jolanrensen]

@asm0dey I noticed the lambda serialization as well in my own project! Glad to hear it's being solved.

I'm curious, how will the interop be improved?

[asm0dey]

@Jolanrensen it looks like you need to set up a free SSL certificate at https://www.jolanrensen.nl/ ;)

[Jolanrensen]

@asm0dey my website is TLS secured right? It shows up as secure in Chrome for me.

[asm0dey]

Yes, fixed now. Yesterday Firefox told me, that it's not secured.