Optimize Iterable<Map>.toDataFrame conversion

[koperagen]

Now it hits a lot of heavy reflection calls like type.isSubtypeOf<AnyRow?>(), AnyFrame on each Map -> DataRow.
Collecting values first, inferring type for all gives a visible improvement

17 s -> 1.5 s

Code i used for testing:

val rowCount: Int = 1_000_000
val columnCount: Int = 10
val columns = (0 until columnCount).map { "col$it" }
val maps: List<Map<String, Any?>> = (0 until rowCount).map { rowIdx ->
    columns.associate { col ->
        col to when (columns.indexOf(col) % 4) {
            0 -> rowIdx
            1 -> "value_$rowIdx"
            2 -> rowIdx * 1.5
            3 -> rowIdx % 2 == 0
            else -> null
        }
    }
}

maps.toDataFrame()

fixes #90

[Jolanrensen]

:o TIL what a linked HashSet is

[Jolanrensen]

though, it seems mutableSetOf produces the same thing in Kotlin :) so I thought all sets behaved this way. Still it's nice to be expressive :D

[Jolanrensen]

I wonder... could we do the same trick with Iterable<DataRow<*>>.toDataFrame()?

Currently, it's only optimized when all DataRows originate from the same DF, but else it just calls iterable.map { it.toDataFrame() }.concat() :(

Actually, we can just call map { it.toMap() }.toDataFrame().cast() now XD

[koperagen]

Actually, we can just call map { it.toMap() }.toDataFrame().cast() now XD

Lol. How tables have turned! Yes, i'm positively surprised by how relatively straightforward this is, at least without recursive value conversion. With DataRow we probably need to create nested column groups, so it will be a bit trickier

[Jolanrensen]

Actually, we can just call map { it.toMap() }.toDataFrame().cast() now XD

Lol. How tables have turned! Yes, i'm positively surprised by how relatively straightforward this is, at least without recursive value conversion. With DataRow we probably need to create nested column groups, so it will be a bit trickier

ah yes, of course! let's do that separately :) I like this recent performance-improvement-trend ;P