Records per partition

[sbottelli]

Hi!
I have a question: how can I see the number of records per partition?
I want to know how the records are distributed in order to see some skewness in the data and eventually adjust them with some workarounds (to speed up performances)
Thanks for the hard work that you do for mantain this very outstanding package!
Regards
Stefano

[yitao-li]

@sbottelli Good question!

At the moment you can run spark_apply(sdf, function(x) nrow(x)) to get the partition sizes (probably not very efficient, but does get the job done). Example:

library(sparklyr)
sc <- spark_connect(master = "local")

sdf <- sdf_len(sc, 100, repartition = 10L)
print(spark_apply(sdf, function(x) nrow(x)))  # result is implementation-dependent but most likely it will show 10 partitions each of size 10

[yitao-li]

Also notice if you just want to know whether some partition strategy will produce skew or not, it probably makes sense to apply that partition strategy to some subset of data first and then run spark_apply(...) from my previous comment on that subset. spark_apply could be slow when running on large number of rows.

[sbottelli]

Thanks @yitao-li ! I will test it!

[yitao-li]

@sbottelli I think it may also be a good idea to first only select the subset of column(s) you are partitioning with into a smaller dataframe -- otherwise it looks like Spark might start computing all other columns which are irrelevant to distribution of rows in partitions (I could be wrong on this one though but it doesn't hurt to always guard against computations that are unnecessary)

[sbottelli]

yes, I partition the table by using an ID column and then calculating the distribution of rows in each partition to examine skew.

Thanks for the code, the tricks and the clear answer!

[benmwhite]

spark_apply(sdf, function(x) nrow(x))

@yitao-li this is one of the hackiest uses of spark_apply() I've ever seen, I love it