Stats computation not working

[SrTangente]

Stat computation like this snippet:

val statsDF = DeltaLog.forTable(spark, path).update().allFiles.map(_.stats)

is not working for qbeast-spark 1.0.0. Maybe an alternative can be found to extract this information

[osopardo1]

I cannot reproduce the error that you are experiencing.

I've tried:

• Version 1.0.0-SNAPSHOT working with Spark 3.4.1 and Delta 2.4.0. Fine
• Version 1.0.0-6a780ea1-SNAPSHOT working with Spark 3.5.0 and Delta 3.0.0. Fine too

Please check if you are using the right Delta Versions when reading the DeltaLog. You should use the same version to read and write.

[Jiaweihu08]

Closed for inactivity.

[SrTangente]

The stats are extracted for only 32 random columns and only one of the cubes, not all of them

[osopardo1]

1. The number of columns used to compute the stats can be set with a table property from Delta: delta.dataSkippingNumIndexedCols. Since it's a table property, you should create the table before hand with the custom property and write your dataframe with saveAsTable or insertInto to make the change effective.
   This is not a feature from Qbeast.
2. The stats are always collected per file, not per cube. :( We rely on Delta implementation of stats collection, and that structure is not designed to fit multiple min and max for each column in a single file.