Support column level statistics

[nicor88]

https://aws.amazon.com/about-aws/whats-new/2023/11/aws-glue-data-catalog-generating-column-level-statistics/

Add additional configurations that allow the user to add column level statistics to the table.
Minimal config to make it work:

• collect_statistics: boolean (can be configured on the project level)
• glue role that collects statistics
• columns to get statistics (if empty or None we must collect statistics for the all columns) - optional if statistics are enabled

Open questions

• for incremental loads do we drop all statistics and recreate new ones? Or we just run a new start_column_statistics_task_run
• are all table types supported? seems only supported by hive tables, not iceberg.

Notes

Currently not available in all regions

[roslovets]

Did anyone find a use case for the column statistics feature?

I tried to apply it to the unique Id field of a big table and after several minutes it computed a totally wrong number of unique values. Also it did not speed up simple sql queries at all.

I agree that it looks appealing to automate these statistics with dbt. But would it be useful in real life? Given that it can slow down project building significantly.

[jessedobbelaere]

@roslovets I believe the main reason is a potential performance gain indeed, according to this new Cost-Based Optimizer for Athena. I haven't seen hands-on test results yet though.

• https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-athena-cost-based-optimizer-enhance-query-performance/
• https://aws.amazon.com/blogs/big-data/speed-up-queries-with-cost-based-optimizer-in-amazon-athena/

[nicor88]

Here another article where AWS shows improved performances
https://aws.amazon.com/blogs/big-data/enhance-query-performance-using-aws-glue-data-catalog-column-level-statistics/

[roslovets]

Thank you for the links folks. According to their fancy examples we should be able to really save time on downstream models and tests even if it takes up to several minutes to compute statistics for one table.

But I still cannot get why select count distinct differs from the value I see in computed statistics. It ruins the whole idea and potentially makes a query planning and data processing inadequate.

Maybe you could do tests on your big tables as well?