100 Trillion Row Challenge

[alxmrs]

How fast could we calculate the min, max, avg temperature for every atmospheric level of ARCO-ERA5?

https://www.morling.dev/blog/one-billion-row-challenge/

https://medium.com/coiled-hq/one-trillion-row-challenge-5bfd4c3b8aef

[alxmrs]

AKA what's the fastest way to calculate a climatology?

[alxmrs]

I easily get confused by order of magnitude. The number of rows is the product of the dimensions of ARCO-ERA5, which I calculate as:

= 732,072 hours * 720 lat * 1440 lng * 37 lvl
= 2.8083453e+13
= 28 Trillion

[alxmrs]

The latest version of ARCO-ERA5 supports all 137 model levels! Thus, the calculation would be:

732,072 hours * 720 lat * 1440 lng * 137 lvl

= 1.0398468e+14
= ~100 Trillion

[alxmrs]

@JacobZuliani, @jayendra-info, @TomNicholas If I were building the 100 Trillion row challenge today, here's how I imagine it would work:

• We'd use IceChunk's new maxed out Zarr IO layer: https://earthmover.io/blog/i-o-maxing-tensors-in-the-cloud
• We'd open ARCO-ERA5 in xql and calculate a partitioned min/max/mean for every weather variable at for every lat/lon across time and pressure level.
• We'd run this partitioned query on Burla's remote_parallel_map across a lot of VMs. https://docs.burla.dev/examples/process-2.4tb-in-parquet-files-in-76s (sorry, forgot to add the link).

I think this could make for a really killer blog post! Let me know if you'd like to team up to build this together.

[jayendra-info]

Thanks for pointing the Burla, new for me worth to check.

[jayendra-info]

We can write a better spec for the challenge , by

1. specifying the source of the data, probably from the GCS buckets mentioned here https://github.com/google-research/arco-era5.
2. specifying the clarity about number of variables, which would be more clearer after specifying the exact source.

[alxmrs]

I plan to write something more complete when we have a first candidate submission, but here is what I had in mind for now:

• Let's adapt the original 1 billion row challenge to gridded datasets: can we calculate the min, max, avg value for every lat/lon/level of weather data, for every data variable present in the ARCO-ERA5 dataset (~12 PiBs worth of data).
• Possibility: instead of aggregating across a single time dimension, it's more realistic (and useful) for the data domain to produce a climatology, specifically an aggregation per day of the year for every location.
• Since the data sizes significantly exceeds what a single machine can do, this will inherently be a distributed systems challenge. To make the metric more fair, this challenge should try to minimize two metrics: time and cost. In other words, what is the fastest, cheapest way to produce this aggregation?