Optimizing `PolarisFS` for Zarr files up to 10GB

[cwognum]

Context

To create signed URLs for the directory-based file format called Zarr, Polaris implements a custom Zarr storage class called PolarisFS. We have a first version of this subsystem implemented, but performance is not great yet.

Description

This ticket is concerned with profiling and optimizing the current implementation such that we can read and write larger datasets. Using the internet in the Valence office in Montreal and using Cloudflare's speed test, I found that we have an up- and download speed of about 210 Mbps. This would mean we can up- and download 10GB files in about 6.5 minutes. The goal of this ticket is to be able to read and write a Zarr archive at most three times as slow as when we would've uploaded / downloaded the archive directly. (In other words, using Polaris in the Valence office in Montreal, the upload should take at most ~19.5 min).

Acceptance Criteria

• We can read a 10GB Zarr archive in 13 minutes.
• We can write a 10GB Zarr archive in 13 minutes.
• The log details which optimizations were tried and what their effect was.

Links

• First implementation tracked here: https://github.com/polaris-hub/polaris-hub/issues/266#issue-2109315242
• Related PRs:
  • feat: consolidating metadata for opening zarr files in read mode #82
  • feat: Copy a Zarr archive to the Hub using copy_store for increased efficient #92
  • feat: Listings Caching implementation using fsspec dircache #93
• Document with all optimizations: https://docs.google.com/document/d/1eznBBa_e-WIjsfuXXB875lAouoFpoQtJizxG2yeqRqI/edit?usp=sharing

[cwognum]

@lmtroper We could split this up in multiple separate issues. I can imagine that each of the ideas I listed is a challenge on its own.

[cwognum]

This looks maybe useful: https://filesystem-spec.readthedocs.io/en/latest/features.html#listings-caching

[cwognum]

Some baseline performance we can start optimizing against:

Profiling Report
====================================================================================================
Date: 2024-03-28
Time: 17:38:23
Size: 10.0 KB
Repeats: 5
Polaris version: 0.0.2.dev191+g82e7db2
Zarr version: 2.17.1
====================================================================================================
                         Creating the Zarr archive: 0:00:00.003004 ± 0:00:00.000525
         Creating dataset from Source Zarr archive: 0:00:00.003055 ± 0:00:00.000120
                      Uploading dataset to the Hub: 0:00:28.592137 ± 0:00:00.535505
                          Loading dataset from Hub: 0:00:01.579963 ± 0:00:00.038873
                   Iterating over dataset (remote): 0:00:32.747246 ± 0:00:00.391358
                          Caching dataset to local: 0:00:18.136496 ± 0:00:00.639902
                    Iterating over dataset (local): 0:00:00.003943 ± 0:00:00.003039
           Baseline Zarr only upload to Cloudflare: 0:00:00.874145 ± 0:00:00.336886
             Baseline dataset upload to Cloudflare: 0:00:03.278125 ± 0:00:00.256176
       Baseline Zarr only download from Cloudflare: 0:00:00.327072 ± 0:00:00.047580
         Baseline dataset download from Cloudflare: 0:00:00.682121 ± 0:00:00.400986
====================================================================================================
                        Actual / Baseline - Upload: 8.767 ± 0.583
                      Actual / Baseline - Download: 35.021 ± 15.324

[cwognum]

This may be relevant: https://zarr.readthedocs.io/en/stable/api/storage.html#zarr.storage.LRUStoreCache

[cwognum]

Closing this in favor of #173