NPZ replacement format (only)

[farizrahman4u]

No description provided.

[verbose-void]

commit docstring suggestions and it looks good

[AbhinavTuli]

Looks good, approving as we need this quickly.
If there's time, assertions should be replaced with exceptions everywhere except for tests.

[codecov]

Codecov Report

Merging #1047 (4d03080) into main (b180668) will not change coverage.
The diff coverage is n/a.

❗ Current head 4d03080 differs from pull request most recent head 7007c59. Consider uploading reports for the commit 7007c59 to get more accurate results

@@           Coverage Diff           @@
##             main    #1047   +/-   ##
=======================================
  Coverage   89.16%   89.16%           
=======================================
  Files          88       88           
  Lines        3998     3998           
=======================================
  Hits         3565     3565           
  Misses        433      433           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b180668...7007c59. Read the comment docs.

[verbose-void]

mac failing, looks like an install error so you can probably just rerun it

[verbose-void]

benchmarks

note: these times are for the full script run. for pytorch this includes process spin-up times. was run on my local macbook pro.

mnist upload times (local to local)

this branch (commit d9a846b): 43s
main branch (commit 4d03080): 45s

mnist pytorch times (local, 1 worker)

this branch (commit d9a846b): 17s
main branch (commit 4d03080): 18s

[verbose-void]

let's make sure this change is 100% necessary before merging

[verbose-void]

after profiling, we deduced that these serializers are 3x faster for 512x512 data and 3.8x faster for 1024x1024 data in comparison to npz. let's get it merged!