Skip to content

v0.1.0 sz3 layer-by-layer compression

Choose a tag to compare

View all tags
@legendPerceptor legendPerceptor released this 10 Apr 19:25
· 122 commits to master since this release
0.1.0
4759362

We add the layer-by-layer compression method and NUMA-adapted parallel compression support.

What problems did we solve?

  • The conventional tensor compression methods including SZ3 cannot compress exceptionally huge 3D tensors because of memory limit. We solve this problem by compressing the large file layer by layer.
  • Most previous compression methods cannot use multiple nodes to compress a single file on supercomputers. We provide a NUMA-adapted method that utilizes MPI programming to compress a single file with many processors on multiple nodes.
  • We also provide a multi-threading compression implementation for systems without a parallel file system or single-node compression with multiple processors.
Assets 2
Loading