Managing ZSTD‐Compressed Data From The Beiwe Platform

So what is ZSTD?

ZSTD is a newer compression technology developed by Meta. We went through some pretty extensive testing and found ZSTD to be the best option for both speed and compression ratio.

On average and with the settings chosen on the server, it compresses your data down to 1/5th the raw text size, but it can be as good as 1/10th. (If you are very patient, you can recompress data from the Beiwe platform using more aggressive zstd settings to save even more space, down to 2/3rds of the already-compressed size.)

So why am I seeing a zip file?

That .zip file is not compressed, it is just a wrapper that allows you to download data in a single request and without using Mano, our CLI tool for easy syncing. If you are downloading many gigabytes of data you should use Mano.

The .zst files are compressed individually, they are the raw form we store them in on Beiwe's cloud infrastructure.

Dude, I just need to decompress these

right. so... um. 😅 So it turns out that almost all GUI decompression tools break if you hand them a .zst file that was created by the reference implementation of ZSTD. I'm filing bug reports, this is just silly.

You will have to go through your terminal program to decompress your files

Installing ZSTD as a Python Package

Got pip? If you run pip install pyzstd it will include the zstd binary on all platforms. (If you are using conda just replace pip with conda. Their main, anaconda, and conda-forge repository all have pyzstd)

The pyzstd library includes a cli wrapper that you can invoke with python -m pyzstd

Note: there is also a "zstd" and a "zstandard" python library. You want pyzstd.

Installing ZSTD as a system package

Mac users can install zstd via homebrew, it is simply brew install zstd

Windows users are special and (can download a build directly)[https://github.com/facebook/zstd/releases/] from the github repository, or if you are using chocolatey to manage packages just run choco install zstandard.

Linux users, its just going to be your distribution's zstd library, something like apt install zstd, dnf install zstd...

The CLI tools

The zstd commandline tool details you care about is pretty simple:

• -d to decompress a target file.
• no -d will compress the target file using a similar amount of compression as we use on the server.
• -o to specify the output file.
• no -o will cause the decompressed to appear right next to the .zst, but without .zst on the name.

The zstdcat commandline tool will be available if you installed as a system package

• this tool lets you view the contents of a .zst file in the terminal without decompressing it
• (there are multiple zcat and bzcat commands for different compression standards.
• usage is just like cat zstdcat some_file.csv.zst will print the content to the terminal.

Decompress Everything

Using some very simple BASH and tools on Mac and Linux you can easily decompress many files at once.

• if you are using the pyzstd library just replace it with python -m pyzstd

To decompress many files in the current folder you can do zstd -d *.zst

To decompress many files in two layers of subfolders (Beiwe data) you can do zstd -d */*.zst */*/*.zst

If you have enough files you will get error stating too many arguments were passed in, in which case you can try this incantation:

shopt -s globstar nullglob
for file in **/*.zst; do
  zstd -d "$file"
done

or this one-liner using the find tool: find . -type f -name "*.zst" -exec zstd -d {} \;

Decompress data in Python

If you are working inside Python you will save storage space and execution time by decompressing on the fly.

The basic pattern is very simple:

import pyzstd
with open("some_file_path.zst", 'rb') as f:                    # Open in read mode as a binary files; with statement for safety.
    decompressed_bytes_output = pyzstd.decompress(f.read())    # Read the whole file, you can pass it right in to be decompressed.
print(decompressed_bytes_output)                               # Do Your Thing.

Here's a more complex example of how to walk through a file tree, read in compressed files and implement some minor logic

import os
from pyzstd import decompress
from datetime import datetime
TIME_FORMAT = "%Y-%m-%d %I:%M:%S%p (%Z)"  # 12hr time format

def list_zst_files(path_to_data_folder) -> list[tuple[str, str]]:                    # Get a list of all the file information we could need
    folder_and_files = []
    for current_folder, _subfolders, files in os.walk(path_to_data_folder):          # use os.walk because its easy
        for f in files:
            if f.endswith(".zst"):                                                   # we only care about .zst files
                folder_and_files.append((current_folder, f"{current_folder}/{f}"))
    if not folder_and_files:
        raise FileNotFoundError(f"No .zst files found in `{path_to_data_folder}`")   # yell (politely)
    return folder_and_files

# The folder structure out of a downloaded zip has a structure where the folder always has some
# useful information in it, the file names are just the start time of the dataset in UTC.
for folder, file in list_zst_files("path_to_some_folder_with_nested_zst_files"):
    
    with open(file, "rb") as f:
        compressed_file = f.read()
    decompressed_file = decompress(compressed_file)
    
    info = folder.split("/")[-1]             # the last section is the most relevant folder name
    file_name_time = file.split("/")[-1]     # the last section is the UTC datetime of the file's data
    
    # filenames look like "2022-12-03 09_00_00+00_00.csv.zst"
    file_name_time = file_name_time.split(".")[0]                    # remove the extensions
    file_name_time = file_name_time.replace("_", ":")                # _ -> : = isoformat!
    file_time = datetime.fromisoformat(file_name_time).astimezone()  # convert to computer's timezone
    
    calculation = len(compressed_file) / len(decompressed_file) * 100  # life-or-death calculation
    print(f"{info} - {file_time.strftime(TIME_FORMAT)} - compression ratio: {calculation:.2f}%")