Skip to content

Fast Tensors Packaging library for text, image, video, and audio data compatible with PyTorch, TensorFlow, & NumPy 🖼️🎵🎥 ➡️ 🧠

Notifications You must be signed in to change notification settings

ashvardanian/TenPack

Repository files navigation

TenPack

Three simple things this library does to you:

  1. Guess the media type from raw bytes,
  2. Parse its' dimensions, sizes, lengths, etc.,
  3. Unpack data into regular preallocated Tensors.

Where do we use it? To connect the Data Storage layer of UKV to High-Performance Computing libraries like TensorFlow and PyTorch.

How it works?

Most common file-formats have "signatures" or "magic numbers" embedded into them. Often, as the prefix of the byte-stream.

Libraries implementing the first step have been implemented for other languages:

Alternatives for Tensor Exports

In fact, TenPack is just a CMake-friendly generalization of those libraries with a C interface and focus on memory reusing.

About

Fast Tensors Packaging library for text, image, video, and audio data compatible with PyTorch, TensorFlow, & NumPy 🖼️🎵🎥 ➡️ 🧠

Topics

Resources

Stars

Watchers

Forks