AnyBlob - Universal Cloud Object Storage Download Manager
In this repository, we present AnyBlob. AnyBlob is a universal download manager that allows to retrieve and upload objects to different cloud object stores. Our download manager uses less CPU resources than cloud-vendor provided libraries while retaining maximum throughput performance. AnyBlob leverages IO_uring for superior performance per core. For experimental results, please visit our research paper at PVLDB 16.
Building AnyBlob
AnyBlob relies on some third-party libraries:
- uring
- openssl
- jemalloc
For Ubuntu 22.04+ the following command installs the third-party libraries:
sudo apt update && sudo apt install liburing-dev openssl libssl-dev libjemalloc-dev lld g++ cmake
For building, use the following commands:
git clone --recursive https://github.com/durner/AnyBlob
mkdir -p build/Release
cd build/Release
cmake -DCMAKE_BUILD_TYPE=Release ../..
make -j16
Using AnyBlob
AnyBlob is a modern C++ library that can be linked to your program. We provide two examples how to use AnyBlob in your project. Please find a simple example that does only download a single object from AWS, and a more sophisticated example that runs download benchmarks. There, you can also find information on how to easily integrate the building process into existing CMake projects for an automated compilation and static linkage of AnyBlob. The anyblob.cmake should be used in your project to integrate this repository as external project.
Contribution
If you have bug fixes or improvement, please do not hesitate to open a new merge request.
For coverage testing you can simply make coverage and open the coverage report found in the build directory at coverage/index.html.
Cite this work
If you are using AnyBlob in your scientific work, please cite:
Exploiting Cloud Object Storage for High-Performance Analytics
Dominik Durner, Viktor Leis, and Thomas Neumann
PVLDB 16 (11), 2023, 49th International Conference on Very Large Data Bases