Skip to content

durner/AnyBlob

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

This commit adds the clang toolchain to the CI. We turn the main jobs
into a test matrix that either runs with the clang, or gcc toolchain.

When using the gcc toolchain everything stays as-is. When using the
clang toolchain we use clang/clang++ as C/C++ compilers, respectively.
We also build `AnyBlob` using `-DANYBLOB_LIBCXX_COMPAT=1`, which ensures
that `libcxx` is used as a standard library.

In principle, the test matrix could be even bigger: we could build with
libcxx and gcc, and gnu c++ and clang. But we don't tests this for now.
ab4be80

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
November 28, 2023 23:24
November 28, 2023 23:24
src
November 28, 2023 23:24
November 28, 2023 23:24
May 24, 2023 11:19
May 24, 2023 11:19
May 24, 2023 11:19
November 28, 2023 23:24
May 24, 2023 11:19

AnyBlob - Universal Cloud Object Storage Download Manager

In this repository, we present AnyBlob. AnyBlob is a universal download manager that allows to retrieve and upload objects to different cloud object stores. Our download manager uses less CPU resources than cloud-vendor provided libraries while retaining maximum throughput performance. AnyBlob leverages IO_uring for superior performance per core. For experimental results, please visit our research paper at PVLDB 16.

Building AnyBlob

AnyBlob relies on some third-party libraries:

  • uring
  • openssl
  • jemalloc

For Ubuntu 22.04+ the following command installs the third-party libraries:

sudo apt update && sudo apt install liburing-dev openssl libssl-dev libjemalloc-dev lld g++ cmake

For building, use the following commands:

git clone --recursive https://github.com/durner/AnyBlob
mkdir -p build/Release
cd build/Release
cmake -DCMAKE_BUILD_TYPE=Release ../..
make -j16

Using AnyBlob

AnyBlob is a modern C++ library that can be linked to your program. We provide two examples how to use AnyBlob in your project. Please find a simple example that does only download a single object from AWS, and a more sophisticated example that runs download benchmarks. There, you can also find information on how to easily integrate the building process into existing CMake projects for an automated compilation and static linkage of AnyBlob. The anyblob.cmake should be used in your project to integrate this repository as external project.

Contribution

If you have bug fixes or improvement, please do not hesitate to open a new merge request. For coverage testing you can simply make coverage and open the coverage report found in the build directory at coverage/index.html.

Cite this work

If you are using AnyBlob in your scientific work, please cite:

Exploiting Cloud Object Storage for High-Performance Analytics
Dominik Durner, Viktor Leis, and Thomas Neumann
PVLDB 16 (11), 2023, 49th International Conference on Very Large Data Bases