The Bitcode Database (bcdb)

A database and infrastructure for distributed processing of LLVM bitcode.

The Bitcode Database (BCDB) is a research tool being developed as part of the ALLVM Project at UIUC. It has the following subprojects:

• MemoDB: A content-addressable store and a memoizing distributed processing framework backed by SQLite or RocksDB. It can cache the results of various analyses and optimizations.
• BCDB proper: Builds on MemoDB to store massive amount of LLVM bitcode and automatically deduplicate the bitcode at the function level. All the other subprojects are build on BCDB.
• Guided Linking: A tool that can optimize dynamically linked code as though it were statically linked.
• Outlining: A work-in-progress optimization to reduce code size using outlining.
• SLLIM: A easy-to-use tool to apply various code size optimizations (including our outliner) to existing software without messing around with build systems.
• Nix bitcode overlay: Nix expressions to automatically build lots of Linux packages in the form of LLVM bitcode.

Table of Contents

• Background
• Install
  • Dependencies
  • Building dependencies automatically with Nix
• Usage
• Maintainer
• License

Background

The BCDB has been developed primarily by Sean Bartell, to support his PhD research on code size optimization.

This project initially grew out of the ALLVM Project, started by Will Dietz, which aims to explore the new possibilities that would be enabled if all software on a computer system were shipped in the form of LLVM IR. The BCDB was originally designed as a way to store massive amounts of LLVM IR more efficiently by using deduplication.

Install

mkdir build
cd build
cmake ..
make
make check

Dependencies

Building BCDB without Nix is not officially supported. If you want to try it anyway, you'll need to install these dependencies first:

• C++ compiler with C++17 support.
• LLVM version 11 through 14 (development versions up to 15 may work, but this is not guaranteed)
  • LLVM must be built with exception handling support. Official packages have this disabled, so you'll need to build LLVM yourself with cmake -DLLVM_ENABLE_EH=ON (or use Nix).
  • LLVM's FileCheck and not programs must be installed as well. Some packages (including some of LLVM's official packages) exclude these programs or split them off into a separate package.
  • When working on the BCDB code, you should make sure LLVM is built with assertions enabled (cmake -DLLVM_ENABLE_ASSERTIONS=ON).
• Clang, same version as LLVM.
  • Clang doesn't need exception handling or assertions, so you can use an official Clang package.
• CMake, at least version 3.13.
• Libsodium
• SQLite
• Python, at least 3.6.
• Boost, at least 1.75.
• Optional dependencies:
  • RocksDB, preferably at least 6.19, with LZ4 and Zstandard support (ROCKSDB_LITE is not supported).

Building dependencies automatically with Nix

If you have Nix installed, it can automatically build BCDB along with known-working versions of its dependencies. See default.nix for the list of attributes you can build. For example:

nix-build -A bcdb
result/bin/bcdb -help

If you want to modify the BCDB code, you can instead build just the dependencies with Nix, and enter a shell that has them installed:

nix-shell -A bcdb
mkdir build
cd build
cmake ..
make

If you install and enable direnv, it will effectively set up the Nix shell every time you enter the bcdb directory.

In any case, you can speed up Nix by using our Cachix cache, which includes prebuilt versions of LLVM. Simply install Cachix and run cachix use bcdb.

Usage

See the subproject subdirectories for usage instructions and more documentation.

Maintainer

Sean Bartell.

Contributing

There's no formal process for contributing. You're welcome to submit a PR or contact smbarte2@illinois.edu.

License

Apache License 2.0 with LLVM Exceptions, copyright 2018–2022 Sean Bartell and other contributors. See license in LICENSE.TXT.