HEBench

HEBench is a flexible benchmarking framework for Homomorphic Encryption that allows fair performance comparison among different implementations (in hardware and/or software) of a collection of homomorphic encryption workloads.

For an overview of HEBench, please visit the homepage here HEBench Home.

This repo contains the frontend for HEBench which is the starting point for all HEBench framework functionality.

Table of Contents

1. About Frontend
   1. Test Harness
   2. Online Documentation
2. Build Requirements
3. Build Configuration
   1. Build Type
   2. Third-Party Components
   3. Advanced Configuration
      1. Required Third-party Components
      2. Configuring Pre-Installed API Bridge
      3. Configuring Pre-Installed YAML CPP
4. Building
5. Running a Benchmark
6. Building the Documentation
7. Contributing
8. Citing HEBench

About Frontend

The frontend is the starting point for all HEBench benchmarking framework functionality. It consists of the components relevant to users running benchmarks, extending the test harness with custom workloads, or contributing to the framework.

Test Harness

Test Harness is the core component for HEBench frontend. It orchestrates the benchmarking operations, submits and times requests to backends. A user wanting to run any HEBench benchmark should start here as they must call the Test Harness executable with appropriate arguments in order for the benchmarking to occur.

Online Documentation

The detailed online documentation for HEBench Framework can be found here HEBench online documentation.

Build Requirements

Current build system uses CMake.

• Ubuntu 16.04, 18.04, or 20.04
• C++17 capable compiler (tested with GCC 9.3)
• CMake 3.13 or above
• Git 2.7.4 or above
• Doxygen 1.9.1 (optional - for documentation generation)

Build Configuration

Build Type

If no build type is specified, the build system will build in Debug mode. Use -DCMAKE_BUILD_TYPE configuration variable to set your preferred build type:

• -DCMAKE_BUILD_TYPE=Debug : debug mode (default if no build type is specified).
• -DCMAKE_BUILD_TYPE=Release : release mode. Compiler optimizations for release enabled.
• -DCMAKE_BUILD_TYPE=RelWithDebInfo : release mode with debug symbols.
• -DCMAKE_BUILD_TYPE=MinSizeRel : release mode optimized for size.

Third-Party Components

By default, the build system will pull the required third-party component versions from remote repos and build locally. Thus, this build system requires Internet access and being able to clone the required third-party repos using git.

The majority of warnings generated by third-party libraries are shown by default. If users want to hide these (e.g. to make debugging this project easier), set CMake flag -DHIDE_EXT_WARNINGS=ON. This will hide most warnings generated by third-party components built by this build system.

Advanced Configuration

Users can pull, build and pre-install the required third-party components. This feature is mostly tailored for development, testing, and debugging. Otherwise, using the default behavior is recommended and this section can be skipped.

If no pre-installed version of a component is found, the correct version will be pulled from remote repo and build locally.

The build system will search for pre-installed third-party components in each of the locations below, in order, until one is found:

1. -D{COMPONENT_NAME}_INCLUDE_DIR and/or -D{COMPONENT_NAME}_LIB_DIR (lib dir is not needed for header only components).
2. -D{COMPONENT_NAME}_INSTALL_DIR
3. /usr/local

Note that if pre-installed versions are incompatible with the current version of the frontend, the build will fail.

See the following sections for the specific configuration variable names for each required third-party component.

Required Third-party Components

• HEBench API Bridge
• YAML CPP

Configuring Pre-Installed API Bridge

The API Bridge is the component in HEBench that allows communication between Test Harness and backends.

If API Bridge has been pre-built, users can point the build system to the pre-installed version of API Bridge using the following CMake config variables:

-DAPI_BRIDGE_INCLUDE_DIR: include directory for API Bridge.

-DAPI_BRIDGE_LIB_DIR: lib directory for API Bridge.

-DAPI_BRIDGE_INSTALL_DIR: base directory for API Bridge. Contains both include and lib. This will be used if any of the previous config variables for API Bridge have not been set.

Configuring Pre-Installed YAML CPP

YAML CPP third-party component is required to read and parse yaml files.

If YAML CPP has been pre-built, users can point the build system to the pre-installed version using the following CMake config variables:

-DYAML_CPP_INCLUDE_DIR: include directory for YAML CPP.

-DYAML_CPP_LIB_DIR: lib directory for YAML CPP.

-DYAML_CPP_INSTALL_DIR: base directory for YAML CPP. Contains both include and lib. This will be used if any of the previous config variables for YAML CPP have not been set.

Building

Build from the top level of frontend with CMake as follows:

# assuming frontend is already cloned
cd ~/frontend
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALL_LOCATION # change install location at will
make -j
make install

The install step will copy the executable binaries, libraries, and includes into bin, lib, and include, respectively.

Running a Benchmark

The executable for the Test Harness is test_harness.

A benchmark is implemented into a shared library that exposes the API Bridge interface. This library is called a HEBench backend. To benchmark a HEBench backend, users must specify the --backend_lib_path flag to point to the shared library when calling the Test Harness executable.

cd $INSTALL_LOCATION/bin
./test_harness --backend_lib_path /path/to/specified/libbackend.so --report_root_path $REPORT_OUTPUT_PATH

The Test Harness will save the reports and summary of the run to the path specified in $REPORT_OUTPUT_PATH.

The API Bridge builds with an included simple backend example. Assuming the API Bridge has been built and installed (also in $INSTALL_LOCATION) successfully, users may then run as follows:

cd $INSTALL_LOCATION/bin
./test_harness --backend_lib_path $INSTALL_LOCATION/lib/libhebench_example_backend.so --report_root_path $REPORT_OUTPUT_PATH

Note that a backend can implement multiple benchmarks that may support multiple configurations. Configuration files are supported to selected which benchmarks to run and what configuration parameters to use beyond default behavior.

Full details on using the Test Harness can be found in the Test Harness User Guide.

For a list of available backends, visit Published Backends.

Building the Documentation

To build the complete documentation use the provided script in scripts/build_docs.sh.

$ scripts/build_docs.sh

By default, this script will generate the documentation at the root where this repository was cloned where a docs directory will be generated.

Use the -h command line flag to see options to override the default behavior:

$ scripts/build_docs.sh -h

Doxygen 1.9.1 for Ubuntu is required to build this documentation. Building with any other version may result in formatting and content errors. The script to generate the documentation will detect the version of Doxygen. If it doesn't match the requirement, the script will attempt to install the correct version. The user will be asked if the installation should proceed. Note that proceeding with this installation will overwrite the system global version of doxygen with version 1.9.1. If this behavior is not desired, a manual installation is recommended instead. Use the script's command line options to override the location searched for Doxygen.

Contributing

This project welcomes external contributions. To contribute to HEBench, see CONTRIBUTING.md. We encourage feedback and suggestions via Github Issues as well as discussion via Github Discussions.

Citing HEBench

To Cite HEBench in academic papers or otherwise, please use the following BibTeX entry:

@misc{hebench,
    title = {Homomorphic Encryption Benchmarking Framework - HEBench (release v0.9)},
    howpublished = {Online: \url{https://github.com/hebench/frontend}},
    month = nov,
    year = 2022,
    note = {Algorand Foundation, CryptoLab, Deloitte, Duality, IBM Research, Intel, KU Leuven, Microsoft Research, Tune Insight SA}
}