This site provides the replication package for the TPCTC 2021 paper Tell-Tale Tail Latencies: Pitfalls and Perils in Database Benchmarking
supplementary_material contains the figures for the experiments conducted with PostgreSQL as well as the time series plots with all queries.
NOTE: An archival version of the pre-built docker image, together with a copy of the git repository and the measured data, are available at the DOI 10.5281/zenodo.5112729.
- Clone the repository
git clone https://github.com/sdbs-uni-p/tpctc2021.git
- Build the Docker image from scratch
cd tpctc2021
docker build -t tpctc2021:latest .
- Create a new container
docker run -t -d --tmpfs /tmp --name tpctc2021 tpctc2021:latest
- Attach to the container
docker exec -it tpctc2021 /bin/bash
- Generate the database data directories used for benchmarking. The database data directories are stored in
$HOME/db_data_dir
.
cd $HOME/measure
./generate_db_data_dir.sh
- Execute benchmark(s). The benchmark data is stored in
$HOME/measure/results
.
cd $HOME/measure/benchmark
./benchmark-noop.sh
./benchmark-ycsb.sh
./benchmark-tpcc.sh
- Create plots based on the results stored in
$HOME/measure/results
. The plots are stored in$HOME/measure/plots
.
cd $HOME/measure/plot
./plot_all.sh
Our experiments were performed without Docker. Still, we got the same results when we ran the experiments in Docker (on a Linux host) by using this image.
We assume that sufficient main memory is available. Epsilon GC reserves 180 GiB of main memory while 160 GiB are pre-allocated. This can be adjusted in benchmark/benchmark-execution.
We assume NUMA-awareness with non-overlapping cores. The database process is pinned to cores 2-22:2 (2, 4, 6, ..., 22) and OLTPBench is pinned to cores 24-44:2 (24, 26, 28, ..., 44). All of these cores should reside on one CPU (NUMA node). For measurements, disable simultaneous multithreading (SMT), Turbo Boost and all cores should operate in the performance P-State with a constant maximum CPU frequency.
The pinned cores can be configured in benchmark/execution-helper (variables taskset_db
and taskset_oltpbench
).
We assume that /tmp
is mounted as tmpfs.
See Dockerfile for all details about the image.
The environment variable SOURCE_FROM_ONLINE
(Dockerfile) controls whether to download the sources (e.g. MariaDB, PostgreSQL, ...) from the internet (SOURCE_FROM_ONLINE=1
) or get them from a directory residing on disk (SOURCE_FROM_ONLINE=0
).
If SOURCE_FROM_ONLINE
is set to 0
, the location of the expected directory (and its sources) can be adjusted in the scripts in scripts/build. The default is a directory sources
relative to this directory containing the following files:
mariadb.tar.bz2
- MariaDB source code (git).
postgres.tar.bz2
- PostgreSQL source code (git).
oltpbench.tar.bz2
- OLTPBenchmark source code (git).
openjdk16-hotspot.tar.gz
- HotSpot JVM downloaded from AdoptOpenJDK.net.
openjdk16-hotspot.tar.gz
- OpenJ9 JVM downloaded from AdoptOpenJDK.net.
While benchmarking, before a database process is started, the file system buffer is flushed (sync
) and the cache is cleared (echo 3 > /proc/sys/vm/drop_caches
). As /proc is mounted as read-only file system and we execute inside a docker container, these commands have been commented out. To enable them, modify benchmark/execution-helper (function start_db
).