CuPy Benchmark
Switch branches/tags
Nothing to show
Clone or download
kmaehashi Merge pull request #13 from asi1024/remove-compile-mode
Remove 'compile' mode temporarily
Latest commit 2a1a757 Aug 25, 2018


CuPy Benchmark

Benchmarking CuPy with Airspeed Velocity.


You need to install asv benchmark framework.

pip install asv


# Enable ccache for performance (optional).
export PATH="/usr/lib/ccache:${PATH}"
export NVCC="ccache nvcc"

# Run benchmark against target commit-ish of CuPy.
asv run --step 1 master
asv run --step 1 v4.0.0b4

# Compare the benchmark results between two commits to see regression
# and/or performance improvements in command line.
alias git_commit='git show --format="%H"'
asv compare $(git_commit v4.0.0b4) $(git_commit master)

# Convert the results into HTML.
# The result will be in `html` directory.
asv publish

# Start the HTTP server to browse HTML.
asv preview

Alternatively you can use Docker.

# Build docker image for benchmark.
docker build -t cupy-benchmark docker

# Create a machine configuration file (`.asv-machine.json`) in this directory (first time only).
nvidia-docker run --rm -u $(id -u):$(id -g) -v ${PWD}:/benchmarks -w /benchmarks -e HOME=/benchmarks cupy-benchmark asv machine --machine $(hostname)

# Run benchmark.
nvidia-docker run --rm -u $(id -u):$(id -g) -v ${PWD}:/benchmarks -w /benchmarks -e HOME=/benchmarks cupy-benchmark asv run --step 1 master

NumPy Benchmark

benchmarks/numpy directory contains benchmark scripts forked from NumPy 1.14.2, with the following modifications:

  • Replace import numpy as np with import cupy as np.
  • Add @sync decorator to each benchmark classes.

Some other tiny modifications (remove unsupported dtypes etc.) are also applied to fill the gap between NumPy and CuPy.

Benchmarks for features implemented in NumPy but not in CuPy are left intentionally. It is an expected behavior to see such error (e.g., AttributeError: module 'cupy' has no attribute 'tril') during benchmark.