Skip to content

Latest commit

 

History

History
47 lines (32 loc) · 1.42 KB

README.md

File metadata and controls

47 lines (32 loc) · 1.42 KB
Raw

Benchmark tools

This directory contains some scripts to benchmark translation systems.

Requirements

  • Python 3
  • Docker
python3 -m pip install -r requirements.txt

Usage

python3 benchmark.py <IMAGE> <SOURCE> <REFERENCE>

The Docker image must contain 3 executable files at its root:

  • /tokenize $input $output
  • /detokenize $input $output
  • /translate $device $input $output, where:
    • $device is "CPU" or "GPU"
    • $input is the path to the tokenized input file
    • $output is the path where the tokenized output should be written

The benchmark script will report multiple metrics. The results can be aggregated over multiple runs using the option --num_samples N. See python3 benchmark.py -h for additional options.

Note: the script focuses on raw decoding performance so the following steps are not included in the translation time:

  • tokenization
  • detokenization
  • model initialization (obtained by translating an empty file)

Reproducing the benchmark numbers from the README

We use the script benchmark_all.py to produce the benchmark numbers in the main README. The script builds all Docker images defined in subdirectories and reports the results as a Markdown table. The execution can take up to 3 hours.

# Run CPU benchmark:
python3 benchmark_all.py cpu

# Run GPU benchmark:
python3 benchmark_all.py gpu