Skip to content

OpenSSL Performance Benchmark 1.0.15: hardware-accel impact + ML‑DSA variance benchmarks, richer reporting

Choose a tag to compare

View all tags
@tobrien tobrien released this 05 Jan 01:27
· 4 commits to working since this release
v1.0.15
2c87ee4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

  • Hardware acceleration impact is now measured and reported (AVX/AVX2 on x86_64, NEON/Crypto on aarch64), including “with vs without” deltas for AES-256-GCM, SHA-256, and ML-KEM (when available).
  • New ML‑DSA (Dilithium) rejection-sampling variance benchmark (OpenSSL 3.5+) to surface signing tail-latency behavior (P95/P99/P99.9/P99.99, CV%, outliers).
  • Benchmark metadata is more actionable: CPU architecture, core count, flags, and key feature bits are captured in the JSON output.
  • More stable published results by default: config bumps benchmark iterations from 1 → 3 (with regenerated report/dashboard artifacts).
  • Expanded benchmark coverage: SHA‑256 block-size sensitivity metrics are captured (16B/64B/256B) to complement existing AES block-size metrics.

What changed

1) Hardware acceleration (AVX/NEON) comparison tooling

  • Added src/avx_benchmark.sh and integrated it into the main src/benchmark.sh run.
  • Added a standalone runner: scripts/test-avx-impact.sh.
  • Results are exposed both as a nested object (avx_tests) and as first-class metrics.* fields, including:
    • metrics.aes_256_gcm_with_avx_kbs, metrics.aes_256_gcm_without_avx_kbs, metrics.aes_256_gcm_avx_improvement_percent
    • metrics.sha256_with_avx_kbs, metrics.sha256_without_avx_kbs, metrics.sha256_avx_improvement_percent
    • metrics.ml_kem_768_with_avx_ops, metrics.ml_kem_768_without_avx_ops, metrics.ml_kem_768_avx_improvement_percent
    • plus metrics.avx_available / metrics.avx_tests_* status flags

Why it matters: ML-KEM and other modern primitives can be highly SIMD-sensitive. This closes a major interpretability gap when comparing numbers across different runners/hosts.

2) ML‑DSA (Dilithium) rejection sampling variance + tail latency

  • Added src/mldsa_bench.c and compiled it in Docker images only for OpenSSL 3.5+.
  • Integrated into src/benchmark.sh:
    • Captures throughput plus timing distribution metrics for ML‑DSA‑65 (primary), and flags likely rejection-sampling behavior when variance is high.
  • Added scripts/test-mldsa-retry.sh for running the analysis in Docker.
  • Added documentation: docs/MLDSA_REJECTION_SAMPLING.md.

Captured metrics (for ML‑DSA‑65) include:

  • metrics.ml_dsa_available
  • metrics.ml_dsa_65_sign_ops_sec, metrics.ml_dsa_65_verify_ops_sec
  • metrics.ml_dsa_65_sign_cv_percent, metrics.ml_dsa_65_sign_outlier_percent, plus min/max and P50/P95/P99/P99.9/P99.99
  • metrics.ml_dsa_rejection_sampling_detected (boolean)

Why it matters: Dilithium/ML‑DSA signing can have real tail-latency spikes due to internal retries. This release makes that visible in both raw JSON and generated dashboards.

3) More complete CPU/system metadata in benchmark output

src/benchmark.sh now detects and records:

  • metadata.cpu_architecture, metadata.cpu_cores, metadata.cpu_flags
  • metadata.cpu_features (AES, AVX, AVX2/512, SSE4, SHA-NI; ARM maps to NEON/AES/SHA/SVE where possible)

Impact: Report generation and dashboards can show why two runs differ (architecture/features), not just that they differ.

4) Block-size sensitivity: SHA‑256

  • src/benchmark.sh now parses SHA-256 throughput at 16B/64B/256B (in addition to the existing 1K/8K metrics).
  • Adds warnings/debug output when expected openssl speed lines can’t be parsed.

Why it matters: Small-message performance is frequently dominated by per-call overhead; these metrics help explain regressions that don’t show up in large-block throughput.

5) Docker + report/dashboard updates

  • docker/Dockerfile now:
    • Compiles mldsa_bench for OpenSSL 3.5+.
    • Copies benchmark.sh, avx_benchmark.sh, and measure_memory.sh into the image.
  • Report and multipage dashboard generators were updated and regenerated:
    • New/updated pages include hardware acceleration views and a details page.
    • Navigation/breadcrumb copy was cleaned up (emoji removed to match tests).
    • Published pages include a “preliminary result” banner and embed the suite version.

Config changes

  • config/versions.json: default iterations increased from 1 → 3.

Practical impact: more stable numbers, but longer runtime and higher CI cost.

Developer/consumer impact

For users running benchmarks

  • Expect longer benchmark runs, especially on OpenSSL 3.5+ where ML‑DSA analysis is included.
  • New scripts:
    • ./scripts/test-avx-impact.sh <openssl-version>
    • ./scripts/test-mldsa-retry.sh <openssl-version>

For downstream consumers of results/summary.json

  • No breaking schema changes were detected, but the output now includes additional fields.
  • If you validate results with a strict schema, update it to allow the new metadata.cpu_* fields and the new metrics.* keys (AVX/ML‑DSA/SHA small blocks).

Breaking changes

  • No explicit breaking changes were detected in automated analysis.
  • Note: while existing keys are preserved, benchmark runtime and output surface area increased (new metrics and new report pages). If you have tooling that assumes a fixed set of metrics or a short runtime, plan accordingly.
Assets 2
Loading