v1.0.16 — OpenSSL build diagnostics + HW acceleration verification, refreshed dashboards

Overview

This release focuses on making benchmark results more explainable and trustworthy by capturing detailed OpenSSL build diagnostics (including assembly/CPU acceleration signals) and surfacing that information in the generated dashboards. It also improves the robustness of memory sampling and refreshes published results/artifacts.

What changed

Benchmarking: capture build diagnostics and verify hardware acceleration

The benchmark runner now records additional environment/build metadata and performs a runtime verification to confirm whether OpenSSL is actually using hardware acceleration.

New metadata captured in summary.json:

• metadata.build_diagnostics
  • version_full: output from openssl version -a
  • openssl_options: output from openssl version -o
  • asm_flags_detected: boolean flags for common ASM indicators (AES/SHA/Poly1305/BN/ECP)
  • hw_accel_verification: a runtime “proof” test that compares AES-256-GCM throughput:
    • normal run (“with HW accel”)
    • run with CPU capabilities masked (OPENSSL_ia32cap on x86_64, OPENSSL_armcap on aarch64)
    • outputs include verified, speedup_ratio, and raw *_kbs values
  • arm (aarch64 only): availability of ARM crypto extensions and whether OpenSSL build appears to enable ARM ASM

Why it matters:
Performance regressions/improvements between OpenSSL versions are often dominated by build configuration and whether assembly/hardware paths are actually active. This release makes those factors explicit so benchmark output is easier to interpret and compare across environments.

Impact:

• Slightly longer benchmark runtime due to additional diagnostics and quick speed checks.
• More verbose stderr logs (JSON output is still on stdout).

Memory sampling: more robust and less brittle

src/measure_memory.sh was hardened to avoid failing the benchmark due to missing tools or platform differences:

• No longer uses set -e (handles errors gracefully).
• Works without bc (falls back to bash arithmetic for sampling interval and averaging).
• Checks for /proc availability and process existence; exits cleanly if unavailable.
• Handles process termination during sampling.

Why it matters:
Memory metrics collection is now less likely to fail in minimal or constrained environments.

Docker image: include required utilities + add build/ASM verification steps

The benchmark Docker image now:

• Installs bc and procps (needed for timing/math and process inspection).
• Adds build-time and post-install checks intended to catch misconfigured builds:
  • greps configdata.pm and Makefile for ASM-related indicators
  • runs openssl version -a after install
  • attempts to locate common ASM symbols in libcrypto via nm

Why it matters:
Helps prevent “silent” benchmark runs where OpenSSL was built without expected optimizations.

Dashboards/visualizations: new build-info page and formatting improvements

Visualization generators were updated (single-page and multipage):

• Adds a “Build Configuration” page that reads from the new metadata and summarizes build/acceleration status.
• Standardizes chart styling:
  • larger axis label font sizes
  • consistent tick formatting using k suffix (lowercase)
• Updates the shared color scale to include the 3.6 series.
• Pages now include a versioned header/footer and additional run context (e.g., last run date from summary.json mtime and per-version iteration counts when present).

Results/artifacts refreshed

• results/*.html and results/summary.json were regenerated to reflect the latest benchmark run and updated templates.
• Release housekeeping: package.json and package-lock.json updated to 1.0.16; generated tooling artifacts under node_modules/ refreshed.

Breaking changes

No breaking API changes were detected.

Compatibility note (for downstream consumers):

• The summary.json schema is extended with a new metadata.build_diagnostics object. This is additive, but any consumers that strictly validate/whitelist fields should allow unknown keys.

Practical guidance

• If you’re investigating unexpected performance differences, start with the new Build Configuration page (or inspect metadata.build_diagnostics in summary.json) to confirm:
  • OpenSSL build options and full version info
  • ASM flags detected
  • whether hardware acceleration was verified via the throughput delta test
• For consistent memory sampling, bc is preferred, but the scripts now degrade gracefully without it.

OpenSSL Performance Benchmark 1.0.15: hardware-accel impact + ML‑DSA variance benchmarks, richer reporting

Highlights

• Hardware acceleration impact is now measured and reported (AVX/AVX2 on x86_64, NEON/Crypto on aarch64), including “with vs without” deltas for AES-256-GCM, SHA-256, and ML-KEM (when available).
• New ML‑DSA (Dilithium) rejection-sampling variance benchmark (OpenSSL 3.5+) to surface signing tail-latency behavior (P95/P99/P99.9/P99.99, CV%, outliers).
• Benchmark metadata is more actionable: CPU architecture, core count, flags, and key feature bits are captured in the JSON output.
• More stable published results by default: config bumps benchmark iterations from 1 → 3 (with regenerated report/dashboard artifacts).
• Expanded benchmark coverage: SHA‑256 block-size sensitivity metrics are captured (16B/64B/256B) to complement existing AES block-size metrics.

---

What changed

1) Hardware acceleration (AVX/NEON) comparison tooling

• Added src/avx_benchmark.sh and integrated it into the main src/benchmark.sh run.
• Added a standalone runner: scripts/test-avx-impact.sh.
• Results are exposed both as a nested object (avx_tests) and as first-class metrics.* fields, including:
  • metrics.aes_256_gcm_with_avx_kbs, metrics.aes_256_gcm_without_avx_kbs, metrics.aes_256_gcm_avx_improvement_percent
  • metrics.sha256_with_avx_kbs, metrics.sha256_without_avx_kbs, metrics.sha256_avx_improvement_percent
  • metrics.ml_kem_768_with_avx_ops, metrics.ml_kem_768_without_avx_ops, metrics.ml_kem_768_avx_improvement_percent
  • plus metrics.avx_available / metrics.avx_tests_* status flags

Why it matters: ML-KEM and other modern primitives can be highly SIMD-sensitive. This closes a major interpretability gap when comparing numbers across different runners/hosts.

2) ML‑DSA (Dilithium) rejection sampling variance + tail latency

• Added src/mldsa_bench.c and compiled it in Docker images only for OpenSSL 3.5+.
• Integrated into src/benchmark.sh:
  • Captures throughput plus timing distribution metrics for ML‑DSA‑65 (primary), and flags likely rejection-sampling behavior when variance is high.
• Added scripts/test-mldsa-retry.sh for running the analysis in Docker.
• Added documentation: docs/MLDSA_REJECTION_SAMPLING.md.

Captured metrics (for ML‑DSA‑65) include:

• metrics.ml_dsa_available
• metrics.ml_dsa_65_sign_ops_sec, metrics.ml_dsa_65_verify_ops_sec
• metrics.ml_dsa_65_sign_cv_percent, metrics.ml_dsa_65_sign_outlier_percent, plus min/max and P50/P95/P99/P99.9/P99.99
• metrics.ml_dsa_rejection_sampling_detected (boolean)

Why it matters: Dilithium/ML‑DSA signing can have real tail-latency spikes due to internal retries. This release makes that visible in both raw JSON and generated dashboards.

3) More complete CPU/system metadata in benchmark output

src/benchmark.sh now detects and records:

• metadata.cpu_architecture, metadata.cpu_cores, metadata.cpu_flags
• metadata.cpu_features (AES, AVX, AVX2/512, SSE4, SHA-NI; ARM maps to NEON/AES/SHA/SVE where possible)

Impact: Report generation and dashboards can show why two runs differ (architecture/features), not just that they differ.

4) Block-size sensitivity: SHA‑256

• src/benchmark.sh now parses SHA-256 throughput at 16B/64B/256B (in addition to the existing 1K/8K metrics).
• Adds warnings/debug output when expected openssl speed lines can’t be parsed.

Why it matters: Small-message performance is frequently dominated by per-call overhead; these metrics help explain regressions that don’t show up in large-block throughput.

5) Docker + report/dashboard updates

• docker/Dockerfile now:
  • Compiles mldsa_bench for OpenSSL 3.5+.
  • Copies benchmark.sh, avx_benchmark.sh, and measure_memory.sh into the image.
• Report and multipage dashboard generators were updated and regenerated:
  • New/updated pages include hardware acceleration views and a details page.
  • Navigation/breadcrumb copy was cleaned up (emoji removed to match tests).
  • Published pages include a “preliminary result” banner and embed the suite version.

---

Config changes

• config/versions.json: default iterations increased from 1 → 3.

Practical impact: more stable numbers, but longer runtime and higher CI cost.

---

Developer/consumer impact

For users running benchmarks

• Expect longer benchmark runs, especially on OpenSSL 3.5+ where ML‑DSA analysis is included.
• New scripts:
  • ./scripts/test-avx-impact.sh <openssl-version>
  • ./scripts/test-mldsa-retry.sh <openssl-version>

For downstream consumers of results/summary.json

• No breaking schema changes were detected, but the output now includes additional fields.
• If you validate results with a strict schema, update it to allow the new metadata.cpu_* fields and the new metrics.* keys (AVX/ML‑DSA/SHA small blocks).

---

Breaking changes

• No explicit breaking changes were detected in automated analysis.
• Note: while existing keys are preserved, benchmark runtime and output surface area increased (new metrics and new report pages). If you have tooling that assumes a fixed set of metrics or a short runtime, plan accordingly.

Benchmark Run 20723215850

Automated benchmark run.

See attached REPORT.md and visualizations.html for details.

Benchmark Run 20702412232

Automated benchmark run.

See attached REPORT.md and visualizations.html for details.

{

"RELEASE_NOTES": {
"title": "v1.0.14 — ML‑KEM hypothesis tooling, OpenSSL target updates, reports regenerated, docs clarification",
"body": "## Summary

This release marks a production version bump to 1.0.14 and delivers three focused updates:

• New reproducible tooling (scripts + Docker) to test the hypothesis that OpenSSL 3.5.x ML‑KEM default keyshare behavior causes TLS handshake slowdowns.
• Updated OpenSSL target versions (patch bumps) and the addition of 3.6.0 to config/versions.json; iteration defaults for quick runs changed to 1.
• Regenerated dashboard and JSON report artifacts and a new ECDH vs ECDSA clarification document that explains how key-exchange and signature benchmarks differ (bandwidth/latency analysis, real-world guidance).

Other smaller changes remove emoji/cosmetic UI text in generated dashboards and add developer helper rules (.cursorrules).

---

Why this matters

• The new ML‑KEM hypothesis tooling gives maintainers and users a reproducible, automated way to verify whether ML‑KEM (the post‑quantum key encapsulation used as a key‑exchange primitive) is the primary cause of the handshake regression observed in some 3.5.x OpenSSL builds. That tooling includes a ready Dockerfile for consistent builds and small benchmark wrappers.
• Updating the OpenSSL target versions and the report generator metadata keeps the benchmark suite aligned with recent OpenSSL patch releases and adds an explicit 3.6.0 target for analysis.
• The ECDH/ECDSA clarification reduces confusion for readers of the site and clearly separates key-exchange (ECDH/ML‑KEM) benchmarking from signature (ECDSA/ML‑DSA) benchmarking — including practical bandwidth and latency implications and mitigation strategies.

---

Highlights (what changed)

1. Version bump

   • package.json: 1.0.13 → 1.0.14. This is the meaningful release version change; other node_modules file changes included are generated artifacts and not semantically significant.

2. New ML‑KEM hypothesis tooling

   • scripts/test-mlkem-hypothesis.sh — end‑to‑end script that builds OpenSSL variants and runs s_time-based handshake benchmarks to compare default vs ECDH-only vs no-ml-kem builds. Adds a simple CSV summary and an analysis step.
   • src/mlkem_comparison_bench.sh — lightweight bench runner used by the Docker image; outputs JSON-style results for programmatic parsing.
   • docker/Dockerfile.mlkem-comparison — Docker image that builds two OpenSSL variants (default and no-ml-kem) and runs the bench script. ARGs available for OPENSSL_VERSION/URL and it installs build dependencies used in CI/local testing.
   • package.json: added npm script test:mlkem-hypothesis -> runs scripts/test-mlkem-hypothesis.sh.

3. Configuration and metadata updates

   • config/versions.json: patch upgrades for multiple series and addition of 3.6.0. The default iterations value was reduced from 10 → 1 to allow quick runs. (This affects generated report iteration counts.)
   • scripts/generate-report.js: added metadata for the bumped versions (3.0.18, 3.1.8, 3.2.6, 3.3.5, 3.4.3, 3.5.4) and new 3.6.0 entry; updated narrative text and generated report header to reference 3.6.0.

4. Documentation / clarification

   • docs/ECDH_VS_ECDSA_CLARIFICATION.md: long-form doc explaining distinction between ECDH (key exchange) and ECDSA (signatures), real‑world impact, bandwidth & latency math, mitigation strategies, and recommended measurement practices.
   • .cursorrules: developer helper rules / project conventions added (note: includes rules like not changing package.json version without maintainer approval).

5. Generated report/dashboard updates

   • results/* artifacts regenerated: updated index.html, overview.html, pqc.html, tls‑comparison.html, openssl_version_analysis.html, memory.html, schmatz.html, plus summary.json and detailed-iterations.json. Notable UI changes: removed emoji, added a "Last run" banner and generation date, iteration text updated to reflect new iterations count and version list.
   • scripts/generate-viz.js and scripts/generate-viz-multipage.js: small text/template and iteration-count handling changes (removed emoji and adjusted copy). Console output messages also simplified.

6. Small cosmetic and tooling updates

   • Many generated files in results/ updated (copy/text/formatting) to reflect new versions and the latest generation date (2026-01-04).

---

Impact and migration notes

• Running the benchmark / reproduction of results

  • Quick local run: npm run test:mlkem-hypothesis (calls scripts/test-mlkem-hypothesis.sh). The script builds OpenSSL in /tmp, runs s_time benchmarks, and writes results to /tmp/openssl-mlkem-test/results.csv.
  • Docker: use docker/Dockerfile.mlkem-comparison to create a container that builds the two OpenSSL variants and runs the bundled benchmark script. This provides a consistent environment across machines and CI.

• Dependencies and environment considerations

  • The build scripts and Dockerfile assume a typical Debian build toolchain and that openssl apps include s_time and s_server. Dockerfile installs build-essential, curl, perl, zlib1g-dev, jq, bc, etc.
  • The build flags used in scripts/Dockerfile: enable-ktls, enable-ec_nistp_64_gcc_128 and no-ml-kem (for the no-ml-kem build). Confirm your toolchain and target platform support these options.

• Reports and iteration counts

  • config/versions.json iteration change (10 → 1) reduces the default iteration count used by the generator — results in quicker runs but higher variance. If you need reproducible/low-variance data, set iterations back to a higher value before generating results.

• Version/URL checks

  • The release includes updated OpenSSL version strings and URLs. Reviewers should confirm those URLs match the intended OpenSSL release artifacts before running automated builds.
  • There are a couple of places where default OPENSSL_VERSION values differ between scripts and Docker ARGs (e.g., Dockerfile ARG default is 3.5.3, test scripts use 3.5.4). Verify the concrete version you intend to test and update ARGs/variables if needed.

• Generated artifacts in node_modules

  • This commit includes regenerated generated files under node_modules (lock/test-results). They are environment artifacts and can generally be ignored for semantic review. CI/release pipelines should rely on package.json for the release version (1.0.14).

---

Breaking changes & important considerations

• No runtime library or public API breaking changes were introduced in this repository. There are no changes that directly break consumers of any published library API.

• Behavior/configuration changes to be aware of:

  • Default iterations changed to 1 in config/versions.json — this changes how many iterations the viz/report generator will run by default and affects reported means/stddev. If you depend on multi-iteration results, update this value.
  • The report generator and visual templates no longer include emoji and include slightly different explanatory copy; dashboards will look different (cosmetic changes only).
  • The ML‑KEM hypothesis tooling configures specific build flags (enable-ktls, enable-ec_nistp_64_gcc_128). These are build-time flags and may require appropriate compilers and system support. They are not applied to your system OpenSSL; they are used when building OpenSSL from source in the benchmark environment.

• Potential minor mismatch to check: Dockerfile.mlkem-comparison default ARG OPENSSL_VERSION is 3.5.3 while some scripts target 3.5.4 — confirm the intended test target and align ARGs/variables.

---

Files added / important changes (high level)

Added

• docker/Dockerfile.mlkem-comparison — reproducible Docker build + bench runner
• scripts/test-mlkem-hypothesis.sh — E2E benchmark + analysis script
• src/mlkem_comparison_bench.sh — lightweight bench runner used by Docker
• docs/ECDH_VS_ECDSA_CLARIFICATION.md — detailed clarification and real-world guidance
• .cursorrules — developer/project guidance

Changed

• package.json — version bumped to 1.0.14; added npm script test:mlkem-hypothesis
• config/versions.json — patched OpenSSL targets, added 3.6.0, iterations set to 1
• scripts/generate-report.js — updated version metadata and report text
• scripts/generate-viz.js, scripts/generate-viz-multipage.js — template/copy tweaks, iteration handling
• Many generated results files (results/.html, results/.json) — regenerated to reflect new versions, copy, and generation date

Note: Many results/ files are regenerated artifacts — treat them as outputs of the updated generator rather than hand-edited source.

---

How to run the new hypothesis test (examples)

• Local quick run (builds in /tmp):

  npm run test:mlkem-hypothesis

  The script will:

  • download the specified OpenSSL tarball (default: 3.5.4 in the script),
  • build three variants (default, no-ml-kem),
  • run s_time handshake benchmarks for new and resumed connections,
  • write results to /tmp/openssl-mlkem-test/results.csv and print a small analysis.

• Reproducible containerized run:

  docker build -f docker/Dockerfile.mlkem-comparison -t openssl-mlkem-test --build-arg OPENSSL_VERSION=3.5.4 .
  docker run --rm --cap-add=SYS_PTRACE openssl-mlkem-test

  (Adjust ARGs to your desired OpenSSL version and confirm build-time flags are acceptable in your environment.)

---

Review checklist for maintainers / integrators

• Confirm the OpenSSL URLs (config/versions.json and generate-report.js) point to the intended release artifacts.
• Decide whether the default iterations: 1 in config/versions.json is acceptable for CI/regression testing, or revert to a higher default for lower variance.
• Verify Dockerfile ARG defaults and scripts use the same OpenSSL version if you expect deterministic tests (e.g., align 3.5.3 vs 3.5.4).
• Ensure CI/release pipelines rely on package.json version (1.0.14) and not on any generated node_modules artifacts.
• If you run the hypothesis tests in ...

Benchmark Run 20697179374

Automated benchmark run.

See attached REPORT.md and visualizations.html for details.

1.0.13 — Memory measurement, PQC comparisons, visualization multipage + OpenSSL compatibility fixes

Summary

This release (1.0.13) adds handshake memory measurement, improves Post‑Quantum Cryptography (PQC) visualizations and documentation, fixes parsing and TLS‑1.2 invocation issues for older/newer OpenSSL versions, and improves GitHub Actions packaging so the full multi‑page report deploys correctly.

Highlights:

• Add RAM (RSS) measurement during handshake tests and a dedicated Memory Consumption report page (memory.html).
• Add PQC context and comparison content (ML‑KEM vs ECDH) and ensure pqc.html is deployed.
• Fix RSA sign/verify parsing for OpenSSL 3.2+ and restore TLS 1.2 tests on OpenSSL 1.1.1 by omitting unsupported flags.
• Ensure all generated HTML pages are included in the release artifact and GitHub Pages packaging.

---

What this release solves

• Observability: captures and surfaces resident memory (RSS) used by the OpenSSL s_server process during handshake tests so memory impact across versions (1.1.1 vs 3.x) can be compared.
• Visibility: multi‑page visualizations (overview, pqc.html, memory.html, etc.) are now packaged and deployed instead of just a single visualizations.html file, so PQC and memory pages are visible on GitHub Pages.
• Compatibility: TLS 1.2 tests no longer produce zero metrics on OpenSSL 1.1.1 because the benchmark script omits the unsupported -tls1_2 flag for that version and relies on cipher negotiation.
• Correctness: RSA sign/verify metrics were mis‑parsed on OpenSSL 3.2+; the parser now explicitly extracts sign/verify fields and prevents swapped metrics.
• Reproducibility/testing: adds quick and full memory test suites and documentation describing measurement methodology and troubleshooting.

---

Major changes (developer & user impact)

1. Handshake memory measurement

• New helper: src/measure_memory.sh — samples VmRSS via /proc/<pid>/status, averages samples and returns average RSS in KB.
• Integrated into src/benchmark.sh so handshake tests write memory metrics into the results JSON.
• New visual report page: results/memory.html (generated by visualization scripts) with grouped charts and TLS 1.2 vs TLS 1.3 comparisons.
• Documentation: docs/MEMORY_MEASUREMENT.md explains methodology, interpretation, troubleshooting, and platform limitations.
• Test tools: test-memory-quick.sh (quick checks) and test-memory-complete.sh (comprehensive validation) to validate measurement, integration and viz generation.

New JSON metric names added (examples):

• tls1_3_rsa_new_memory_kb
• tls1_3_rsa_resume_memory_kb
• tls1_3_ecdsa_new_memory_kb
• tls1_3_ecdsa_resume_memory_kb
• tls1_2_ecdhe_rsa_memory_kb
• tls1_2_rsa_resume_memory_kb
• tls1_2_ecdhe_ecdsa_memory_kb
• tls1_2_ecdsa_resume_memory_kb

Impact and considerations:

• Memory measurement requires Linux /proc to obtain VmRSS — on macOS/BSD these values will be 0 (fallback behavior exists). The tests and docs explain this and recommend running quick tests on Linux CI.
• bc is used by the measurement script; ensure bc is available in environments that run the full memory tests.
• The benchmark script attempts to chmod +x ./measure_memory.sh at runtime, but maintainers should ensure the script is executable in repositories/pipelines.

2. PQC (Post‑Quantum) visualizations and docs

• New docs: docs/PQC_CONTEXT.md, docs/PQC_PAGE_OVERVIEW.md, and docs/PQC_VISUALIZATION_FIX.md providing background on ML‑KEM vs ECDH, performance tradeoffs, and the visualization fix history.
• Multi‑algorithm comparison added to PQC page: ML‑KEM‑768 vs ECDH P‑256/P‑384, with explanatory content and migration guidance.
• results/pqc.html regenerated to include the new comparison and explanatory content.

3. Visualization generator and multi‑page reports

• scripts/generate-viz-multipage.js and scripts/generate-viz.js updated to consume memory metrics safely (fall back when missing), generate memory.html, and add explanatory content (TLS comparison table) to the dashboard.
• results/index.html updated to link to the new memory and PQC pages and to improve discoverability.
• The generators tolerate older result files that lack memory keys (they show an explanatory fallback message), preserving backward compatibility for existing summary.json files.

4. Benchmark robustness and OpenSSL compatibility

• src/benchmark.sh:

  • Integrates memory sampling around each handshake test, storing memory metrics into JSON.
  • Adds conditional invocation for TLS 1.2 tests so OpenSSL 1.1.1 systems (which don't accept -tls1_2) still produce correct metrics — avoids zeroed TLS 1.2 metrics.
  • Makes measure_memory.sh executable at runtime as a defensive step.

• Parsing fix for asymmetric throughput parsing: parse_asymmetric() now explicitly extracts fields 6 and 7 for sign/s and verify/s to handle OpenSSL 3.2+ output which added encrypt/decrypt columns; this prevents sign/verify values being swapped.

5. CI / GitHub Actions / deployment changes

• .github/workflows/benchmark.yml:
  • Copies all generated HTML files (results/*.html) and detailed iterations JSON to the release package so multi‑page reports are included in the release artifact uploaded to Pages.
  • Adds more logging and a safer fallback index.html (redirects to overview.html) only if the multipage generator fails to create an index.
  • Improves listing/logging of files before upload to help diagnose missing pages.

This fixes a long‑standing problem where only a single page (visualizations.html) was packaged and deployed, causing other pages (pqc.html, memory.html, etc.) to be missing from the published site.

6. Housekeeping and docs consolidation

• Bumped package.json version to 1.0.13 and regenerated lock artifacts for the release. (Commit included some regenerated node_modules artifacts — consider ignoring/removing node_modules if your policy disallows committing generated files.)
• Removed many ad‑hoc summary files that were replaced by consolidated docs and release notes (e.g., CHANGES_SUMMARY.txt, DOCS_INDEX.md, NEXT_STEPS.md, etc.). The important content was consolidated into docs/ and docs/RELEASE_NOTES_1.0.13.md.

---

Files added / changed (not exhaustive)

Added:

• src/measure_memory.sh (memory measurement helper, executable)
• test-memory-quick.sh, test-memory-complete.sh (validation suites)
• docs/MEMORY_MEASUREMENT.md, docs/PQC_CONTEXT.md, docs/PQC_PAGE_OVERVIEW.md, docs/PQC_VISUALIZATION_FIX.md, docs/RELEASE_NOTES_1.0.13.md, docs/TLS_1.2_FIX.md
• results/memory.html and regenerated results/* pages (pqc.html, overview.html, visualizations.html, etc.)

Changed (notable):

• src/benchmark.sh — memory hooks, TLS 1.2 branching for OpenSSL 1.1.1, and JSON metric writes
• scripts/generate-viz-multipage.js and scripts/generate-viz.js — memory/ TLS comparison rendering and safer handling of missing metrics
• .github/workflows/benchmark.yml — packaging & deployment logic
• package.json — version bumped to 1.0.13
• parse_asymmetric behavior fixed in src/benchmark.sh

Removed (cleaned up):

• Several ad‑hoc docs and summaries replaced by consolidated docs (see commit for full list).

---

Impact on users

• New: you can now view memory consumption across OpenSSL versions in the generated reports (results/memory.html) to help migration and sizing decisions.
• PQC: the PQC page now explains ML‑KEM vs ECDH and includes side‑by‑side performance comparisons (requires OpenSSL 3.5+ to collect ML‑KEM measurements).
• Reports: multi‑page reports are now packaged and deployable to GitHub Pages; if you publish the benchmark results, expect pqc.html, memory.html, overview.html, etc. to be available.
• Backward compatibility: old result files without memory metrics still work; visualizers will show an explanatory message rather than failing.

---

Impact on maintainers / developers

• New metrics are written to results JSON. If you have downstream tooling that consumes summary.json, be aware of the additional *_memory_kb keys. They are optional; visualization code handles missing values.
• The memory measurement approach depends on Linux /proc and bc in the runner environment — CI pipelines should run memory tests on Linux runners and ensure bc is installed.
• The benchmark script now conditionally omits -tls1_2 for OpenSSL 1.1.1 compatibility. If you test locally with older OpenSSL, you should re‑run benchmarks to repopulate TLS 1.2 metrics.
• The RSA parsing change fixes swapped sign/verify values with OpenSSL 3.2+. If you are comparing historic parsed values across versions, be aware earlier commits may have had swapped RSA sign/verify metrics for OpenSSL 3.2+ runs.
• CI packaging: workflow now copies all results/*.html into the release pack. If you consume the artifact programmatically, expect the full multi‑page set.

---

Upgrade / migration notes

1. Pull the code, then run:

   • npm install (if you use node tooling locally)
   • npm run report (or node scripts/generate-viz-multipage.js) to regenerate visualizations from your results/summary.json files.

2. If you want memory measurements:

   • Run full benchmark on a Linux runner (CI or local) where /proc is available and bc is installed.
   • Run the quick validation: ./test-memory-quick.sh (should be run on Linux).
   • For full verification run ./test-memory-complete.sh.

3. CI: ensure benchmark workflow runs on Linux and that the runner has bc installed if memory tests are included in the run.

4. Node / generators: new pages (memory.html, pqc.html) are generated by existing viz scripts; run npm run report or the appropriate Node script to regenerate them after updating results.

---

Breaking changes & important considerations

• No API breaking changes to the benchmark data format that remove existing keys. The release adds memory keys and improves robustness. Visualizers will fall back when memory keys are missing.
• Platform limitation: memory measurement is Linux‑specific (reads /proc/<pid>/status); macOS/BSD will...

{

"RELEASE_NOTES": {
"title": "v1.0.11 — Conditional ML-KEM benchmark compilation; release/version bump",
"body": "Overview

• This release bumps the package version to 1.0.11 and introduces a safe build-time change to the Docker image: the ML-KEM benchmark (mlkem_bench) is now only compiled when OpenSSL 3.x is available. No other functional/source-code changes were made.

What changed

1. Docker: only compile ML-KEM benchmark when OpenSSL 3.x is present

• Previously the Dockerfile always attempted to compile src/mlkem_bench.c during image build. That could fail when the image is built against OpenSSL releases that do not provide the ML-KEM APIs.
• The Dockerfile's compile step now checks the OPENSSL_VERSION environment variable and:
  • compiles mlkem_bench only if OPENSSL_VERSION starts with "3." (OpenSSL 3.x), and
  • otherwise logs a message and skips compilation.

Why this matters

• Prevents build-time failures on images that use OpenSSL < 3.x (or where OPENSSL_VERSION does not indicate a 3.x release).
• Avoids producing a broken or non-functional mlkem_bench binary when the necessary OpenSSL APIs are unavailable.

Impact and migration notes

• For users running or building the Docker image:

  • The mlkem_bench binary may be absent from the final image when OpenSSL < 3.x or when OPENSSL_VERSION is unset/not matching "3.". Build scripts or runtime callers that assume mlkem_bench is always present should handle its absence (for example, check for the executable before invoking it).
  • If you need mlkem_bench in the image, ensure the build environment exposes OPENSSL_VERSION beginning with "3." (i.e., OpenSSL 3.x). How you set that depends on your CI/build pipeline (set the environment variable earlier in the Dockerfile or map a build argument into the environment). If you prefer a stricter policy (error when OPENSSL_VERSION is unset or not 3.x), update the Dockerfile to fail instead of skipping.

• For developers:

  • No API or source-code behavior changes besides the Dockerfile behavior described above.
  • Unit tests and runtime files are unchanged; only the compile step is conditional.

2. Package version bump to 1.0.11

• package.json version updated from 1.0.10 to 1.0.11 to mark a non-dev release.
• There were no functional/source-code changes associated with this version bump.

Repository hygiene note

• The release commit included regenerated artifacts under node_modules (node_modules/.package-lock.json and node_modules/.vite/vitest/results.json). These are generated/test-run artifacts and are typically not tracked in source control.
• If these files were committed accidentally, consider removing them from the repo and adding/updating .gitignore rules to prevent future commits. For example, maintainers can remove them with git rm --cached and commit the change.

Breaking changes

• There are no breaking API changes.
• Behavioral change to be aware of: the mlkem_bench binary may not be produced on image builds that do not have OpenSSL 3.x (or where OPENSSL_VERSION does not indicate 3.x). Treat this as a deliberate safety change (avoids build failures), but update any automation that assumed the binary was always present.

Practical examples and checks

• To detect presence of the benchmark in a built image before attempting to run it:

  • check for existence and execute permission: [ -x /benchmark/mlkem_bench ] && /benchmark/mlkem_bench || echo "mlkem_bench not present"

• To ensure compilation happens, make sure the build environment exposes OPENSSL_VERSION beginning with 3. (Exact method depends on your Dockerfile and CI; ensure the shell that runs RUN sees the variable.)

Summary

This release is primarily a maintenance/versioning release (1.0.11) with a single, practical Docker improvement: the ML-KEM benchmark build is now conditional on OpenSSL 3.x being present. That prevents needless build failures and avoids shipping a broken benchmark binary when the required OpenSSL APIs are unavailable. There are no source/API breaking changes, but consumers should handle the potential absence of mlkem_bench in images built without OpenSSL 3.x.
"
}
}

v1.0.10 — Improved visualizations, complete asymmetric crypto metrics, ML‑KEM PQC benchmark, more iterations, docs & report tweaks

Summary

This release focuses on visualization robustness, completeness of benchmark metrics, improved reporting, and adding a simple ML‑KEM (post‑quantum) micro‑benchmark. The most visible user changes are clearer and properly scaled charts (small differences are now visible), regenerated result pages that include RSA/ECDSA metrics, and documentation that explains handshake metric naming. Internally: benchmark iterations were increased for better statistics, the PQC benchmark tool was added and wired into the benchmark script and Dockerfile, and report/viz generators were hardened to handle missing data.

Why this matters

• Charts now show small performance differences (0.1%–1%) clearly — no more “collapsed” or empty charts when data is present.
• Older/incomplete result files that lacked RSA/ECDSA metrics no longer cause broken charts; when data is missing the UI shows a clear message explaining what to run to generate it.
• Benchmarks now run with more iterations by default so means and stddevs are more reliable (but runs take longer).
• Basic ML‑KEM (ML‑KEM‑768) microbenchmark is included and can be measured when OpenSSL + tool support exists.

What changed (grouped)

1. Visualization & charting

• Small Multiples improvements (scripts/generate-viz.js and scripts/generate-viz-multipage.js):
  • Dynamic Y‑axis auto‑scaling based on actual data range (with padding) instead of a fixed ±10% domain.
  • Increased chart/card heights and margins for better label placement.
  • Prominent percentage labels on bars (one decimal place), color coded (green/red), and a dashed zero reference line.
  • TLS chart width/padding fix to prevent SVG overflow and cramped slope lines (getWidth usage corrected).
  • Added table and explanation areas for block size sensitivity and other helpful UI content.
• When a group of metrics is entirely missing (e.g., RSA/ECDSA), visualizations now show a descriptive message rather than rendering a zero domain.
• Multi‑page generator now conditionally generates the Mráz optimization page only when optimized data is present and exposes hasOptimizedData to navigation.

Files of interest: scripts/generate-viz.js, scripts/generate-viz-multipage.js, results/visualizations.html (regenerated), results/* (regenerated pages)

2. Missing asymmetric-crypto metrics: investigation + results

• Root cause documented: some committed results were from an earlier run that did not capture RSA/ECDSA metrics.
• Result JSON files under results/ were updated to include RSA (sign/verify) and ECDSA (sign/verify) metrics so those charts render correctly.
• Investigation and next‑steps docs added: INVESTIGATION_RESULTS.md and NEXT_STEPS.md describing how to re-run benchmarks and why the data was missing.

Impact: If you see empty RSA/ECDSA charts on GitHub Pages, re-run the full benchmark suite (see Actions below) to populate results and regenerate the pages.

3. Benchmark iterations and statistics

• Default iterations increased from 3 → 10 (config/versions.json).
  • Benefit: better statistical confidence for metrics (mean ± stddev).
  • Cost: benchmark runs will take longer (~3× more wall time unless parallelized/CI adjusted).
• generate-report.js: standard deviation / ± display is now conditional on iterationCount > 1 and report text expanded with more insight into block-size behavior.

4. PQC (ML‑KEM) microbenchmark support

• Added a small C benchmark tool: src/mlkem_bench.c.
• Dockerfile: compiles mlkem_bench into the container so it can be used during benchmark runs.
• Benchmark script (src/benchmark.sh): detects and runs the custom mlkem_bench when present and parses its output to set metrics.ml_kem_768_ops_sec. If the tool isn’t present or parsing fails the metric is set to 0.

5. Benchmark script hardening and metric naming

• src/benchmark.sh: more robust parsing (many metrics now set defensively to 0 if empty), improved warnings when tests return empty values, and uses explicit tls1_3_* and tls1_2_* metric keys alongside deprecated legacy names.
• README and docs: clarified that legacy metrics handshakes_new_per_sec and handshakes_resume_per_sec are actually TLS 1.3 RSA handshake metrics and are deprecated — explicit metric names are recommended (e.g. tls1_3_rsa_new_cps, tls1_3_rsa_resume_cps).

6. Docs and developer guidance

• Many new or updated docs describing the chart improvements, TLS chart width fix, investigation findings, and how to regenerate results and visualizations (docs/SMALL_MULTIPLES_IMPROVEMENTS.md, CHART_IMPROVEMENTS_SUMMARY.md, VIEW_IMPROVEMENTS.md, YOUR_CHART_PREVIEW.md, METRIC_CLARIFICATION_CHANGES.md, INVESTIGATION_RESULTS.md, NEXT_STEPS.md, etc.).
• scripts/regenerate-from-local.sh updated: it now skips printing a mraz.html message when that file is absent.

7. Results & generated pages

• Several generated result files and HTML pages were regenerated and added to the repository under results/ and downloaded-gh-results/ (including index.html, visualizations.html, overview.html, bellingrath.html, schmatz.html, pqc.html, mraz.html when applicable, summary.json, detailed-iterations.json, REPORT.md).

8. Version bump & repository note

• package.json version bumped to 1.0.10 and committed.
• The release includes committed node_modules artifacts (node_modules/.package-lock.json and node_modules/.vite/vitest/results.json) — these look like generated/test artifacts and are typically not committed. Consider removing them from the repo history if they were added unintentionally.

Impact and compatibility

• User-visible impact

  • Charts will look different (height, label precision, auto scale). This is an improvement for readability but may change snapshots or visual tests.
  • To see RSA/ECDSA charts you may need to regenerate results by re‑running the full benchmark suite if your current results are incomplete.
  • Benchmarks will take longer by default due to iterations increasing from 3 to 10. Adjust CI/workflow timeouts or run modes if needed.
  • New PQC metric ml_kem_768_ops_sec may appear in results; consumers of summary.json should be prepared to see this new key (or 0 when unavailable).

• Developer impact

  • Visualization code changed substantially in scripts/generate-viz.js and scripts/generate-viz-multipage.js. If you have tests or automation that parse generated HTML, you may need to update them.
  • generate-report.js now conditionally shows stddev and includes expanded narrative text — report parsing should still work but tables may change when iterationCount > 1.
  • Legacy handshake metric names are still written for backward compatibility, but they are marked deprecated. Migrate any downstream analysis to the explicit tls1_3_* / tls1_2_* metric keys where appropriate.

Breaking changes and important considerations

• No intentional breaking changes to external metric names were made — legacy metrics remain present for backward compatibility. However:
  • New explicit metric keys were added (tls1_3_rsa_, tls1_2_, optimized_*, ml_kem_768_ops_sec). Consumers that expect a fixed set of metrics should verify and accept the new keys.
  • The default iterations increase (3→10) will change the runtime and statistical output (means and stddevs). CI configurations and timeout values should be checked.
• Node_modules artifacts committed in this release are likely accidental and should be removed. They are not functional changes but can bloat the repo and confuse reviewers.

Recommended actions for users and maintainers

• If you rely on up‑to‑date RSA/ECDSA data in charts: re‑run the full benchmark suite and regenerate visualizations.
  • Quick commands (from repo root):
    • Run all benchmarks: npm run benchmark # may take ~60 minutes locally depending on environment
    • Or trigger the benchmark workflow in GitHub Actions (preferred for full set): use the repository's benchmark.yml workflow
    • Aggregate & report locally: npm run aggregate:local && npm run report
    • Regenerate visualizations: node scripts/generate-viz.js (or npm run visualize if configured)
• Review CI/workflow timeouts and resource allocations to account for the iterations change.
• For downstream tooling that parses results/summary.json, add handling for new metrics (tls1_3_, optimized_, ml_kem_768_ops_sec) and tolerate metrics being zero when not present.
• Remove node_modules artifacts from version control if they were committed accidentally (recommended):
  • git rm --cached node_modules/.package-lock.json node_modules/.vite/vitest/results.json
  • Add an exception to .gitignore if appropriate and re-commit.

Developer notes (where to review)

• Visualization logic: scripts/generate-viz.js and scripts/generate-viz-multipage.js
• Missing data handling and rendering: renderGroupedBarChart / grouped render functions in the above scripts
• TLS metric naming & deprecation notes: METRIC_CLARIFICATION_CHANGES.md and README.md additions
• Benchmark script and PQC: src/benchmark.sh and src/mlkem_bench.c; Dockerfile changes compile the mlkem_bench helper
• Report generation: scripts/generate-report.js
• Result files regenerated: results/.json and results/.html (inspect these to verify no placeholder/sensitive data was left in exported HTML/JSON)

Changelog highlights (short)

• Visualizations: auto‑scaling Y axes, one‑decimal percentage labels, taller charts, zero reference line, improved TLS width handling.
• Results: RSA/ECDSA sign/verify metrics populated into result-*.json so related charts render correctly.
• Benchmark: iterations increased (3 → 10), report formatting improved, PQC support added via mlkem_bench.c and Dockerfile compilation.
• Docs: many explanatory docs and investigation notes added to explain missing data and how to re‑run benchmarks.
• Version: package.json bumped to 1.0.10; node_modules generated artifacts were committed (review/remove if accidental).

If anything in these notes needs to be expanded (commands, locations of specific lines to review, or help removing accidental a...

Benchmark Run 20642775194

Automated benchmark run.

See attached REPORT.md and visualizations.html for details.