Skip to content

{

Choose a tag to compare

View all tags
@tobrien tobrien released this 04 Jan 18:21
· 5 commits to working since this release
v1.0.14
3346d29
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

"RELEASE_NOTES": {
"title": "v1.0.14 — ML‑KEM hypothesis tooling, OpenSSL target updates, reports regenerated, docs clarification",
"body": "## Summary

This release marks a production version bump to 1.0.14 and delivers three focused updates:

  • New reproducible tooling (scripts + Docker) to test the hypothesis that OpenSSL 3.5.x ML‑KEM default keyshare behavior causes TLS handshake slowdowns.
  • Updated OpenSSL target versions (patch bumps) and the addition of 3.6.0 to config/versions.json; iteration defaults for quick runs changed to 1.
  • Regenerated dashboard and JSON report artifacts and a new ECDH vs ECDSA clarification document that explains how key-exchange and signature benchmarks differ (bandwidth/latency analysis, real-world guidance).

Other smaller changes remove emoji/cosmetic UI text in generated dashboards and add developer helper rules (.cursorrules).

Why this matters

  • The new ML‑KEM hypothesis tooling gives maintainers and users a reproducible, automated way to verify whether ML‑KEM (the post‑quantum key encapsulation used as a key‑exchange primitive) is the primary cause of the handshake regression observed in some 3.5.x OpenSSL builds. That tooling includes a ready Dockerfile for consistent builds and small benchmark wrappers.
  • Updating the OpenSSL target versions and the report generator metadata keeps the benchmark suite aligned with recent OpenSSL patch releases and adds an explicit 3.6.0 target for analysis.
  • The ECDH/ECDSA clarification reduces confusion for readers of the site and clearly separates key-exchange (ECDH/ML‑KEM) benchmarking from signature (ECDSA/ML‑DSA) benchmarking — including practical bandwidth and latency implications and mitigation strategies.

Highlights (what changed)

  1. Version bump

    • package.json: 1.0.13 → 1.0.14. This is the meaningful release version change; other node_modules file changes included are generated artifacts and not semantically significant.

  2. New ML‑KEM hypothesis tooling

    • scripts/test-mlkem-hypothesis.sh — end‑to‑end script that builds OpenSSL variants and runs s_time-based handshake benchmarks to compare default vs ECDH-only vs no-ml-kem builds. Adds a simple CSV summary and an analysis step.
    • src/mlkem_comparison_bench.sh — lightweight bench runner used by the Docker image; outputs JSON-style results for programmatic parsing.
    • docker/Dockerfile.mlkem-comparison — Docker image that builds two OpenSSL variants (default and no-ml-kem) and runs the bench script. ARGs available for OPENSSL_VERSION/URL and it installs build dependencies used in CI/local testing.
    • package.json: added npm script test:mlkem-hypothesis -> runs scripts/test-mlkem-hypothesis.sh.

  3. Configuration and metadata updates

    • config/versions.json: patch upgrades for multiple series and addition of 3.6.0. The default iterations value was reduced from 10 → 1 to allow quick runs. (This affects generated report iteration counts.)
    • scripts/generate-report.js: added metadata for the bumped versions (3.0.18, 3.1.8, 3.2.6, 3.3.5, 3.4.3, 3.5.4) and new 3.6.0 entry; updated narrative text and generated report header to reference 3.6.0.

  4. Documentation / clarification

    • docs/ECDH_VS_ECDSA_CLARIFICATION.md: long-form doc explaining distinction between ECDH (key exchange) and ECDSA (signatures), real‑world impact, bandwidth & latency math, mitigation strategies, and recommended measurement practices.
    • .cursorrules: developer helper rules / project conventions added (note: includes rules like not changing package.json version without maintainer approval).

  5. Generated report/dashboard updates

    • results/* artifacts regenerated: updated index.html, overview.html, pqc.html, tls‑comparison.html, openssl_version_analysis.html, memory.html, schmatz.html, plus summary.json and detailed-iterations.json. Notable UI changes: removed emoji, added a "Last run" banner and generation date, iteration text updated to reflect new iterations count and version list.
    • scripts/generate-viz.js and scripts/generate-viz-multipage.js: small text/template and iteration-count handling changes (removed emoji and adjusted copy). Console output messages also simplified.

  6. Small cosmetic and tooling updates

    • Many generated files in results/ updated (copy/text/formatting) to reflect new versions and the latest generation date (2026-01-04).

Impact and migration notes

  • Running the benchmark / reproduction of results

    • Quick local run: npm run test:mlkem-hypothesis (calls scripts/test-mlkem-hypothesis.sh). The script builds OpenSSL in /tmp, runs s_time benchmarks, and writes results to /tmp/openssl-mlkem-test/results.csv.
    • Docker: use docker/Dockerfile.mlkem-comparison to create a container that builds the two OpenSSL variants and runs the bundled benchmark script. This provides a consistent environment across machines and CI.

  • Dependencies and environment considerations

    • The build scripts and Dockerfile assume a typical Debian build toolchain and that openssl apps include s_time and s_server. Dockerfile installs build-essential, curl, perl, zlib1g-dev, jq, bc, etc.
    • The build flags used in scripts/Dockerfile: enable-ktls, enable-ec_nistp_64_gcc_128 and no-ml-kem (for the no-ml-kem build). Confirm your toolchain and target platform support these options.

  • Reports and iteration counts

    • config/versions.json iteration change (10 → 1) reduces the default iteration count used by the generator — results in quicker runs but higher variance. If you need reproducible/low-variance data, set iterations back to a higher value before generating results.

  • Version/URL checks

    • The release includes updated OpenSSL version strings and URLs. Reviewers should confirm those URLs match the intended OpenSSL release artifacts before running automated builds.
    • There are a couple of places where default OPENSSL_VERSION values differ between scripts and Docker ARGs (e.g., Dockerfile ARG default is 3.5.3, test scripts use 3.5.4). Verify the concrete version you intend to test and update ARGs/variables if needed.

  • Generated artifacts in node_modules

    • This commit includes regenerated generated files under node_modules (lock/test-results). They are environment artifacts and can generally be ignored for semantic review. CI/release pipelines should rely on package.json for the release version (1.0.14).

Breaking changes & important considerations

  • No runtime library or public API breaking changes were introduced in this repository. There are no changes that directly break consumers of any published library API.

  • Behavior/configuration changes to be aware of:

    • Default iterations changed to 1 in config/versions.json — this changes how many iterations the viz/report generator will run by default and affects reported means/stddev. If you depend on multi-iteration results, update this value.
    • The report generator and visual templates no longer include emoji and include slightly different explanatory copy; dashboards will look different (cosmetic changes only).
    • The ML‑KEM hypothesis tooling configures specific build flags (enable-ktls, enable-ec_nistp_64_gcc_128). These are build-time flags and may require appropriate compilers and system support. They are not applied to your system OpenSSL; they are used when building OpenSSL from source in the benchmark environment.

  • Potential minor mismatch to check: Dockerfile.mlkem-comparison default ARG OPENSSL_VERSION is 3.5.3 while some scripts target 3.5.4 — confirm the intended test target and align ARGs/variables.

Files added / important changes (high level)

Added

  • docker/Dockerfile.mlkem-comparison — reproducible Docker build + bench runner
  • scripts/test-mlkem-hypothesis.sh — E2E benchmark + analysis script
  • src/mlkem_comparison_bench.sh — lightweight bench runner used by Docker
  • docs/ECDH_VS_ECDSA_CLARIFICATION.md — detailed clarification and real-world guidance
  • .cursorrules — developer/project guidance

Changed

  • package.json — version bumped to 1.0.14; added npm script test:mlkem-hypothesis
  • config/versions.json — patched OpenSSL targets, added 3.6.0, iterations set to 1
  • scripts/generate-report.js — updated version metadata and report text
  • scripts/generate-viz.js, scripts/generate-viz-multipage.js — template/copy tweaks, iteration handling
  • Many generated results files (results/.html, results/.json) — regenerated to reflect new versions, copy, and generation date

Note: Many results/ files are regenerated artifacts — treat them as outputs of the updated generator rather than hand-edited source.

How to run the new hypothesis test (examples)

  • Local quick run (builds in /tmp):

    npm run test:mlkem-hypothesis

    The script will:

    • download the specified OpenSSL tarball (default: 3.5.4 in the script),
    • build three variants (default, no-ml-kem),
    • run s_time handshake benchmarks for new and resumed connections,
    • write results to /tmp/openssl-mlkem-test/results.csv and print a small analysis.

  • Reproducible containerized run:

    docker build -f docker/Dockerfile.mlkem-comparison -t openssl-mlkem-test --build-arg OPENSSL_VERSION=3.5.4 .
    docker run --rm --cap-add=SYS_PTRACE openssl-mlkem-test

    (Adjust ARGs to your desired OpenSSL version and confirm build-time flags are acceptable in your environment.)

Review checklist for maintainers / integrators

  • Confirm the OpenSSL URLs (config/versions.json and generate-report.js) point to the intended release artifacts.
  • Decide whether the default iterations: 1 in config/versions.json is acceptable for CI/regression testing, or revert to a higher default for lower variance.
  • Verify Dockerfile ARG defaults and scripts use the same OpenSSL version if you expect deterministic tests (e.g., align 3.5.3 vs 3.5.4).
  • Ensure CI/release pipelines rely on package.json version (1.0.14) and not on any generated node_modules artifacts.
  • If you run the hypothesis tests in CI, make sure the runner/agent has enough RAM/CPU and the required build tools for building OpenSSL with the specified configure flags.

Short technical summary

  • New: ML‑KEM test harness (shell scripts + Dockerfile) to compare default vs ECDH-only vs no-ml-kem builds.
  • Updated: OpenSSL target versions bumped and 3.6.0 added; iteration default lowered to 1.
  • Added: ECDH vs ECDSA clarification doc with bandwidth/latency analysis and mitigation guidance.
  • Regenerated: dashboards and JSON results, removed emoji and added "Last run" banner and generation date.
  • Version: package.json bumped to 1.0.14 (release-ready).

If you need a shorter summary of one specific area (e.g., how to run the Docker image, or the exact files changed) say which area and I will produce it."
}
}

Assets 2
Loading