Skip to content

1.0.13 — Memory measurement, PQC comparisons, visualization multipage + OpenSSL compatibility fixes

Choose a tag to compare

View all tags
@tobrien tobrien released this 01 Jan 17:39
· 6 commits to working since this release
v1.0.13
c83e923
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Summary

This release (1.0.13) adds handshake memory measurement, improves Post‑Quantum Cryptography (PQC) visualizations and documentation, fixes parsing and TLS‑1.2 invocation issues for older/newer OpenSSL versions, and improves GitHub Actions packaging so the full multi‑page report deploys correctly.

Highlights:

  • Add RAM (RSS) measurement during handshake tests and a dedicated Memory Consumption report page (memory.html).
  • Add PQC context and comparison content (ML‑KEM vs ECDH) and ensure pqc.html is deployed.
  • Fix RSA sign/verify parsing for OpenSSL 3.2+ and restore TLS 1.2 tests on OpenSSL 1.1.1 by omitting unsupported flags.
  • Ensure all generated HTML pages are included in the release artifact and GitHub Pages packaging.

What this release solves

  • Observability: captures and surfaces resident memory (RSS) used by the OpenSSL s_server process during handshake tests so memory impact across versions (1.1.1 vs 3.x) can be compared.
  • Visibility: multi‑page visualizations (overview, pqc.html, memory.html, etc.) are now packaged and deployed instead of just a single visualizations.html file, so PQC and memory pages are visible on GitHub Pages.
  • Compatibility: TLS 1.2 tests no longer produce zero metrics on OpenSSL 1.1.1 because the benchmark script omits the unsupported -tls1_2 flag for that version and relies on cipher negotiation.
  • Correctness: RSA sign/verify metrics were mis‑parsed on OpenSSL 3.2+; the parser now explicitly extracts sign/verify fields and prevents swapped metrics.
  • Reproducibility/testing: adds quick and full memory test suites and documentation describing measurement methodology and troubleshooting.

Major changes (developer & user impact)

  1. Handshake memory measurement
  • New helper: src/measure_memory.sh — samples VmRSS via /proc/<pid>/status, averages samples and returns average RSS in KB.
  • Integrated into src/benchmark.sh so handshake tests write memory metrics into the results JSON.
  • New visual report page: results/memory.html (generated by visualization scripts) with grouped charts and TLS 1.2 vs TLS 1.3 comparisons.
  • Documentation: docs/MEMORY_MEASUREMENT.md explains methodology, interpretation, troubleshooting, and platform limitations.
  • Test tools: test-memory-quick.sh (quick checks) and test-memory-complete.sh (comprehensive validation) to validate measurement, integration and viz generation.

New JSON metric names added (examples):

  • tls1_3_rsa_new_memory_kb
  • tls1_3_rsa_resume_memory_kb
  • tls1_3_ecdsa_new_memory_kb
  • tls1_3_ecdsa_resume_memory_kb
  • tls1_2_ecdhe_rsa_memory_kb
  • tls1_2_rsa_resume_memory_kb
  • tls1_2_ecdhe_ecdsa_memory_kb
  • tls1_2_ecdsa_resume_memory_kb

Impact and considerations:

  • Memory measurement requires Linux /proc to obtain VmRSS — on macOS/BSD these values will be 0 (fallback behavior exists). The tests and docs explain this and recommend running quick tests on Linux CI.
  • bc is used by the measurement script; ensure bc is available in environments that run the full memory tests.
  • The benchmark script attempts to chmod +x ./measure_memory.sh at runtime, but maintainers should ensure the script is executable in repositories/pipelines.
  1. PQC (Post‑Quantum) visualizations and docs
  • New docs: docs/PQC_CONTEXT.md, docs/PQC_PAGE_OVERVIEW.md, and docs/PQC_VISUALIZATION_FIX.md providing background on ML‑KEM vs ECDH, performance tradeoffs, and the visualization fix history.
  • Multi‑algorithm comparison added to PQC page: ML‑KEM‑768 vs ECDH P‑256/P‑384, with explanatory content and migration guidance.
  • results/pqc.html regenerated to include the new comparison and explanatory content.
  1. Visualization generator and multi‑page reports
  • scripts/generate-viz-multipage.js and scripts/generate-viz.js updated to consume memory metrics safely (fall back when missing), generate memory.html, and add explanatory content (TLS comparison table) to the dashboard.
  • results/index.html updated to link to the new memory and PQC pages and to improve discoverability.
  • The generators tolerate older result files that lack memory keys (they show an explanatory fallback message), preserving backward compatibility for existing summary.json files.
  1. Benchmark robustness and OpenSSL compatibility

  • src/benchmark.sh:

    • Integrates memory sampling around each handshake test, storing memory metrics into JSON.
    • Adds conditional invocation for TLS 1.2 tests so OpenSSL 1.1.1 systems (which don't accept -tls1_2) still produce correct metrics — avoids zeroed TLS 1.2 metrics.
    • Makes measure_memory.sh executable at runtime as a defensive step.

  • Parsing fix for asymmetric throughput parsing: parse_asymmetric() now explicitly extracts fields 6 and 7 for sign/s and verify/s to handle OpenSSL 3.2+ output which added encrypt/decrypt columns; this prevents sign/verify values being swapped.

  1. CI / GitHub Actions / deployment changes
  • .github/workflows/benchmark.yml:
    • Copies all generated HTML files (results/*.html) and detailed iterations JSON to the release package so multi‑page reports are included in the release artifact uploaded to Pages.
    • Adds more logging and a safer fallback index.html (redirects to overview.html) only if the multipage generator fails to create an index.
    • Improves listing/logging of files before upload to help diagnose missing pages.

This fixes a long‑standing problem where only a single page (visualizations.html) was packaged and deployed, causing other pages (pqc.html, memory.html, etc.) to be missing from the published site.

  1. Housekeeping and docs consolidation
  • Bumped package.json version to 1.0.13 and regenerated lock artifacts for the release. (Commit included some regenerated node_modules artifacts — consider ignoring/removing node_modules if your policy disallows committing generated files.)
  • Removed many ad‑hoc summary files that were replaced by consolidated docs and release notes (e.g., CHANGES_SUMMARY.txt, DOCS_INDEX.md, NEXT_STEPS.md, etc.). The important content was consolidated into docs/ and docs/RELEASE_NOTES_1.0.13.md.

Files added / changed (not exhaustive)

Added:

  • src/measure_memory.sh (memory measurement helper, executable)
  • test-memory-quick.sh, test-memory-complete.sh (validation suites)
  • docs/MEMORY_MEASUREMENT.md, docs/PQC_CONTEXT.md, docs/PQC_PAGE_OVERVIEW.md, docs/PQC_VISUALIZATION_FIX.md, docs/RELEASE_NOTES_1.0.13.md, docs/TLS_1.2_FIX.md
  • results/memory.html and regenerated results/* pages (pqc.html, overview.html, visualizations.html, etc.)

Changed (notable):

  • src/benchmark.sh — memory hooks, TLS 1.2 branching for OpenSSL 1.1.1, and JSON metric writes
  • scripts/generate-viz-multipage.js and scripts/generate-viz.js — memory/ TLS comparison rendering and safer handling of missing metrics
  • .github/workflows/benchmark.yml — packaging & deployment logic
  • package.json — version bumped to 1.0.13
  • parse_asymmetric behavior fixed in src/benchmark.sh

Removed (cleaned up):

  • Several ad‑hoc docs and summaries replaced by consolidated docs (see commit for full list).

Impact on users

  • New: you can now view memory consumption across OpenSSL versions in the generated reports (results/memory.html) to help migration and sizing decisions.
  • PQC: the PQC page now explains ML‑KEM vs ECDH and includes side‑by‑side performance comparisons (requires OpenSSL 3.5+ to collect ML‑KEM measurements).
  • Reports: multi‑page reports are now packaged and deployable to GitHub Pages; if you publish the benchmark results, expect pqc.html, memory.html, overview.html, etc. to be available.
  • Backward compatibility: old result files without memory metrics still work; visualizers will show an explanatory message rather than failing.

Impact on maintainers / developers

  • New metrics are written to results JSON. If you have downstream tooling that consumes summary.json, be aware of the additional *_memory_kb keys. They are optional; visualization code handles missing values.
  • The memory measurement approach depends on Linux /proc and bc in the runner environment — CI pipelines should run memory tests on Linux runners and ensure bc is installed.
  • The benchmark script now conditionally omits -tls1_2 for OpenSSL 1.1.1 compatibility. If you test locally with older OpenSSL, you should re‑run benchmarks to repopulate TLS 1.2 metrics.
  • The RSA parsing change fixes swapped sign/verify values with OpenSSL 3.2+. If you are comparing historic parsed values across versions, be aware earlier commits may have had swapped RSA sign/verify metrics for OpenSSL 3.2+ runs.
  • CI packaging: workflow now copies all results/*.html into the release pack. If you consume the artifact programmatically, expect the full multi‑page set.

Upgrade / migration notes

  1. Pull the code, then run:

    • npm install (if you use node tooling locally)
    • npm run report (or node scripts/generate-viz-multipage.js) to regenerate visualizations from your results/summary.json files.

  2. If you want memory measurements:

    • Run full benchmark on a Linux runner (CI or local) where /proc is available and bc is installed.
    • Run the quick validation: ./test-memory-quick.sh (should be run on Linux).
    • For full verification run ./test-memory-complete.sh.

  3. CI: ensure benchmark workflow runs on Linux and that the runner has bc installed if memory tests are included in the run.

  4. Node / generators: new pages (memory.html, pqc.html) are generated by existing viz scripts; run npm run report or the appropriate Node script to regenerate them after updating results.

Breaking changes & important considerations

  • No API breaking changes to the benchmark data format that remove existing keys. The release adds memory keys and improves robustness. Visualizers will fall back when memory keys are missing.
  • Platform limitation: memory measurement is Linux‑specific (reads /proc/<pid>/status); macOS/BSD will yield 0 memory values (fallback is documented). Be careful running memory tests on non‑Linux CI.
  • The commit includes regenerated node_modules artifacts in the repository (packaged lock/test artifacts). If your repo policy forbids committing node_modules, remove/ignore those artifacts; they are generated files and not required in source control.

Testing & review checklist (recommended)

Reviewers / maintainers should focus on:

  • src/measure_memory.sh — sampling approach, sample count/interval, assumptions about /proc, and numeric robustness.
  • src/benchmark.sh — memory hooks timing and JSON key names, the conditional logic for OpenSSL 1.1.1 TLS 1.2 tests, and the parse_asymmetric() changes (fields 6 and 7 extraction).
  • scripts/generate-viz-multipage.js and scripts/generate-viz.js — ensure they handle missing memory metrics and do not throw when a metric is absent.
  • .github/workflows/benchmark.yml — verify artifact packaging still matches your Pages deployment expectations and that the fallback index.html behavior is acceptable.
  • tests: run test-memory-quick.sh on a Linux environment and test-memory-complete.sh if you want a full validation before enabling memory collection on CI.

Notes & references

  • Release bumped to version: 1.0.13 in package.json.
  • If you see unexpected file changes in node_modules/ from the release commit, those are regenerated artifacts from lockfile tooling and can be removed if your repo policy requires it.

If you need a short list of the most relevant files to inspect: src/benchmark.sh, src/measure_memory.sh, scripts/generate-viz-multipage.js, .github/workflows/benchmark.yml, docs/MEMORY_MEASUREMENT.md, and the new test scripts (test-memory-quick.sh, test-memory-complete.sh).

Assets 2
Loading