Skip to content

Fix memory leak in EdDSA PrintableString handling#858

Merged
bukka merged 1 commit intosofthsm:mainfrom
olszomal:fix-eddsa-leak
May 3, 2026
Merged

Fix memory leak in EdDSA PrintableString handling#858
bukka merged 1 commit intosofthsm:mainfrom
olszomal:fix-eddsa-leak

Conversation

@olszomal
Copy link
Copy Markdown
Contributor

@olszomal olszomal commented Apr 14, 2026

Fix a small memory leak in OSSL::byteString2oid() when CKA_EC_PARAMS is decoded as ASN1_PRINTABLESTRING for EdDSA keys.

Also replace strcmp() on ASN.1 string data with length-aware memcmp() using ASN1_STRING_get0_data() and ASN1_STRING_length(), as ASN.1 strings are not guaranteed to be NUL-terminated.

For consistency, also free the temporary ASN1_OBJECT in the OID path.

Detected via Valgrind (ASN1_PRINTABLESTRING leak during C_SignInit with EdDSA keys).

Summary by CodeRabbit

  • Bug Fixes
    • Improved parsing and validation for Edwards25519/Curve25519 and Edwards448/Curve448 curve names.
    • Added null-safety and exact-length checks to prevent misidentification.
    • Ensured proper cleanup of decoded objects to improve stability and memory safety.
    • No public API changes.

@olszomal olszomal requested a review from a team as a code owner April 14, 2026 11:32
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 14, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 53986e2e-038c-46d5-a796-0bb771602d6c

📥 Commits

Reviewing files that changed from the base of the PR and between be57521 and f9c0b97.

📒 Files selected for processing (1)
  • src/lib/crypto/OSSLUtil.cpp
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/lib/crypto/OSSLUtil.cpp

📝 Walkthrough

Walkthrough

Reworks OSSL::byteString2oid() to use a local nid return value, adds null-safe, length-aware PrintableString parsing for curve names, replaces unsafe strcmp with ASN1_STRING_get0_data/ASN1_STRING_length + memcmp, and ensures ASN1/OBJ objects are freed on all paths.

Changes

Cohort / File(s) Summary
OpenSSL OID Conversion
src/lib/crypto/OSSLUtil.cpp
Introduce local nid variable and return it; decode ASN.1 OBJECT to local then convert and free; add null checks for ASN1_PRINTABLESTRING; replace strcmp with length-aware ASN1_STRING_get0_data + ASN1_STRING_length + memcmp for curve name checks (edwards25519, curve25519, edwards448, curve448); ensure ASN1_OBJECT_free and ASN1_PRINTABLESTRING_free are called in all branches.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

Suggested reviewers

  • bukka

Poem

🐰 I hopped through bytes and winding code,
I checked each curve with careful mode,
I memcmp'd lengths and freed the rest,
Tidied ASN1 to pass the test,
A tiny hop — the leak's deposed.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Fix memory leak in EdDSA PrintableString handling' directly describes the main change: addressing a memory leak in EdDSA string handling. It is concise, specific, and aligns with the primary objective of the PR.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
Review rate limit: 7/8 reviews remaining, refill in 7 minutes and 30 seconds.

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/lib/crypto/OSSLUtil.cpp (1)

211-229: Consider using else if chain for minor efficiency.

Since the curve names are mutually exclusive, using else if would skip unnecessary comparisons after a match is found.

♻️ Suggested refactor
-		if (data_len == 12 && memcmp(data, "edwards25519", data_len) == 0)
+		if (data_len == 12 && memcmp(data, "edwards25519", 12) == 0)
 		{
 			nid = EVP_PKEY_ED25519;
 		}
-
-		if (data_len == 10 && memcmp(data, "curve25519", 10) == 0)
+		else if (data_len == 10 && memcmp(data, "curve25519", 10) == 0)
 		{
 			nid = EVP_PKEY_X25519;
 		}
-
-		if (data_len == 10 && memcmp(data, "edwards448", 10) == 0)
+		else if (data_len == 10 && memcmp(data, "edwards448", 10) == 0)
 		{
 			nid = EVP_PKEY_ED448;
 		}
-
-		if (data_len == 8 && memcmp(data, "curve448", 8) == 0)
+		else if (data_len == 8 && memcmp(data, "curve448", 8) == 0)
 		{
 			nid = EVP_PKEY_X448;
 		}

Note: Also changed line 211's memcmp third argument from data_len to 12 for consistency with the other comparisons.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/lib/crypto/OSSLUtil.cpp` around lines 211 - 229, The sequence of
independent if statements comparing curve name bytes is inefficient and can
cause redundant checks; change the four separate ifs that compare data/data_len
and assign nid (currently checking "edwards25519", "curve25519", "edwards448",
"curve448") into an else-if chain so once a match is found subsequent
comparisons are skipped, and also make the first memcmp call use a fixed length
12 (instead of data_len) to match the other comparisons; update the conditions
referencing data, data_len and assign EVP_PKEY_ED25519, EVP_PKEY_X25519,
EVP_PKEY_ED448, EVP_PKEY_X448 accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@src/lib/crypto/OSSLUtil.cpp`:
- Around line 211-229: The sequence of independent if statements comparing curve
name bytes is inefficient and can cause redundant checks; change the four
separate ifs that compare data/data_len and assign nid (currently checking
"edwards25519", "curve25519", "edwards448", "curve448") into an else-if chain so
once a match is found subsequent comparisons are skipped, and also make the
first memcmp call use a fixed length 12 (instead of data_len) to match the other
comparisons; update the conditions referencing data, data_len and assign
EVP_PKEY_ED25519, EVP_PKEY_X25519, EVP_PKEY_ED448, EVP_PKEY_X448 accordingly.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 167f5af6-7c32-4e3c-952a-1aeb5443606d

📥 Commits

Reviewing files that changed from the base of the PR and between 5a3466f and f4235fa.

📒 Files selected for processing (1)
  • src/lib/crypto/OSSLUtil.cpp

Copy link
Copy Markdown
Member

@bukka bukka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch of the mem leak! Just a minor suggestion of not repeating those data_len checks which I'm not 100% sure every compiler optimizes out...

Comment thread src/lib/crypto/OSSLUtil.cpp Outdated
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/lib/crypto/OSSLUtil.cpp`:
- Around line 198-236: The branch that decodes a PrintableString into curve_name
(via d2i_ASN1_PRINTABLESTRING) frees curve_name only on matched returns, leaking
curve_name on the fall-through return nid; path; fix by mirroring the OID
branch: do not return inside each match, instead set nid to the appropriate
EVP_PKEY_* (e.g., EVP_PKEY_ED25519, EVP_PKEY_X25519, EVP_PKEY_ED448,
EVP_PKEY_X448) when memcmp matches, then after the checks call
ASN1_PRINTABLESTRING_free(curve_name) once and return nid so every exit path
frees curve_name; ensure curve_name is freed before any return and unchanged
behavior when d2i_ASN1_PRINTABLESTRING returns NULL.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 37f8c77d-e6a1-4e6b-a625-ad3016d4ebb0

📥 Commits

Reviewing files that changed from the base of the PR and between f4235fa and 799586c.

📒 Files selected for processing (1)
  • src/lib/crypto/OSSLUtil.cpp

Comment thread src/lib/crypto/OSSLUtil.cpp
@bukka
Copy link
Copy Markdown
Member

bukka commented Apr 25, 2026

Oh sorry I was just going through PR in order from the latest and merged #863 . I was thinking that it is familiar but didn't realise that you fixed the same thing. This actually still has got that curve check and slightly more optimal comparison so if you could strip what got merged and keep what's left, that would be awesome.

@olszomal
Copy link
Copy Markdown
Contributor Author

No worries. I rebased onto #863, removed the already merged parts, and kept the remaining improvements. Should be clean now.

@bukka bukka merged commit d23ea09 into softhsm:main May 3, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants