Fix VS Code extension self-signed certificate serial flake

[Adam Ratzman (adamint)]

Description

The VS Code extension unit test that repeatedly creates a self-signed certificate has been intermittently failing in new X509Certificate(...) with an OpenSSL ASN.1 INVALID_INTEGER / illegal-padding error.

The serial generator was already masking off the high bit so the value stayed positive, but it could still hand node-forge a serial with leading zero bytes. node-forge strips at most one redundant leading 00 while DER-encoding INTEGERs, so a draw with multiple leading zeroes can still leave invalid padding in the certificate serial.

The earlier 0.392% number was a leading-zero proxy, not the actual bad-DER rate. A direct probe of the node-forge encoder path saw 7 invalid encodings in 200,000 generated serials, which is in the expected ~1/65,536 range per generated certificate.

This keeps generated serials positive and non-zero in the first byte before passing them to node-forge, and adds a deterministic regression test for the malformed DER INTEGER case (80 00 01 ...) instead of relying on the probabilistic ten-certificate loop.

No dedicated issue yet; this was observed while investigating the VS Code extension unit-test failure in PR #19069 run 31222085874.

Validation:

corepack yarn run compile-tests
./node_modules/.bin/mocha --ui tdd --timeout 20000 out/test/security.test.js
corepack yarn run lint

Mutation check:

With the zero-bucket production fix reverted, the targeted regression test fails deterministically with:
Actual message: "error:068000DD:asn1 encoding routines::illegal padding"

Checklist

• Is this feature complete?
  • Yes. Ready to ship.
  • No. Follow-up changes expected.
• Are you including unit tests for the changes and scenario tests if relevant?
  • Yes
  • No
• Did you add public API?
  • Yes
    • If yes, did you have an API Review for it?
      • Yes
      • No
    • Did you add <remarks /> and <code /> elements on your triple slash comments?
      • Yes
      • No
  • No
• Does the change make any security assumptions or guarantees?
  • Yes
    • If yes, have you done a threat model and had a security review?
      • Yes
      • No
  • No

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 19176

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 19176"

[Copilot]

Pull request overview

Fixes intermittent malformed certificate serials in the VS Code extension.

Changes:

• Ensures generated serials begin with a positive, non-zero byte.
• Adds deterministic OpenSSL regression coverage.

Show a summary per file

File	Description
extension/src/utils/security.ts	Normalizes certificate serial bytes safely.
extension/src/test/security.test.ts	Tests malformed DER regression and normalization.

Review details

Tip

Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 2/2 changed files
• Comments generated: 0
• Review effort level: Balanced

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

[github-actions]

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.