Pkcs12Kdf can stack overflow with an exceptionally large password

[vcsjones]

The following code will produce a stack overflow within Pkcs12Kdf.

using System.Security.Cryptography;
using System.Security.Cryptography.Pkcs;

Pkcs12Builder builder = new();
ReadOnlySpan<char> password = new string('0', (int.MaxValue - 255) / 2);
builder.SealWithMac(password, HashAlgorithmName.SHA512, 1);

This overflows calculating the length of P, which causes a negative value to go in to the stackalloc here:

runtime/src/libraries/Common/src/System/Security/Cryptography/Pkcs12Kdf.cs

Lines 133 to 142 in 9a0b22a

	int PLen = ((passLen - 1 + vBytes) / vBytes) * vBytes;
	// 4. Set I=S||P to be the concatenation of S and P.
	int ILen = SLen + PLen;
	Span<byte> I = stackalloc byte[0];
	byte[]? IRented = null;
	if (ILen <= 1024)
	{
	I = stackalloc byte[ILen];

This is an unusual use case to have a password that large, but we should probably have a checked arithmetic operation here.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-security, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

The following code will produce a stack overflow within Pkcs12Kdf.

using System.Security.Cryptography;
using System.Security.Cryptography.Pkcs;

Pkcs12Builder builder = new();
ReadOnlySpan<char> password = new string('0', (int.MaxValue - 255) / 2);
builder.SealWithMac(password, HashAlgorithmName.SHA512, 1);

This overflows calculating the length of P, which causes a negative value to go in to the stackalloc here:

runtime/src/libraries/Common/src/System/Security/Cryptography/Pkcs12Kdf.cs

Lines 133 to 142 in 9a0b22a

	int PLen = ((passLen - 1 + vBytes) / vBytes) * vBytes;
	// 4. Set I=S||P to be the concatenation of S and P.
	int ILen = SLen + PLen;
	Span<byte> I = stackalloc byte[0];
	byte[]? IRented = null;
	if (ILen <= 1024)
	{
	I = stackalloc byte[ILen];

This is an unusual use case to have a password that large, but we should probably have a checked arithmetic operation here.

Author:	vcsjones
Assignees:	-
Labels:	area-System.Security
Milestone:	-