Namespace System.IO.Hashing usage with Powershell

[jaffa-bw]

i create the following powershell script to calculate the MD5 and SHA512. i use chunks to use the loop for multiple algorithms and for speed.

$path = "C:\MyLocalFolder\somefile.iso"
$stream = [IO.File]::Open($Path, [IO.FileMode]::Open, [IO.FileAccess]::Read, [IO.FileShare]::Read)
$buffer = [Byte[]]::new(10MB)
$readCount = [int] 0

$a1 = [Security.Cryptography.HashAlgorithmName]::MD5
$a2 = [Security.Cryptography.HashAlgorithmName]::SHA512
$algorithm1 = [Security.Cryptography.HashAlgorithm]::Create($a1)
$algorithm2 = [Security.Cryptography.HashAlgorithm]::Create($a2)

while (($readCount = $stream.Read($buffer, 0, $buffer.Length)) -ne 0)
{
    $null = $algorithm1.TransformBlock($buffer, 0, $readCount, $null, 0)
    $null = $algorithm2.TransformBlock($buffer, 0, $readCount, $null, 0)
}

$null = $algorithm1.TransformFinalBlock($buffer, 0,0)
$null = $algorithm2.TransformFinalBlock($buffer, 0,0)
[BitConverter]::ToString($algorithm1.Hash).Replace('-','')
[BitConverter]::ToString($algorithm2.Hash).Replace('-','')

but, i need CRC32 additionally, which is available at System.IO.Hashing. this namespace is not available at powershell like Security.Cryptography.HashAlgorithm.
https://learn.microsoft.com/en-us/dotnet/api/system.io.hashing?view=net-9.0-pp

the dll is available at nuget, but import module is not working (error with "Could not load file or assembly 'System.Memory, Version=4.0.1.2, Culture=neutral").
https://www.nuget.org/packages/System.IO.Hashing/

briefly summarized: please make namespace System.IO.Hashing usable with powershell.

[huoyaoyuan]

i create the following powershell script to calculate the MD5 and SHA512.

Note: PowerShell has built-in cmdlet for this, Get-FileHash -Algorithm MD5 etc.

If you are targeting PowerShell 7 instead of Windows PowerShell 5, you can also use the direct methods like [System.Security.Cryptography.MD5]::HashData.

the dll is available at nuget, but import module is not working (error with "Could not load file or assembly 'System.Memory, Version=4.0.1.2, Culture=neutral").

You need to bring dlls from dependent packages, shown at dependency on nuget.org.

It's also unclear about which PowerShell version you are targeting. For PowerShell 7, the correct dll to use is lib\net8.0\System.IO.Hashing.dll or lib\net9.0\System.IO.Hashing.dll, which doesn't have additional dependency.

[jaffa-bw]

i dont understand the note to the cmdlet Get-FileHash. with the mentioned cmdlet is not possible to perform TransformBlock.
the [System.Security.Cryptography.MD5]::HashData is only for the complete file, not for chunks from buffer.

1. import-module C:\dotnet-sdk-9.0.300-win-x64\shared\Microsoft.NETCore.App\9.0.5\System.Buffers.dll
   no error
2. import-module C:\dotnet-sdk-9.0.300-win-x64\shared\Microsoft.NETCore.App\9.0.5\System.Memory.dll

import-module : Could not load type 'System.Object' from assembly 'System.Private.CoreLib, Version=9.0.0.0,
Culture=neutral, PublicKeyToken=7cec85d7bea7798e' because the parent does not exist.
At line:1 char:1
+ import-module C:\xxx\xxx\Downloads\dotnet-sdk-9.0.300-win-x64\ ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Import-Module], TypeLoadException
    + FullyQualifiedErrorId : System.TypeLoadException,Microsoft.PowerShell.Commands.ImportModuleCommand

3. import-module C:\dotnet-sdk-9.0.300-win-x64\shared\Microsoft.NETCore.App\9.0.5\System.Private.CoreLib.dll

import-module : Could not load type 'System.Object' from assembly 'System.Private.CoreLib, Version=9.0.0.0,
Culture=neutral, PublicKeyToken=7cec85d7bea7798e' because the parent does not exist.
At line:1 char:1
+ import-module C:\xxx\xxx\Downloads\dotnet-sdk-9.0.300-win-x64\ ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Import-Module], TypeLoadException
    + FullyQualifiedErrorId : System.TypeLoadException,Microsoft.PowerShell.Commands.ImportModuleCommand

the import of the System.IO.Hashing.dll is not possible, due to dependencies.

[huoyaoyuan]

i dont understand the note to the cmdlet Get-FileHash. with the mentioned cmdlet is not possible to perform TransformBlock.
the [System.Security.Cryptography.MD5]::HashData is only for the complete file, not for chunks from buffer.

It's just a choice of implementation to manually manage the blocks. What's the purpose of your code? If you are calculating the hash of independent files, not combination of multiple files, then Get-FileHash is sufficient.

1. import-module C:\dotnet-sdk-9.0.300-win-x64\shared\Microsoft.NETCore.App\9.0.5\System.Buffers.dll

You should not import any thing from shared .NET SDK. They are the .NET Core version, present at C:\Program Files\WindowsApps\Microsoft.PowerShell_7.5.1.0_x64__8wekyb3d8bbwe\System.Buffers.dll and always loaded by PowerShell 7.
If you are targeting Windows PowerShell which uses .NET Framework, you must use the corresponding dll from the nuget package System.Buffers and System.Memory and their dependent packages, under the path lib\net462. The files with the same name are very different things.

[jaffa-bw]

It's just a choice of implementation to manually manage the blocks. What's the purpose of your code? If you are calculating the hash of independent files, not combination of multiple files, then Get-FileHash is sufficient.

my goal is to read part of the file into RAM, here 10MB, then calculate the hash CRC32, MD5, SHA512 and BLAKE3. i run this already with Get-FileHash -InputStream with System.IO.BufferedStream read file method which is slower and i cannot use multiple algorithms, only one by one.
[System.IO.MemoryMappedFiles.MemoryMappedFile] to load the complete file into RAM is not possible in my case, because i have big files 50-100GB each.

i run the import module with PS7 (7.3.7). not possible.

import-module C:\xxx\xxx\Downloads\system.io.hashing.9.0.5\lib\net9.0\System.IO.Hashing.dll
Import-Module: Could not load file or assembly 'System.Runtime, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

[huoyaoyuan]

i run the import module with PS7 (7.3.7). not possible.

PowerShell 7.3.x targets .NET 7. You need to use .NET 7 compatible version of library.

System.IO.Hashing 9.0.5 has ended support for .NET 7. The latest version supporting .NET 7 is System.IO.Hashing 8.0.0.

[jaffa-bw]

$PSVersionTable

Name                           Value
----                           -----
PSVersion                      5.1.26100.4061
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.26100.4061
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

System.IO.Hashing Version=9.0.5, i checked the dependencies from .net 4.6.2 and i used the dll from folder lib\net462

Import-Module C:\MyFolder\1605\bin\System.Buffers.dll
Import-Module C:\MyFolder\1605\bin\System.Numerics.Vectors.dll
Import-Module C:\MyFolder\1605\bin\System.Runtime.CompilerServices.Unsafe.dll
Import-Module C:\MyFolder\1605\bin\System.Memory.dll
Import-Module C:\MyFolder\1605\bin\System.IO.Hashing.dll

but still error

Import-Module : Die Datei oder Assembly "System.Memory, Version=4.0.1.2, Culture=neutral,
PublicKeyToken=cc7b13ffcd2ddd51" oder eine Abhängigkeit davon wurde nicht gefunden. Das System kann die angegebene
Datei nicht finden.
In Zeile:1 Zeichen:1
+ Import-Module C:\MyFolder\1605\bin\System.IO.Hashing.dll
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Import-Module], FileNotFoundException
    + FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.PowerShell.Commands.ImportModuleCommand

[huoyaoyuan]

You are still getting binary from the wrong version of System.Memory package. .NET Framework is more sensitive about assembly version, and doesn't roll over to higher version by default.

I manually extracted binaries from the following package versions:

• System.IO.Hashing 8.0.0
• System.Buffers 4.5.1
• System.Memory 4.5.5
• System.Numerics.Vectors 4.5.0
• System.Runtime.CompilerServices.Unsafe 4.5.3

Putting them under the same folder, and PowerShell 5.1 successfully load them. The correct files are having following modification time:

You can create an empty C# Exe project targeting net462, install the System.IO.Hashing package, build the application, then find the resolved dlls under output folder. NuGet will resolves the correct binaries for you, and what I was saying is doing the process manually.

For System.IO.Hashing 8.0.0, this approach works fine. However if binding redirect is involved, I'm not sure whether it can work seamlessly. It's a known pain for .NET Framework.

[huoyaoyuan]

For PowerShell 7.3.x, you can follow a similar process: first using "".GetType().Assembly.FullName to get the framework that PowerShell is compiled with (.NET 7 in this case), then create a new .NET 7 console project, install the package and let NuGet resolves the correct files for you. On PowerShell Core the confidence is higher since .NET Core is more permissive about assembly versioning.