Why Dart sha512 and js sha512 has different result?

[mayqiyue]

No description provided.

[jtmcdole]

Do you have any information to go on?

[Nico04]

I think I've got the same issue with Dart and C#...

[Nico04]

I've just tried the same code with the plugin pointycastle instead, and it work fine.
I hope you'll find a fix :)

[jtmcdole]

Can you please give us some data to go on? What did you try to hash, what was the expected outcome, what did you get instead? What version of the DartVM were you using?

[Nico04]

Can you please give us some data to go on? What did you try to hash, what was the expected outcome, what did you get instead? What version of the DartVM were you using?

Unfortunately as it is used for the auth security protocol of my client I cannot share the full code.
But here you can find a part of it

/// dart-lang/crypto method
var digest = List<int>();
for (int i = 0; i < repeatCount; i++) {
  digest = sha512.convert([...digest, ...saltedPassword]).bytes;
}

/// pointycastle method.
var sha512Digest = new SHA512Digest();

var digest = Uint8List(0);
for (int i = 0; i < repeatCount; i++) {
  digest = sha512Digest.process(Uint8List.fromList([...digest, ...saltedPassword]));
}

with repeatCount few hundreds.

I'm using last stable Flutter with Dart v2.4.0

I hope it will help you fix it, thanks for your time !

[jtmcdole]

I just imported both crypto and pointycastle into a program just like your examples:

main() {
  final int rounds = 2000;
  final saltedPassword = "now this is a strong password!".codeUnits;

  var digest = <int>[];
  print("crypto");
  for (int i = 0; i < rounds; i++) {
    digest = sha512.convert([...digest, ...saltedPassword]).bytes;
  }
  print(toHexPad(digest));

  final sha512p = SHA512Digest();
  var pointyDigest = Uint8List(0);
  print("pointy");
  for (int i = 0; i < rounds; i++) {
    pointyDigest = sha512p
        .process(Uint8List.fromList([...pointyDigest, ...saltedPassword]));
  }
  print(toHexPad(pointyDigest));
}

And as you can see the output is identical after 2000 rounds, and we're 57ms faster (64bit intel).

pub run main.dart
crypto
5f781fba196e4bacdd09ba4aa7f1b34d127bed3bfd952734085d1f16a23d5e95cc948ad7f978af6bc1cb45db127fad22f58ce049a319bf8da86d23a0aec8ef22
pointy
5f781fba196e4bacdd09ba4aa7f1b34d127bed3bfd952734085d1f16a23d5e95cc948ad7f978af6bc1cb45db127fad22f58ce049a319bf8da86d23a0aec8ef22

[jtmcdole]

I also just built a quick and dirty Flutter program... and the results are exactly the same with dart:crypto beating pointy castle 93ms to 260ms.

[jtmcdole]

Please note, I added the nist.gov's monte carlo tests for all sha-2 digests, and we match the expected outputs.

[Nico04]

Thank a lot for your time and test. I will take the time to investigate this more precisely too soon, I'll be back

[jtmcdole]

Hey, @Nico04 - Is C# Using SHA-3 512 or SHA-2 512?

[Nico04]

Not sure, but here the officiel doc :
https://docs.microsoft.com/en-us/dotnet/api/system.security.cryptography.sha512managed?view=netframework-4.8

And a post about that :
https://stackoverflow.com/questions/24098253/difference-between-hashlib-and-system-security-cryptography-hashalgorithm

Hope it'll help

[jtmcdole]

Thanks for the docs - are you using the Hashlib version or the system.security one? From the second link:

SHA-512 is SHA-2 512 bit - not the same as SHA-3 512. That said, to use it, you simply import System.Security.Cryptography - using in this case imports the namespace - making the classes inside the namespace available to your code.

We only have the SHA-2 512 version.

[Nico04]

I'm using the first one - System.Security.
So if I understand I'm using Sha-2 512 which is the same as this plugin, so it should output the same result right ?

[jtmcdole]

Yes, but I'd need more information since I believe my test code above matches your pseudocode above.

[Nico04]

I understrand, please let me few days to build some sample code.

[Nico04]

It took me few days, but here is the small sample that shows input that works (both methods return the same) and input that doens't (cryto output doesn't work, whereas PointyCastle does).

var saltedPassword = "now this is a strong password!".codeUnits;    //This works OK
saltedPassword = utf8.encode("now this is a strong password!"); //This works OK
saltedPassword = utf8.encode("AAAA{SKJSQDGQSGDKJGSDKJGSDKJQSGDKGDKJQGSDKJQGSDKJQGSKDJGQSD"); //This works OK
saltedPassword = utf8.encode("AAAA{3FXhiiyc5gGWlRrVQ2RlJ.6xj.DKvf6l0bJxqh0BzA}"); //This DOESN'T work
saltedPassword = utf8.encode("AAAA{3FXhiiyc5gGWlRrVQ2RlJDKvf6l0bJxqh0BzA}"); //This works OK
saltedPassword = utf8.encode("AAAA{3FXhiiyc5gGWlRrVQ2RlJ.6xjDKvf6l0bJxqh0BzA}"); //This works OK
saltedPassword = utf8.encode("AAAA{3FXhiiyc5gGWlRrVQ2RlJ6xj.DKvf6l0bJxqh0BzA}"); //This works OK
saltedPassword = utf8.encode("AAAA{3FXhiiyc5gGWlRrVQ.2RlJ6xj.DKvf6l0bJxqh0BzA}"); //This DOESN'T work
saltedPassword = utf8.encode("AAAA{rVQ.2RlJ6xj.DKvf6l0}"); //This works OK
saltedPassword = utf8.encode("AAAA{rFXhiiyc5gGWlVQ.2RlJ6xj.DKvf6lFXhiiyc5gGWl0}"); //This DOESN'T work
saltedPassword = utf8.encode("AAAA{rFXhiiyc5gGWlVQ.2RlJ6xj.DKvf6lFXhiiyc5gGWl0}ds54qQSD"); //This works OK

//----- Pointy Castle -----
var sha512Digest = new SHA512Digest();

var digest1 = Uint8List(0);
for (int i = 0; i < 2000; i++) {
  digest1 = sha512Digest.process(Uint8List.fromList([...digest1, ...saltedPassword]));
}

var resultPointyCastle = base64.encode(digest1);


//----- crypto.sha512 -----
var digest2 = List<int>();
for (int i = 0; i < 2000; i++) {
  digest2 = sha512.convert([...digest2, ...saltedPassword]).bytes;
}

var resultCrypto = base64.encode(digest2);


var isEqual = resultCrypto == resultPointyCastle;
print(isEqual);
print(resultPointyCastle);
print(resultCrypto);

I hope it is enought for you to find out ?
I tried to understand the patern, but I didn't found out...
Thanks

[jtmcdole]

Sorry it's taken me a while to follow up; I converted your test to the following code. This shows no errors on one loop (yay), but reports errors on round 2. Interestingly, it's only 512 that fails... that gives me a place to dig, but it is also not a good sign as 384 and 512 share the same base algorithm.

import 'dart:convert';
import 'dart:typed_data';

import 'package:ansicolor/ansicolor.dart';
import 'package:crypto/crypto.dart';
import 'package:pointycastle/digests/sha224.dart' as p;
import 'package:pointycastle/digests/sha256.dart' as p;
import 'package:pointycastle/digests/sha384.dart' as p;
import 'package:pointycastle/digests/sha512.dart' as p;
import 'package:pointycastle/api.dart' as p;

main() {
  final red = AnsiPen()..red();
  final green = AnsiPen()..green();
  final pens = {
    false: red('fail'),
    true: green('pass'),
  };

  var salts = [
    "now this is a strong password!".codeUnits, //This works OK
    utf8.encode("now this is a strong password!"), //This works OK
    utf8.encode(
        "AAAA{SKJSQDGQSGDKJGSDKJGSDKJQSGDKGDKJQGSDKJQGSDKJQGSKDJGQSD"), //This works OK
    utf8.encode(
        "AAAA{3FXhiiyc5gGWlRrVQ2RlJ.6xj.DKvf6l0bJxqh0BzA}"), //This DOESN'T work
    utf8.encode("AAAA{3FXhiiyc5gGWlRrVQ2RlJDKvf6l0bJxqh0BzA}"), //This works OK
    utf8.encode(
        "AAAA{3FXhiiyc5gGWlRrVQ2RlJ.6xjDKvf6l0bJxqh0BzA}"), //This works OK
    utf8.encode(
        "AAAA{3FXhiiyc5gGWlRrVQ2RlJ6xj.DKvf6l0bJxqh0BzA}"), //This works OK
    utf8.encode(
        "AAAA{3FXhiiyc5gGWlRrVQ.2RlJ6xj.DKvf6l0bJxqh0BzA}"), //This DOESN'T work
    utf8.encode("AAAA{rVQ.2RlJ6xj.DKvf6l0}"), //This works OK
    utf8.encode(
        "AAAA{rFXhiiyc5gGWlVQ.2RlJ6xj.DKvf6lFXhiiyc5gGWl0}"), //This DOESN'T work
    utf8.encode(
        "AAAA{rFXhiiyc5gGWlVQ.2RlJ6xj.DKvf6lFXhiiyc5gGWl0}ds54qQSD"), //This works OK
  ];
  int maxCount = 2;

  final digestors = [
    DigestTest('224', p.SHA224Digest(), sha224),
    DigestTest('256', p.SHA256Digest(), sha256),
    DigestTest('384', p.SHA384Digest(), sha384),
    DigestTest('512', p.SHA512Digest(), sha512),
  ];

  for (final digestor in digestors) {
    print("\n\nDIGEST: ${digestor.name}");
    for (final salt in salts) {
  //----- Pointy Castle -----
      var digest1 = Uint8List(0);
      for (int i = 0; i < maxCount; i++) {
        digest1 = digestor.pointy.process(Uint8List.fromList([...digest1, ...salt]));
      }

      var resultPointyCastle = base64.encode(digest1);

  //----- crypto.sha512 -----
      var digest2 = List<int>();
      for (int i = 0; i < maxCount; i++) {
        digest2 = digestor.crypto.convert([...digest2, ...salt]).bytes;
      }

      var resultCrypto = base64.encode(digest2);

      var isEqual = resultCrypto == resultPointyCastle;
      print(
          '${pens[isEqual]}: salt(${utf8.decode(salt)}):\n pointy v. crypto:'
          '\n  $resultPointyCastle\n  $resultCrypto');
    }
  }
}

class DigestTest {
  final String name;
  final p.Digest pointy;
  final Hash crypto;
  DigestTest(this.name, this.pointy, this.crypto);
}

[jtmcdole]

First hunch: Looks like it's the padding generated higher up. They account for only 224/256 needing 64bit, where 384/512 requires 128bit of space at the end.

[jtmcdole]

Yep; its ugly, but I was able to pass after hacking around only in the _finalizeData. I'll work on a patch for later this week.

[Nico04]

I've tested the patch and confirm it works (Sorry for the delay).
Moreover, it's way faster than the pointycastle one 👍
Thanks :)