LCS: Hash Functions Specification

[runrevmark]

This specification describes the cryptographic hash function support which is present in LiveCode Script.

The current missing hash function we need is the SHA-2 family. The SHA-3 family would be nice to add, but it is not yet officially standardized and so it seems prudent to wait for that to happen.

Given the importance of cryptographic hashes throughout IT, it seems prudent that we should back-port SHA-2 implementation to 6.7 onwards.

Rendered version

[peter-b]

I edited your comment to add a link to the rendered version (sorry)

[peter-b]

Looks great. 👍

[livecodefraser]

Should we also have a hash(algorithm, data) function? e.g. hash("sha-1", textEncode("hello", "utf8"))

[runrevmark]

@livecodefraser: That seems like a reasonable idea - it should probably be digest(...) since we have md5digest / sha1digest at the moment.

One thing I'm not 100% sure about is the 'variant' parameter to the sha2 digest - is it a string, is it an integer? Is there a better way to describe the variants?

[runrevmark]

Along the general digest route - we can hook into OpenSSL's hash functions for the general one... I.e. we have syntax for the hashes which are built into the engine; but the general one can access any which OpenSSL provides.

This would mean we should probably have a digestNames function too (like the cipherNames).

[runrevmark]

I'm wondering if 'the sha2digest of ...' is the correct syntax to use for SHA-256. Looking at OpenSSL APIs and NSS APIs, they tend to refer to it as SHA256 / SHA384 etc. - rather than SHA2.

What about if we add 'the sha256Digest of ...' and leave the rest of the family accessible through a more general 'digest' function as discussed above. It does seem that this is currently the most popular hash to use for new applications.

Of course, one issue here is that the name of the default SHA-3 variant would be when that becomes standardized - it has similar block size variants to SHA-2. We have to be careful not to introduce something which will cause confusion later on.

[peter-b]

On reflection, I think that the two-parameter form of the sha2digest might be the way to go. It's unambiguous and easily extensible to future SHA standards

[runrevmark]

@peter-b: Yes - the 2 argument form is definitely fine. However, the problem is that you can't write it in 'of' form - only functions of one parameter have the syntactic sugar 'the ... of' (for obvious reasons). The thing I'm unsure about is what name to give one or more one argument forms. It does seem most implementations have settled around sha1, sha256, sha512 etc as prefixes for the underlying APIs - i.e. ditching the relationship to SHA-2 - but that potential means we have a problem at the introduction of SHA-3.

Perhaps saying 'the sha2digest of' is sha-2 256 is fine - then we can pick the most popular variant of sha-3 for 'the sha3digest of' later.

[runrevmark]

Having reflected on this further, then I think it probably makes most sense to go with 'the Digest of' giving you the most 'popular' variant of the hash family as suggested above.

When Open Language comes a long we can make the syntax for variants much nicer:

  the sha2Digest of tVar -- 256-bit
  the 384 bit sha2Digest of tVar

Looking at how various hash functions have been 'blessed' with standard general usage over the years, it seems that generally it is a change in family that occurs, rather than an increase in bit length of an existing family (which makes sense from a security perspective - flaws discovered in one hash family will affect the whole family, the bit length only affects how hard it is to exploit it).

[livecode-vulcan]

📝 Hi @runrevmark, I haven't been able to verify that you've signed our LiveCode Contributor's Agreement.

If you have previously signed the Contributor's Agreement, I may not be able to detect it because you haven't linked your GitHub account to your LiveCode account.

Please see the information for contributors for more information.

[mwieder]

Makes sense to me. I think the 'sha256 / sha512' forms are more intuitive, but obviously lead into some problem areas for the future, so the 'sha2' form in both the single- and double-parameter forms look good to me. I do want to make sure that 'the digestNames' function doesn't get left out here, as I think that's an important addition. And we should flag the sha1digest form as deprecated asap.

[mwieder]

Also, please, we should use the functions from the openssl library and not create our own.

[montegoulding]

@runrevmark I just came across this and I have a tendency to think we should consider a mix of built in and OpenSSL based implementations. If revsecurity is available the list is appended with theirs and it is used in preference otherwise the internal implementations are used. In order to do that I think we need more flexible syntax:

• Listing supported hashes

    the hashNames | the digestNames
  

• Hashing some data

    hash(<data>, <hashName>) | digest(<data>, <digestName>)
  

I don't think there's a huge need to continue with *digest functions so md5Digest and sha1Digest could be deprecated.

This looks like another suitably licensed sha256 implementation.

[peter-b]

@montegoulding I was thinking about this overnight, and I wonder if there isn't a danger of conflating two different classes of function here. Cryptographic hash functions have properties that make them less suitable for use as fast hashes, and vice versa (I guess that SipHash might be an exception to that). Maybe we need a secureHash, providing only cryptographic hash algorithms, and hash, providing good fast hash algorithms.

/cc @livecodefraser

[montegoulding]

@peter-b I'm not sure I see the point as it's up to the scripter to know which algorithm they want/need to use. Perhaps we just need to find a page to link to in the docs that lists hash name, relative speed and relative security?