libprio does not compile with MSVC

[rhelmer]

Most Firefox builders use Clang/LLVM nowadays but one of our builders still uses MSVC so this came up:

https://treeherder.mozilla.org/logviewer.html#?job_id=195697225&repo=mozilla-central&lineNumber=6090

10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(80): error C2057: expected constant expression
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(80): error C2466: cannot allocate an array of constant size 0
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(80): error C2133: 'key_bytes': unknown size
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(84): error C2057: expected constant expression
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(84): error C2466: cannot allocate an array of constant size 0
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(84): error C2133: 'spki_data': unknown size
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(281): error C2057: expected constant expression
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(281): error C2466: cannot allocate an array of constant size 0
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(281): error C2133: 'aadBuf': unknown size
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(319): error C2057: expected constant expression
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(319): error C2466: cannot allocate an array of constant size 0
10:05:12 INFO - z:/build/build/src/third_party/prio/prio/encrypt.c(319): error C2133: 'aad_buf': unknown size

I believe this is because MSVC doesn't support VLAs (variable length arrays), it wants the array size to be specified at compile time:

https://stackoverflow.com/questions/5246900/enabling-vlasvariable-length-arrays-in-ms-visual-c

[rhelmer]

dmajor (our "human-compiler relations" expert :P) suggests:

"given the limited users of PublicKey_import, I'd just set it to the max value currently used (CURVE25519_KEY_LEN+1?) and crash if you need anything bigger"

I think this makes sense, especially given that we call PrioEncoder::IsValidHexPublicKey() https://hg.mozilla.org/mozilla-central/file/190b827aaa2b/dom/prio/PrioEncoder.cpp#l164 before we call PublicKey_import so it's unlikely to actually crash, just throw / reject the promise from JS, if someone messes up the public key in the prefs.

[rhelmer]

Oh, I see this is also a problem in PrivateKey_decrypt and set_gcm_params.

The size for spki_data is: const int spki_len = sizeof (curve25519_spki_zeros)
For aadBuf / aad_buf it is: #define AAD_LEN (strlen (PRIO_TAG) + CURVE25519_KEY_LEN + GCM_IV_LEN_BYTES)

@henrycg, do you think we could just assign constants for these? I assume strlen and sizeof are being used here for convenience, it looks like these are known and don't change right (or if any of them do we could determine some kind of upper bound?)

[rhelmer]

Hm. So, I do think it's valuable for libprio to compile w/ MSVC so this is worth fixing, but since this is a Tier-2 platform for Firefox, and confidence is high that we will switch to Clang soon, I might just not build prio on Firefox when MSVC is in use so we don't have to rush on this.

[henrycg]

If we get libprio compiling with C90, will that be enough to ensure that it compiles with MSVC? I don't have access to a machine running MSVC, but I can patch libprio to work with C90.

[rhelmer]

@henrycg I believe C90 support is the only problem, but won't know for sure until we get past this error and see if there are any other problems :)

I am working on disabling Prio for Firefox when built with MSVC in this bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1485946

But, if it's easy enough to fix, that would be an even smaller change :)

[henrycg]

My sense is that this is no longer an issue. Let me know if MSVC support is still important.