Signatures: Move keys to PKCS#8 PEM Format

[bblfish]

1. Neither Java nor JS subtleCrypto come out of the box with PKCS1 PEM support. (for java see here).
2. It is not immediately obvious to work out that the PEM formats of the private key in in PKCS1 format (the spec could point that out).
3. When building an implementation it helps to have the test suite clearly map to the data in the spec (e.g. here) so as to make error reporting clearer. That is even though using the following openssl command to convert the keys work, it makes it difficult to see if an error was introduced at some point.

$ openssl pkcs8 -topk8 -inform PEM -in spec.private.pem -out private.pem -nocrypt

4. To support PKCS1 the developer has to depend on further less well tested libraries and add additional code (as I did here) leading possibly to new bugs. Furthermore, I will need to go through the same process for JavaScript now, doubling the work.

Since PKCS8 is as expressive as PKCS1 (according to this article and see this nice StackExchange answer), it would make life easier for devs if the keys were all in PKCS8 format. Unless there really is something that PKCS1 offers that PKCS8 does not?

I will try doing this using the Web Crypto API next. But there we find that importKey supports the PKCS#8 format.

[bblfish]

While I am at it. I am having trouble with verifying the signature for 3 out of 4 examples from the 17 December 2021 spec spec that I tried out in SigningHttpMessages.scala.
Currently the only one that works is the §4.3 Example using rsa-v1_5-sha256. I am able to produce and verify a signature for §3.1_Signature, Appendix_B.2.1 and Appendix_B.2.4 examples, but the signatures don't correspond to what is in the spec. That may very well be a mistake in my code, which I am working on as a Pull Request to bobcats scala and scala-js crypto library. But I just wanted to check how confident you were in your examples. I am going to re-implement next in JS, so that will give me another reference point.

[bblfish]

I am working my way through this with JS. I was able to parse the test-key-ecc-p256 and test-key-rsa keys after translating them to pkcs8 format with openssl, but am having trouble importing test-key-rsa-pss in the console of Chrome, Chrome-beta, Chrome-Dev and Safari. I guess that is in pkcs8 format already.

var pemContent = `MIIEvgIBADALBgkqhkiG9w0BAQoEggSqMIIEpgIBAAKCAQEAr4tmm3r20Wd/Pbqv
P1s2+QEtvpuRaV8Yq40gjUR8y2Rjxa6dpG2GXHbPfvMs8ct+Lh1GH45x28Rw3Ry5
3mm+oAXjyQ86OnDkZ5N8lYbggD4O3w6M6pAvLkhk95AndTrifbIFPNU8PPMO7Oyr
FAHqgDsznjPFmTOtCEcN2Z1FpWgchwuYLPL+Wokqltd11nqqzi+bJ9cvSKADYdUA
AN5WUtzdpiy6LbTgSxP7ociU4Tn0g5I6aDZJ7A8Lzo0KSyZYoA485mqcO0GVAdVw
9lq4aOT9v6d+nb4bnNkQVklLQ3fVAvJm+xdDOp9LCNCN48V2pnDOkFV6+U9nV5oy
c6XI2wIDAQABAoIBAQCUB8ip+kJiiZVKF8AqfB/aUP0jTAqOQewK1kKJ/iQCXBCq
pbo360gvdt05H5VZ/RDVkEgO2k73VSsbulqezKs8RFs2tEmU+JgTI9MeQJPWcP6X
aKy6LIYs0E2cWgp8GADgoBs8llBq0UhX0KffglIeek3n7Z6Gt4YFge2TAcW2WbN4
XfK7lupFyo6HHyWRiYHMMARQXLJeOSdTn5aMBP0PO4bQyk5ORxTUSeOciPJUFktQ
HkvGbym7KryEfwH8Tks0L7WhzyP60PL3xS9FNOJi9m+zztwYIXGDQuKM2GDsITeD
2mI2oHoPMyAD0wdI7BwSVW18p1h+jgfc4dlexKYRAoGBAOVfuiEiOchGghV5vn5N
RDNscAFnpHj1QgMr6/UG05RTgmcLfVsI1I4bSkbrIuVKviGGf7atlkROALOG/xRx
DLadgBEeNyHL5lz6ihQaFJLVQ0u3U4SB67J0YtVO3R6lXcIjBDHuY8SjYJ7Ci6Z6
vuDcoaEujnlrtUhaMxvSfcUJAoGBAMPsCHXte1uWNAqYad2WdLjPDlKtQJK1diCm
rqmB2g8QE99hDOHItjDBEdpyFBKOIP+NpVtM2KLhRajjcL9Ph8jrID6XUqikQuVi
4J9FV2m42jXMuioTT13idAILanYg8D3idvy/3isDVkON0X3UAVKrgMEne0hJpkPL
FYqgetvDAoGBAKLQ6JZMbSe0pPIJkSamQhsehgL5Rs51iX4m1z7+sYFAJfhvN3Q/
OGIHDRp6HjMUcxHpHw7U+S1TETxePwKLnLKj6hw8jnX2/nZRgWHzgVcY+sPsReRx
NJVf+Cfh6yOtznfX00p+JWOXdSY8glSSHJwRAMog+hFGW1AYdt7w80XBAoGBAImR
NUugqapgaEA8TrFxkJmngXYaAqpA0iYRA7kv3S4QavPBUGtFJHBNULzitydkNtVZ
3w6hgce0h9YThTo/nKc+OZDZbgfN9s7cQ75x0PQCAO4fx2P91Q+mDzDUVTeG30mE
t2m3S0dGe47JiJxifV9P3wNBNrZGSIF3mrORBVNDAoGBAI0QKn2Iv7Sgo4T/XjND
dl2kZTXqGAk8dOhpUiw/HdM3OGWbhHj2NdCzBliOmPyQtAr770GITWvbAI+IRYyF
S7Fnk6ZVVVHsxjtaHy1uJGFlaZzKR4AGNaUTOJMs6NadzCmGPAxNQQOCqoUjn4XR
rOjr9w349JooGXhOxbu8nOxX`

var binDerStr = window.atob(pemContent)

function str2ab(str) {
  const buf = new ArrayBuffer(str.length);
  const bufView = new Uint8Array(buf);
  for (let i = 0, strLen = str.length; i < strLen; i++) {
    bufView[i] = str.charCodeAt(i);
  }
  return buf;
}
var privArrayBuf = str2ab(binDerStr)
var log = ""
function ok(o) { log = "ok="+o }
function ko(o) { log = "ko="+o }
var keyPrm = window.crypto.subtle.importKey("pkcs8",privArrayBuf, { name: "RSA-PSS", hash: "SHA-1" }, true, ["sign"]).then(ok,ko)

I don't have trouble parsing that key with BouncyCastle...
I guess this is a little config problem, but it's difficult to work out what that could be...
Perhaps someone has an idea?

[jricher]

@bblfish Since this all has to do with the examples in the text and not the spec itself, I think we'd be fine with having the keys in multiple formats if it's useful for developers. However I'll note that in many code libraries you need to add the PEM headers of ---- BEGIN EC KEY ---- and ---- END EC KEY ---- for their compliant parsers to parse an EC key in PEM format, for example. I've been able to parse these keys in both Java and Python, and in fact all the examples are generated using a python script: https://github.com/bspk/sig-example-scripts/blob/main/http_sig_examples.py

Also, note that RSA-PSS and ECDSA are non-deterministic signatures that incorporate a source of randomness into the signature calculation. This might be why you're getting valid signatures that look different from the examples -- you shouldn't expect to replicate the exact signature values. RSA 1.5 is deterministic, and will always give this same signature for the same input (as will HMAC). This is just one of the reasons why it's not recommended in many environments. I think we can add some text in the examples sections that points this out, if that would be helpful.

[jricher]

@bblfish You can also test keys and examples at https://httpsig.org/ for another implementation. I am pretty confident in the values in the examples, but if there are in fact bugs in these implementations (which is always possible!) I'd love to know and fix them.

[bblfish]

@jricher you were right. I had been verifying the keys by string comparison, so I fixed that. I also worked out what the Java names for the various algorithms were and got nearly all the tests to run on JS and Java platforms. But there is one error left that is consistent across both, so I wrote it up here. #1876

[bblfish]

Nevertheless, I have not yet been able to import the private key test-key-rsa-pss PEM file in JS. So I looked for a way to convert it to JWK to see how that would fare.
I looked online, but none of the services worked, I looked at npm pem-jwk but that did not work, finally I found com.nimbusds in Java that did the conversion in 2 lines of code.
With this I could get signing to work:

var privJWK = {
  "p": "5V-6ISI5yEaCFXm-fk1EM2xwAWekePVCAyvr9QbTlFOCZwt9WwjUjhtKRusi5Uq-IYZ_tq2WRE4As4b_FHEMtp2AER43IcvmXPqKFBoUktVDS7dThIHrsnRi1U7dHqVdwiMEMe5jxKNgnsKLpnq-4NyhoS6OeWu1SFozG9J9xQk",
  "kty": "RSA",
  "q": "w-wIde17W5Y0Cphp3ZZ0uM8OUq1AkrV2IKauqYHaDxAT32EM4ci2MMER2nIUEo4g_42lW0zYouFFqONwv0-HyOsgPpdSqKRC5WLgn0VXabjaNcy6KhNPXeJ0AgtqdiDwPeJ2_L_eKwNWQ43RfdQBUquAwSd7SEmmQ8sViqB628M",
  "d": "lAfIqfpCYomVShfAKnwf2lD9I0wKjkHsCtZCif4kAlwQqqW6N-tIL3bdOR-VWf0Q1ZBIDtpO91UrG7pansyrPERbNrRJlPiYEyPTHkCT1nD-l2isuiyGLNBNnFoKfBgA4KAbPJZQatFIV9Cn34JSHnpN5-2ehreGBYHtkwHFtlmzeF3yu5bqRcqOhx8lkYmBzDAEUFyyXjknU5-WjAT9DzuG0MpOTkcU1EnjnIjyVBZLUB5Lxm8puyq8hH8B_E5LNC-1oc8j-tDy98UvRTTiYvZvs87cGCFxg0LijNhg7CE3g9piNqB6DzMgA9MHSOwcElVtfKdYfo4H3OHZXsSmEQ", 
  "e": "AQAB", 
  "qi": "jRAqfYi_tKCjhP9eM0N2XaRlNeoYCTx06GlSLD8d0zc4ZZuEePY10LMGWI6Y_JC0CvvvQYhNa9sAj4hFjIVLsWeTplVVUezGO1ofLW4kYWVpnMpHgAY1pRM4kyzo1p3MKYY8DE1BA4KqhSOfhdGs6Ov3Dfj0migZeE7Fu7yc7Fc", 
  "dp": "otDolkxtJ7Sk8gmRJqZCGx6GAvlGznWJfibXPv6xgUAl-G83dD84YgcNGnoeMxRzEekfDtT5LVMRPF4_AoucsqPqHDyOdfb-dlGBYfOBVxj6w-xF5HE0lV_4J-HrI63Od9fTSn4lY5d1JjyCVJIcnBEAyiD6EUZbUBh23vDzRcE", 
  "dq": "iZE1S6CpqmBoQDxOsXGQmaeBdhoCqkDSJhEDuS_dLhBq88FQa0UkcE1QvOK3J2Q21VnfDqGBx7SH1hOFOj-cpz45kNluB832ztxDvnHQ9AIA7h_HY_3VD6YPMNRVN4bfSYS3abdLR0Z7jsmInGJ9X0_fA0E2tkZIgXeas5EFU0M",
  "n": "r4tmm3r20Wd_PbqvP1s2-QEtvpuRaV8Yq40gjUR8y2Rjxa6dpG2GXHbPfvMs8ct-Lh1GH45x28Rw3Ry53mm-oAXjyQ86OnDkZ5N8lYbggD4O3w6M6pAvLkhk95AndTrifbIFPNU8PPMO7OyrFAHqgDsznjPFmTOtCEcN2Z1FpWgchwuYLPL-Wokqltd11nqqzi-bJ9cvSKADYdUAAN5WUtzdpiy6LbTgSxP7ociU4Tn0g5I6aDZJ7A8Lzo0KSyZYoA485mqcO0GVAdVw9lq4aOT9v6d-nb4bnNkQVklLQ3fVAvJm-xdDOp9LCNCN48V2pnDOkFV6-U9nV5oyc6XI2w"
}

//multline strings work a bit like rfc8792 except they don't delete whitespace following \
var sigText = `"@method": GET
"@path": /foo
"@authority": example.org
"cache-control": max-age=60, must-revalidate
"x-empty-header": \

"x-example": Example header with some whitespace.
"@signature-params": ("@method" "@path" "@authority" \
"cache-control" "x-empty-header" "x-example");created=1618884475\
;keyid="test-key-rsa-pss"`


function aBufTxt(txt) {
	let enc = new TextEncoder();
	return enc.encode(txt)
}

var jwkPrivKey = window.crypto.subtle.importKey(
	"jwk", privJWK,
	{ name: "RSA-PSS", hash: "SHA-256" },
	true, ["sign"])

var sigPromise = jwkPrivKey.then( key => 
	window.crypto.subtle.sign(
		{ name: "RSA-PSS", saltLength: 64 },
		key, aBufTxt(sigText)
	)
)

So there is something problematic with that PEM, at least insofar as it makes it very difficult to get the example to work in JS which should be one of the main platforms.

(I have not yet verified the signature in JS as I am having trouble getting verification in JS to work with a valid PSS key, even though I can create the public key with the PEM, and am able to do that Scala-JS)

[bblfish]

I think I found the problem to be with the private key. After going through this process:

1. transforming the spec key to JWK as described above
2. transforming the JWK to PEM PKCS1 using node's pem-jwk I get the following

-----BEGIN RSA PRIVATE KEY-----
MIIEpgIBAAKCAQEAr4tmm3r20Wd/PbqvP1s2+QEtvpuRaV8Yq40gjUR8y2Rjxa6d
pG2GXHbPfvMs8ct+Lh1GH45x28Rw3Ry53mm+oAXjyQ86OnDkZ5N8lYbggD4O3w6M
6pAvLkhk95AndTrifbIFPNU8PPMO7OyrFAHqgDsznjPFmTOtCEcN2Z1FpWgchwuY
LPL+Wokqltd11nqqzi+bJ9cvSKADYdUAAN5WUtzdpiy6LbTgSxP7ociU4Tn0g5I6
aDZJ7A8Lzo0KSyZYoA485mqcO0GVAdVw9lq4aOT9v6d+nb4bnNkQVklLQ3fVAvJm
+xdDOp9LCNCN48V2pnDOkFV6+U9nV5oyc6XI2wIDAQABAoIBAQCUB8ip+kJiiZVK
F8AqfB/aUP0jTAqOQewK1kKJ/iQCXBCqpbo360gvdt05H5VZ/RDVkEgO2k73VSsb
ulqezKs8RFs2tEmU+JgTI9MeQJPWcP6XaKy6LIYs0E2cWgp8GADgoBs8llBq0UhX
0KffglIeek3n7Z6Gt4YFge2TAcW2WbN4XfK7lupFyo6HHyWRiYHMMARQXLJeOSdT
n5aMBP0PO4bQyk5ORxTUSeOciPJUFktQHkvGbym7KryEfwH8Tks0L7WhzyP60PL3
xS9FNOJi9m+zztwYIXGDQuKM2GDsITeD2mI2oHoPMyAD0wdI7BwSVW18p1h+jgfc
4dlexKYRAoGBAOVfuiEiOchGghV5vn5NRDNscAFnpHj1QgMr6/UG05RTgmcLfVsI
1I4bSkbrIuVKviGGf7atlkROALOG/xRxDLadgBEeNyHL5lz6ihQaFJLVQ0u3U4SB
67J0YtVO3R6lXcIjBDHuY8SjYJ7Ci6Z6vuDcoaEujnlrtUhaMxvSfcUJAoGBAMPs
CHXte1uWNAqYad2WdLjPDlKtQJK1diCmrqmB2g8QE99hDOHItjDBEdpyFBKOIP+N
pVtM2KLhRajjcL9Ph8jrID6XUqikQuVi4J9FV2m42jXMuioTT13idAILanYg8D3i
dvy/3isDVkON0X3UAVKrgMEne0hJpkPLFYqgetvDAoGBAKLQ6JZMbSe0pPIJkSam
QhsehgL5Rs51iX4m1z7+sYFAJfhvN3Q/OGIHDRp6HjMUcxHpHw7U+S1TETxePwKL
nLKj6hw8jnX2/nZRgWHzgVcY+sPsReRxNJVf+Cfh6yOtznfX00p+JWOXdSY8glSS
HJwRAMog+hFGW1AYdt7w80XBAoGBAImRNUugqapgaEA8TrFxkJmngXYaAqpA0iYR
A7kv3S4QavPBUGtFJHBNULzitydkNtVZ3w6hgce0h9YThTo/nKc+OZDZbgfN9s7c
Q75x0PQCAO4fx2P91Q+mDzDUVTeG30mEt2m3S0dGe47JiJxifV9P3wNBNrZGSIF3
mrORBVNDAoGBAI0QKn2Iv7Sgo4T/XjNDdl2kZTXqGAk8dOhpUiw/HdM3OGWbhHj2
NdCzBliOmPyQtAr770GITWvbAI+IRYyFS7Fnk6ZVVVHsxjtaHy1uJGFlaZzKR4AG
NaUTOJMs6NadzCmGPAxNQQOCqoUjn4XRrOjr9w349JooGXhOxbu8nOxX
-----END RSA PRIVATE KEY-----

3. transforming the above using Java's Bouncy Castle to PEM I get this:

-----BEGIN PRIVATE KEY-----
MIIEwAIBADANBgkqhkiG9w0BAQEFAASCBKowggSmAgEAAoIBAQCvi2abevbRZ389
uq8/Wzb5AS2+m5FpXxirjSCNRHzLZGPFrp2kbYZcds9+8yzxy34uHUYfjnHbxHDd
HLneab6gBePJDzo6cORnk3yVhuCAPg7fDozqkC8uSGT3kCd1OuJ9sgU81Tw88w7s
7KsUAeqAOzOeM8WZM60IRw3ZnUWlaByHC5gs8v5aiSqW13XWeqrOL5sn1y9IoANh
1QAA3lZS3N2mLLottOBLE/uhyJThOfSDkjpoNknsDwvOjQpLJligDjzmapw7QZUB
1XD2Wrho5P2/p36dvhuc2RBWSUtDd9UC8mb7F0M6n0sI0I3jxXamcM6QVXr5T2dX
mjJzpcjbAgMBAAECggEBAJQHyKn6QmKJlUoXwCp8H9pQ/SNMCo5B7ArWQon+JAJc
EKqlujfrSC923TkflVn9ENWQSA7aTvdVKxu6Wp7MqzxEWza0SZT4mBMj0x5Ak9Zw
/pdorLoshizQTZxaCnwYAOCgGzyWUGrRSFfQp9+CUh56Teftnoa3hgWB7ZMBxbZZ
s3hd8ruW6kXKjocfJZGJgcwwBFBcsl45J1OflowE/Q87htDKTk5HFNRJ45yI8lQW
S1AeS8ZvKbsqvIR/AfxOSzQvtaHPI/rQ8vfFL0U04mL2b7PO3BghcYNC4ozYYOwh
N4PaYjageg8zIAPTB0jsHBJVbXynWH6OB9zh2V7EphECgYEA5V+6ISI5yEaCFXm+
fk1EM2xwAWekePVCAyvr9QbTlFOCZwt9WwjUjhtKRusi5Uq+IYZ/tq2WRE4As4b/
FHEMtp2AER43IcvmXPqKFBoUktVDS7dThIHrsnRi1U7dHqVdwiMEMe5jxKNgnsKL
pnq+4NyhoS6OeWu1SFozG9J9xQkCgYEAw+wIde17W5Y0Cphp3ZZ0uM8OUq1AkrV2
IKauqYHaDxAT32EM4ci2MMER2nIUEo4g/42lW0zYouFFqONwv0+HyOsgPpdSqKRC
5WLgn0VXabjaNcy6KhNPXeJ0AgtqdiDwPeJ2/L/eKwNWQ43RfdQBUquAwSd7SEmm
Q8sViqB628MCgYEAotDolkxtJ7Sk8gmRJqZCGx6GAvlGznWJfibXPv6xgUAl+G83
dD84YgcNGnoeMxRzEekfDtT5LVMRPF4/AoucsqPqHDyOdfb+dlGBYfOBVxj6w+xF
5HE0lV/4J+HrI63Od9fTSn4lY5d1JjyCVJIcnBEAyiD6EUZbUBh23vDzRcECgYEA
iZE1S6CpqmBoQDxOsXGQmaeBdhoCqkDSJhEDuS/dLhBq88FQa0UkcE1QvOK3J2Q2
1VnfDqGBx7SH1hOFOj+cpz45kNluB832ztxDvnHQ9AIA7h/HY/3VD6YPMNRVN4bf
SYS3abdLR0Z7jsmInGJ9X0/fA0E2tkZIgXeas5EFU0MCgYEAjRAqfYi/tKCjhP9e
M0N2XaRlNeoYCTx06GlSLD8d0zc4ZZuEePY10LMGWI6Y/JC0CvvvQYhNa9sAj4hF
jIVLsWeTplVVUezGO1ofLW4kYWVpnMpHgAY1pRM4kyzo1p3MKYY8DE1BA4KqhSOf
hdGs6Ov3Dfj0migZeE7Fu7yc7Fc=
-----END PRIVATE KEY-----

And with that all the JS crypto API tests pass. See the Java test results here.

[bblfish]

I think to understand why that final private key works with the Java Web Crypto API but not the original one from which it was derived (which worked with Bouncy-Castle) we'd need the input of someone like @dlongley .

With that public key replaced (in commit 168ec47) and a few others commits to get Continuous Integration to work, we have all the tests pass for

• Scala compiled to Java
• Scala compiled to JavaScript
  • running in Firefox Browser
  • running in Chrome Browser

Those running in the browser are a bit interleaved due to asynchronous execution, something that needs still to be fixed.