Unable to load app private key on Heroku

[clstaudt]

I am trying to pass the private key of the Github app to the octomachinery code via a Heroku config var. I am getting this error:

2023-05-22T14:27:52.400901+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.11/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 957, in load_pem_private_key
2023-05-22T14:27:52.401025+00:00 app[web.1]:     return self._load_key(
2023-05-22T14:27:52.401063+00:00 app[web.1]:            ^^^^^^^^^^^^^^^
2023-05-22T14:27:52.401072+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.11/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1152, in _load_key
2023-05-22T14:27:52.401207+00:00 app[web.1]:     self._handle_key_loading_error()
2023-05-22T14:27:52.401221+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.11/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1207, in _handle_key_loading_error
2023-05-22T14:27:52.401361+00:00 app[web.1]:     raise ValueError(
2023-05-22T14:27:52.401408+00:00 app[web.1]: ValueError: ('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [<OpenSSLError(code=503841036, lib=60, reason=524556, reason_text=unsupported)>])
2023-05-22T14:27:52.593072+00:00 heroku[web.1]: Process exited with status 1
2023-05-22T14:27:52.620548+00:00 heroku[web.1]: State changed from starting to crashed

[webknjaz]

@clstaudt when you paste the private key into the env var value field, make sure to remove any trailing spaces on every line, not just beginning and end. This is the only mistake I observed in the past — people copying the key from the terminal and it becomes corrupted because with some terminals/envs, you'd be copying extra whitespaces. It's hard to spot them visually so try using Ctrl+F to highlight them on the settings page, after clicking "edit" on the respective env var.
Or it might be any other type of data corruption — there's no way to check this without knowing the exact value. Make sure to check this first. If it doesn't help, you could generate a new key and revoke the old one, then share the "broken" value here.

[webknjaz]

OTOH, I don't remember if I use Python 3.11 for any of my apps currently. So I'd try downgrading CPython and maybe the cryptography version just to verify that this isn't something related to the deps/runtime updates.

[clstaudt]

Downgrading Python to 3.10 didn't help.

[clstaudt]

@webknjaz Here is the string for the (revoked) key. I am not sure if there is a formatting issue.

-----BEGIN RSA PRIVATE KEY-----\\nMIIEowIBAAKCAQEAyQOr099a7C1EVITTobxEPIBEgNA4J1Pvsm2MF29R89rai6IR\\n5xvlV2LER0Z9K9SKgmN98h0G72C3s/PFZvgO0P7Br+OHrSIWqPfvLfMvVajDftNI\\n1JwLaFMAQux+7oImeRk6sopn4/f1T7ol7Rhuw017eijKuPbNV4FbL2EnFbRtEQZh\\nlDQZ35OU1VcexTCEhFEJ0NQWxjm1P130iXByN8h4eZ7fdqTHBFZxO6AlzYZKzjIu\\nA1Al/e9Os8FjhWtICCTpwX/M8yMlJeown96fSbKBDCgdOIzXv33PGbIyBZH7dudm\\n4MDSbf8lbW2pM+Z3/4Uku9uk0c+O3p9USc6cwwIDAQABAoIBABlvqn2cBOBf9gaP\\n7EvKDARv4rs+Hez0FR/J9vE8N+2YoxEKSCOrhwDg0Z+HTu2N7p0DQ/uwKqF95qER\\n1sfp2c4efpnoO9Hvv1hQT2bMIBBB08Mnr06gFF12K1FMpd8+funSBHh1RSI2zUby\\ncjwnXFTAaECUCtsO9/r7tnMvYwe+3Hft9asCNDruJ1LZyuBJ+5XSiWM4joR5te6d\\nkKLeb1QTs/GJHouCtnbjuze/NaclUgcfd+coL1ar5WiAsqN70XJXDK77ySxiEIYd\\na+cMg9CC5fabUSQs8atRTNS9YttjnoAU8yJVNG4L2GLsJ6KACJE+Td9/+25brTZI\\nwMHWRPECgYEA9ixDwD5C6rTG6Yy0awVAROp5n5fBJ219G90lnlSadblxq4IYcaYf\\n0XJ4pQiVi7lGBHUgoA/eVFvdDDrONNhVAsM/XbsOiWUe0lmMh1bNw/GSt3VvYXi7\\nNNEJcJBv6Lyy6CBMtXCHaonTsfvJR+KMucuyxIMc7EIqLmn5JH46t3kCgYEA0Qnp\\n8Hxg5KzXKCZ9izc441aaqE8OAQDAeeMF052K78pWrTHlsID96AfID/FmXUFiCmLG\\n8T0QwAq+A7vHZ7eZb9VJ10ikkGIsy80EC/3DMY/KY1VslhqyH6LDg4wfgkbJ1z6q\\npVwWQL26lDwBV3NBYoroiViDm7AOOPmS7EP/mxsCgYA3wu0R1ciYm5xW+M1BVK4r\\nLiWgleJlyPJG5oAH7kgSlI//9ncM/0FD0zcqwtgFD1Gh3tomB5In76sRgxyEceBH\\nwOxQkfeH/cTOhwRC/9GaKdU/AzXYkV1uaE9IFR+US3IHDJHzo2q5IkE2lr8QLmwM\\nBe0XgE+D4LTnUI/jyUgJmQKBgQCcf8iWE/hSm5gFZjOti21avlbRVUZNt+tiFXYi\\nRRtkhN65kda9ewUctt6wGPGjNbUZw3Pf3fDxZdl2YBpc6xbDpDJBLBCQS//voJC8\\nDYMaUuWZ0rWV92yNxMSNctWQKkEolbEaVcQPOjw6iGYUu6QjC44mbY8yeu9a9nZp\\nzkrKKwKBgDwJmzy+N4RHUeWPpDcOq8wf3tQr3w5fSTUI7Q0SL7yAKdmL9s97xtbQ\\nsjGFREEMe+wbUyEJcmXW+YGFinL9jStW2wrcAGkXLTg9XR7E0YesNUdPiI8+tON1\\nMxipnaxpQx4DcCD+gwpfaJF2SvNJzw8gWc4tpXBFBX7foiywd4GQ\\n-----END RSA PRIVATE KEY-----\\n

[webknjaz]

@clstaudt are those \n things actually inline? You should put an actual multiline value into the env var there. Using \n is a hack for .env files but is not substituted by a real LF when coming from actual env vars.

[webknjaz]

You should copy the value from the PEM file as is, with zero modifications. Heroku UI allows having multiline env var values.

[clstaudt]

@webknjaz Thanks, that was the issue.

(I trusted GPT4 too much, which was absolutely certain that the line breaks must be escaped and even generated an awk command to do it...)

[webknjaz]

Yeah, that depends on the env. From what I saw, things like Heroku/openshift/k8s are able to accept non-escaped values. And things that read from dotenv or other text file format, may need escaping and that escaping may differ depending on the underlying lib. Some do need escapes, others need EOL escaped but not literals, some need to just be quoted. I've been meaning to replace the underlying library for dotenv for a while and I believe the new lib will need quoting and less escaping.

[clstaudt]

Strange times for coding productivity: I spent about as much time on debugging this line break issue as on building and deploying my first minimal Github bot using octomachinery. Which now works, by the way. Thanks to ChatGPT4 for suggesting this library.

[webknjaz]

Thanks for sharing the source of your frustrations! Maybe, I'll be able to inject some heuristics, plus warnings with suggestions into the private key loading code when I get to refactoring…

I keep forgetting that ML-driven development is a thing, but I remembered this meme from months ago:

I wonder if Anthropic's Claude would do better — on PyCon, Zac suggested that it should be more accurate in what it outputs.

[clstaudt]

There is some truth to the meme but in general GPT4 is performing impressively well. This is one of the rare cases where it led me astray and was unable to spot its own mistake. Very interested in evaluating other models for this project of mine (https://github.com/trIAgelab/trIAge).