Error Parsing CA Pem File #16203
Labels
Comments
|
Hi there - what are the contents of the PEM file? Can it be read with other tools, ie |
hsimon-hashicorp
added
auth/cert
|
I was able to read it with both Unfortunately, I am unable to copy and paste the contents of the PEM. Is there anything in particular I should be looking for? Otherwise I can type out what I can from the output of the |
|
The only other thing I can think of is the environment variable may not be getting passed in correctly. The output says it can't parse the file at ./vault-ca.pem, but your environment variable has a fully-qualified directory in it. |
|
My bad. That was a typo. Corrected it. |
hsimon-hashicorp
added
storage/postgresql
auth/cert
|
Darn - I was hoping I could answer it easily for you. I'll check with engineering to see if there's something else I may have missed. Thanks! :) |
|
Would you be willing to run |
|
Here's the output I get (slightly edited to save time. I have to type it out manually). I can try and fill it in more if what I excluded was important. |
|
Hey @jnapl1 -- A program like this (suggested by my colleague @swenson) might be helpful to diagnose the problem: package main
import (
"crypto/x509"
"encoding/pem"
"fmt"
"os"
)
func main() {
data, err := os.ReadFile(os.Args[1])
if err != nil {
panic(err)
}
block, _ := pem.Decode(data)
if block == nil {
panic("Invalid PEM block")
}
cert, err := x509.ParseCertificate(block.Bytes)
if err != nil {
panic(err)
}
fmt.Printf("Read certificate okay: %+v\n", cert)
}If you run it with: it should let us know if there's an issue with the certificate or if it reads OK: E.g. for an invalid PEM file I get: But for a valid cert I get: The details of the panic should be sufficient to help identify the cause. :-) This is essentially what is being done in that program. The output comes from: https://github.com/hashicorp/go-rootcerts/blob/master/rootcerts.go#L71-L89 -- which calls https://cs.opensource.google/go/go/+/refs/tags/go1.18.3:src/crypto/x509/cert_pool.go;l=206 Seems like it might potentially be a bug in this version of Go; a newer 1.9.7 binary might not have this problem if its already been fixed. |
|
Honestly, I've never used go before. So I haven't been able to get this to run just yet. But I tried replicating the error on my personal VM, using certificates I created with openssl, so that I could share the certificates I was using and for whatever reason it worked. I think the error could be from the v3 extensions, but not sure which ones or why it would be a problem. (sorry, this is going to be a long one) These certificates worked: vault.pem These certificates failed: root2.pem vault2.pem |
To make use of the program shown earlier, all you need to do is:
That's all -
No, I don't think it did - you're showing us a different unrelated failure. Please can you show us the first and last lines of the problematic |
|
Yes, you're right. Sorry. I was in a rush to get my last post up and didn't pay enough attention to the error message. My development environment is a little restrictive, so I was limited in the version of go available to me (v1.9.4). Let me know if this is an issue. But considering that vault 1.9.3 (compiled with go v1.17) couldn't parse it, I'd guess it is the same issue. I receive the following error when running the provided go code. I'm unable to copy/paste, in case there are potential typos. First and last line appear as follows.: Let me know if more details are needed. |
|
Hey @jnapl1, So this says the PEM file is improperly formatted. You might find RFC 1421 interesting. It is surprising that OpenSSL accepts it but Go's doesn't. I wonder if its whitespace? Something like |
|
Quick update (Sorry I got a little preoccupied). So far I've been unable to duplicate my error, which is leading me to consider that our certificate provider may be formatting the certificates in a incompatible way. Certificates generated with openssl commands ran locally on the same machine hosting Vault has, so far, worked as expected. Need to perform a few more tests to confirm this though. I'll post what I find out. |
|
Out of curiosity, if you have the original cert still, could you take and convert it from PEM->DER->PEM via OpenSSL and get something Go would accept? Something like: By going to DER (binary contents of the PEM), you can ensure any issues in the outer wrapper are discarded, and that PEM wrapper recreated from scratch by OpenSSL. |
|
Unfortunately, that didn't seem to work. |
|
Iiiinteresting, and you definitely see If you can get permission from your organization, please share it with our support personnel, I'd love to see the CA cert in question to understand the problem better. |
|
Yes, I get the message on both. And I am not able to provide the certificates. |
|
It looks like the issue may have been caused by subject/authority key identifiers in the v3 extensions being SHA256 hashes. After reissuing the certificate with the values generated with SHA1, it was able to parse the new CA file specified with I haven't been able to test the postgresql connection yet, so I can't say for sure I have everything working yet. |
|
@jnapl1 That's very interesting -- I wouldn't have expected the SKID to result in a X.509 CA Chain with OpenSSL infoAs you can see, the SKID/AKID on the leaf (and on the root) both have SHA-256 hashes (32-bytes) as their SKIDs. When run on the program above, I get a valid parse: Output of `main.go` aboveIndeed, if I go even farther and hex-encode the SKID before passing it as the SKID (creating a 64-byte SKID) -- and then repeat it three times (for a total length of 192 bytes!), it still parses fine in Go: So again... without the root CA public certificate (which will not include private keys necessary to create certs -- and which can be securely sent to our Support team)... I'm unable to reproduce this on the basis of it being a AKID/SKID encoding problem. Perhaps more fruitfully, could you try loading the |
|
My bad. I had mistyped the given example. The PEM blocks are valid. Returned error: Not sure why this is the case, since the basic constraints on these certificates are the same as those with a SHA1 SKID, which are read successfully. Basic Constraints: |
|
This keeps getting more interesting... :-) Can you share just the ASN.1 of that constraint? If you run something like this: OpenSSL command to parse ASN1 structureThe first column lists byte offsets in the ASN.1 contents; If you go up one depth ( For me, that looks like: Command to extract snippetYou can verify this with: The hex is important so I can reconstruct just that portion of the certificate. ASN.1 parse is a touch more useful than the Here's the error in the Go code: func parseBasicConstraintsExtension(der cryptobyte.String) (bool, int, error) {
var isCA bool
if !der.ReadASN1(&der, cryptobyte_asn1.SEQUENCE) {
return false, 0, errors.New("x509: invalid basic constraints a")
}
if der.PeekASN1Tag(cryptobyte_asn1.BOOLEAN) {
if !der.ReadASN1Boolean(&isCA) {
return false, 0, errors.New("x509: invalid basic constraints b") // <<<< YOUR ERROR HERE
}
}
maxPathLen := -1
if !der.Empty() && der.PeekASN1Tag(cryptobyte_asn1.INTEGER) {
if !der.ReadASN1Integer(&maxPathLen) {
return false, 0, errors.New("x509: invalid basic constraints c")
}
}
// TODO: map out.MaxPathLen to 0 if it has the -1 default value? (Issue 19285)
return isCA, maxPathLen, nil
}Per RFC 5280 / Section 4.2.1.9 on Basic Constraints: What's happening is, for some reason, your certificate's ASN.1 is causing Go to fail to parse that See the implementation of // ReadASN1Boolean decodes an ASN.1 BOOLEAN and converts it to a boolean
// representation into out and advances. It reports whether the read
// was successful.
func (s *String) ReadASN1Boolean(out *bool) bool {
var bytes String
if !s.ReadASN1(&bytes, asn1.BOOLEAN) || len(bytes) != 1 {
return false
}
switch bytes[0] {
case 0:
*out = false
case 0xff:
*out = true
default:
return false
}
return true
}There's a Go issue that might be related: golang/go#11091 -- it likely is the same underlying issue and Go declined to fix it. In particular, DER does indeed require Regardless, the hex should tell us. |
|
This is the certificate that fails (vault-ca.pem): Go parses this correctly (created with the SHA1 SKID): If I understand correctly, the issue you're referring to is with end of the hex dump. I have no idea why the hex value would change with the SKID hash method (maybe just some dumb luck). Will need to look into that more. But at least I have a better idea what I'm looking for. Unless you have more to add, I think this issue can be closed. Thanks. |
|
Yep, the earlier Go issue I linked is correct and is the root cause of this behavior. You've got a boolean with value Running this certificate through a linting tool such as This isn't resolvable on our end and Go has closed the corresponding issue as WONTFIX. While not ideal, I do strongly suggest reissuing the certificate to correctly conform to the standards. As such, I'm closing this ticket as the issue is in our dependency, Go. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Vault returns errors when trying to parse a pem file for the server certificate signing CA.
I have gotten this error for the use of
VAULT_CACERTenv variable and thesslrootcertparameter of the Postgresql storage connection url.To Reproduce
Steps to reproduce the behavior:
Case 1:
VAULT_CACERT=/path/to/vault-ca.pem vault statusfailed to read environment: ERROR loading CA File: Couldn't parse PEM in: /path/to/vault-ca.pemCase 2:
postgres://{user}:{password}@{db-host:port}/vaultdb?sslmode=verify-full&sslrootcert=/path/to/vault-ca.pem&sslcert=/path/to/vault-cert.pem&sslkey=/path/to/vault-key.pemvault server -config=/path/to/config.hclError initializing storage of type postgesql: failed to check for native upsert: pq: couldn't parse pem in sslrootcertExpected behavior
I expected these parameters to make use of the signing CA. Use of
VAULT_CACERTwill work with the Vault server certificate, though this seems like a workaround and not really a solution. Use of the server cert forsslrootcerthas not worked so far.Environment:
Vault server configuration file(s):
Additional context
I came across this issue while attempting to implement mTLS between Vault and Postgresql. I didn't go into to much detail on this here because I was trying to keep this focused on what I thought was the bug.
If it would be helpful, more details can be found here: https://discuss.hashicorp.com/t/configure-vault-postgresql-storage-with-mutual-tls-connection/40809
I have also briefly tried Vault v1.10.4, but this was just to see if a newer version would resolve the issue. Most troubleshooting effort has been done with v1.9.3.
The text was updated successfully, but these errors were encountered: