Download with xroot protocol on EL9 fails from storage with root CA signed by SHA1

[vokac]

As described in GGUS:164619 transfers using xroot protocol fails

[lxplus960]~% xrdcp --version
v5.6.3

[lxplus960]~% xrdcp root://atlasxrootd-kit.gridka.de:1094//pnfs/gridka.de/atlas/disk-only/atlasdatadisk/rucio/mc23_13p6TeV/2c/85/AOD.34124855._001368.pool.root.1 /tmp
231213 22:03:38 3439610 secgsi_VerifyCA: CA certificate self-signed: integrity check failed (30ffc224.0)
231213 22:03:38 3439611 secgsi_VerifyCA: CA certificate self-signed: integrity check failed (30ffc224.0)
231213 22:03:38 3439612 secgsi_VerifyCA: CA certificate self-signed: integrity check failed (30ffc224.0)
231213 22:03:38 3439613 secgsi_VerifyCA: CA certificate self-signed: integrity check failed (30ffc224.0)
[0B/0B][100%][==================================================][0B/s]  
Run: [FATAL] Auth failed: No protocols left to try (source)


[lxplus960]~% openssl x509 -text -noout -in /etc/grid-security/certificates/30ffc224.0
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2 (0x2)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C = DE, O = DFN-Verein, OU = DFN-PKI, CN = DFN-Verein PCA Grid - G01
        Validity
            Not Before: Mar 30 00:00:00 2011 GMT
            Not After : Mar 30 00:00:00 2031 GMT
        Subject: C = DE, O = DFN-Verein, OU = DFN-PKI, CN = DFN-Verein PCA Grid - G01
...

with default crypto policy on EL9 and it is necessary to apply workaround. This is different behavior compared to HTTPS download done with curl, e.g.

[lxplus960]~% curl -L --cert /tmp/x509up_u$(id -u) --key /tmp/x509up_u$(id -u) --cacert /tmp/x509up_u$(id -u) --capath /etc/grid-security/certificates --output /tmp/x https://atlaswebdav-kit.gridka.de:2880/pnfs/gridka.de/atlas/disk-only/atlasdatadisk/rucio/mc23_13p6TeV/2c/85/AOD.34124855._001368.pool.root.1 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 7844M  100 7844M    0     0  12.8M      0  0:10:11  0:10:11 --:--:-- 13.3M

People mentioned that it is not really useful to validate signature for self-signed Root CA so and that's why it should not matter which hash function was used to sign Root CA.

[bbockelm]

@vokac: what happens when you drop the --capath from your test of curl?

I think I understand why xrootd fails -- but not why curl would succeed in this case. OpenSSL allows for special headers that indicate trust roots; on EL9, this allows a different policy to be applied to trust roots (CAs) versus host certificates (note: the security library used by Java - and dCache - doesn't understand these). So, if curl is getting a CA chain from the system, I'd expect the above to succeed whereas XRootD might be exclusively using /etc/grid-security/certificates. If both are using /etc/grid-security/certificates, I have no good theory as to why curl is working.

OSG is solving this by shipping all CAs with the special header understood by OpenSSL (and then providing a separate RPM for sites with Java).

[vokac]

Why do you think curl could succeed with OS certificates? It fails because of missing trust chain

[lxplus966]~% curl -L --cert /tmp/x509up_u$(id -u) --key /tmp/x509up_u$(id -u) --cacert /tmp/x509up_u$(id -u) --output /tmp/x https://atlaswebdav-kit.gridka.de:2880/pnfs/gridka.de/atlas/disk-only/atlasdatadisk/rucio/mc23_13p6TeV/2c/85/AOD.34124855._001368.pool.root.1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

It seems to me that instead of deploying special RPMs everywhere it would be much easier if IGTF drops CAs signed by SHA1, those that were not able to move away from SHA1 should not be really trusted...

[bbockelm]

Why do you think curl could succeed with OS certificates?

I just wanted to double check. Multiple IGTF CAs (particularly, those used by InCommon and GEANT) have alternate validation paths that allow them to accepted via system trust roots as well as IGTF. I'm not familiar with the DFN one. Just wanted to see if that was possibly explaining the difference with curl.

It seems to me that instead of deploying special RPMs everywhere it would be much easier if IGTF drops CAs signed by SHA1, those that were not able to move away from SHA1 should not be really trusted...

It isn't my call. However, at the system level, SHA1-signed trust roots are accepted on EL9, so I don't find IGTF's decision unreasonable. The "special" RPM mentioned simply matches the code paths for system CAs.

[vokac]

Where is mentioned RPM? I would like to see what is inside to understand details.

[matyasselmeci]

Here's our Koji page for the build: https://koji.osg-htc.org/koji/buildinfo?buildID=16997

and a download link: https://kojihub2000.chtc.wisc.edu/kojifiles/packages/osg-ca-certs/1.115/2.osg36.el9/noarch/osg-ca-certs-1.115-2.osg36.el9.noarch.rpm

Basically we took all the certificate files for SHA1-signed certificates and duplicated the contents, changing the headers on the first copy from BEGIN/END CERTIFICATE to BEGIN/END TRUSTED CERTIFICATE. So the files would look like

-----BEGIN TRUSTED CERTIFICATE-----
MIIDtzCCAp+gAwIBAgIQDOfg5RfYRv6P5WD8G/AwOTANBgkqhkiG9w0BAQUFADBl
...
-----END TRUSTED CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIDtzCCAp+gAwIBAgIQDOfg5RfYRv6P5WD8G/AwOTANBgkqhkiG9w0BAQUFADBl
...
-----END CERTIFICATE-----

Here is the script we used to make the changes: https://github.com/opensciencegrid/osg-certificates/blob/d00157d726c3135083dbeab4f9f838e088e28884/add-trusted-sha1-certs.sh

[vokac]

Do you know why this update is not applied directly by IGTF when they publish new certificate bundle? Any side effects of this workaround?

[maarten-litmaath]

It does not work for CANL-Java --> dCache + StoRM.

[vokac]

Does this mean that default java voms-proxy-init no longer works when we deploy this workaround?

[maarten-litmaath]

That looks quite likely indeed.

[vokac]

This is a mess ... so we have to tell users not to use recommended voms-proxy-init3, but instead move to the deprecated voms-proxy-init2, because we are not able to adapt our software (libraries) fast enough to the security changes in recent OSes (looks like nobody bothered to create ticket for CaNL). This sounds like a lot of fun, but for different discussion...

I would still like to know why XRootD client xrdcp fails with standard IGTF certificates while curl works fine? Does this mean curl certificate validation is not implemented correctly, because it succeeds while it should fail same way as xrdcp?

[maarten-litmaath]

It is possible for voms-proxy-init2 to become fully supported again, for several reasons.

[bbockelm]

I would still like to know why XRootD client xrdcp fails with standard IGTF certificates while curl works fine? Does this mean curl certificate validation is not implemented correctly, because it succeeds while it should fail same way as xrdcp?

Why indeed!

This question sent me down a long rabbit hole. Findings:

1. OpenSSL (and hence libcurl) do not check the signature of root CAs. I believe OpenSSL's logic is: they're declared as trusted CAs -- verifying the self-signed signature is a waste of cycles.
2. XRootD does check the signatures on a root CA. It differs from OpenSSL because it implements its own verification routines for GSI. You can see the difference by noting that using roots:// with your sample endpoint does result in a successful TLS handshake, only to fail later at GSI authentication.
3. I was completely unable to verify the whole BEGIN TRUSTED CERTIFICATE advice. In fact, going back to the original ticket, I see that it contains a suggestion from a RedHat developer and the only verification that it worked was asking the original reporting site to try again with the new package. (It's not clear whether the fix was the new package or some other OpenSSL internal change in the many months that passed from the original ticket!).
   • I went back and tried to recreate the originally-reported setup and couldn't reproduce the original issue.
   • I can verify that xrdcp with the BEGIN TRUSTED CERTIFICATE CAs does not work. Spent about an hour in the debugger and looking through OpenSSL source code and didn't come up with a theory, on the current version of OpenSSL in RHEL9, as to why it would have worked.

So, it appears that the OS's opinion is curl's behavior is correct and the failures can be completely fixed within the xrootd client code base. Further, we need to go back and figure out how to reproduce other failures because either (a) they were misattributed or (b) OpenSSL patches in EL9 have since worked around them.

[amadio]

This is something that can be tuned in the configuration. Please see the -ca option in the documentation. If the configuration is set to verify everything, the observed behavior should be expected. If the default is set to verify and just warn, but failures are observed, then there may be some logic error to be corrected.

Please note that the configuration can also be affected by an environment variable as shown here:

xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc

Lines 2418 to 2421 in 333a5dd

	// "XrdSecGSICACHECK" CA check level [1]:
	// 0 do not verify;
	// 1 verify if self-signed, warn if not;
	// 2 verify in all cases, fail if not possible

The configuration is set here:

xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc

Lines 455 to 463 in 333a5dd

	// CA verification level
	//
	// 0 do not verify
	// 1 verify if self-signed; warn if not
	// 2 verify in all cases; fail if not possible
	//
	if (opt.ca >= caNoVerify && opt.ca <= caVerify)
	CACheck = opt.ca;
	DEBUG("option CACheck: "<<getOptName(caVerOpts,CACheck));

and the actual checks happen here:

xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc

Lines 4528 to 4604 in 333a5dd

	// Is it self-signed ?
	bool self = (!strcmp(xc->IssuerHash(), xc->SubjectHash())) ? 1 : 0;
	if (!self) {
	String inam;
	if (opt == 2) {
	// We are requested to verify it
	bool notdone = 1;
	// We need to load the issuer(s) CA(s)
	XrdCryptoX509 *xd = xc;
	while (notdone) {
	X509Chain *ch = 0;
	int ncis = -1;
	for (int ha = 0; ha < 2; ha++) {
	inam = GetCApath(xd->IssuerHash(ha));
	if (inam.length() <= 0) continue;
	ch = new X509Chain();
	ncis = (*ParseFile)(inam.c_str(), ch, 0);
	if (ncis >= 1) break;
	SafeDelete(ch);
	}
	if (ncis < 1) break;
	XrdCryptoX509 *xi = ch->Begin();
	while (xi) {
	if (!strcmp(xd->IssuerHash(), xi->SubjectHash()))
	break;
	xi = ch->Next();
	}
	if (xi) {
	// Add the certificate to the requested CA chain
	ch->Remove(xi);
	cca->PutInFront(xi);
	SafeDelete(ch);
	// We may be over
	if (!strcmp(xi->IssuerHash(), xi->SubjectHash())) {
	notdone = 0;
	break;
	} else {
	// This becomes the daughter
	xd = xi;
	}
	} else {
	break;
	}
	}
	if (!notdone) {
	// Verify the chain
	X509Chain::EX509ChainErr e;
	x509ChainVerifyOpt_t vopt = {kOptsCheckSubCA, 0, -1, 0};
	if (!(verified = cca->Verify(e, &vopt)))
	PRINT("CA certificate not self-signed: verification failed for '"<<xc->SubjectHash()<<"': error: "<< cca->X509ChainError(e));
	} else {
	PRINT("CA certificate not self-signed: cannot verify integrity ("<<xc->SubjectHash()<<")");
	}
	} else {
	// Fill CA information
	cca->CheckCA(0);
	// Set OK in any case
	verified = 1;
	// Notify if some sort of check was required
	if (opt == 1) {
	NOTIFY("Warning: CA certificate not self-signed and"
	" integrity not checked: assuming OK ("<<xc->SubjectHash()<<")");
	}
	}
	} else {
	if (CACheck > caNoVerify) {
	// Check self-signature
	if (!(verified = cca->CheckCA()))
	PRINT("CA certificate self-signed: integrity check failed ("<<xc->SubjectHash()<<")");
	} else {
	// Set OK in any case
	verified = 1;
	// Notify if some sort of check was required
	NOTIFY("Warning: CA certificate self-signed but"
	" integrity not checked: assuming OK ("<<xc->SubjectHash()<<")");
	}
	}

[amadio]

I think that the check done here in line 4595 needs to be updated:

xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc

Lines 4593 to 4603 in 333a5dd

	if (CACheck > caNoVerify) {
	// Check self-signature
	if (!(verified = cca->CheckCA()))
	PRINT("CA certificate self-signed: integrity check failed ("<<xc->SubjectHash()<<")");
	} else {
	// Set OK in any case
	verified = 1;
	// Notify if some sort of check was required
	NOTIFY("Warning: CA certificate self-signed but"
	" integrity not checked: assuming OK ("<<xc->SubjectHash()<<")");
	}

It should be like this one:

xrootd/src/XrdSecgsi/XrdSecProtocolgsi.cc

Lines 5183 to 5185 in 333a5dd

	// Check if any CA was in the file
	bool checkselfsigned = (CACheck > caVerifyss) ? true : false;
	po->chain->CheckCA(checkselfsigned);

I will prepare a patch for this.

[bbockelm]

@amadio - I'm not quite understanding the comments in the code:

 //    0   do not verify 
 //    1   verify if self-signed; warn if not 
 //    2   verify in all cases; fail if not possible 

What does "verify if self-signed; warn if not" mean? Don't we want the opposite, we verify signatures until we hit a self-signed CA?

[amadio]

What does "verify if self-signed; warn if not" mean? Don't we want the opposite, we verify signatures until we hit a self-signed CA?

This is the check level for the CA itself, the rest of the chain should always be checked.