Ensembl REST services do not work on Ubuntu 20.04 — SSLv3 alert handshake failure

[tskir]

On stock Ubuntu 20.04 installation, it is not possible to connect to https://rest.ensembl.org using cURL, Python's requests library, or anything else which relies on OpenSSL for encryption:

$ curl -sS https://rest.ensembl.org >/dev/null
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

This happens because of a combination of three reasons: Ensembl server configuration, increased TLS security level in Ubuntu 20.04 by default, and a bug in OpenSSL 1.1.1.

Ensembl server configuration

Certificate chain of rest.ensembl.org includes a redundant certificate signed with RSA-SHA1, which is broken:

$ gnutls-cli -p 443 ensembl.org | grep -B1 broken
- Certificate[3] info:
 - subject `OU=Go Daddy Class 2 Certification Authority,O=The Go Daddy Group\, Inc.,C=US', issuer `OU=Go Daddy Class 2 Certification Authority,O=The Go Daddy Group\, Inc.,C=US', serial 0x00, RSA key 2048 bits, signed using RSA-SHA1 (broken!), activated `2004-06-29 17:06:20 UTC', expires `2034-06-29 17:06:20 UTC', pin-sha256="VjLZe/p3W/PJnd6lL8JVNBCGQBZynFLdZSTIqcO0SJ8="

Increased security level in Ubuntu 20.04

Previous versions of Ubuntu used OpenSSL built with -DOPENSSL_TLS_SECURITY_LEVEL=1. Starting with Ubuntu 20.04, a decision was made to increase the default security level to 2:

$ openssl version -a | grep compiler | tr ' ' '\n' | grep SECURITY_LEVEL
-DOPENSSL_TLS_SECURITY_LEVEL=2

Because of this, the validation is more strict, and the broken certificate, even though it is redundant, trips OpenSSL. It can be seen with using different cURL options:

$ curl -sS --ciphers 'DEFAULT:@SECLEVEL=1' https://rest.ensembl.org >/dev/null
$ curl -sS --ciphers 'DEFAULT:@SECLEVEL=2' https://rest.ensembl.org >/dev/null
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

Bug in OpenSSL 1.1.1

The fact that a redundant broken certificate trips OpenSSL at TLS security level 2 appears to be a bug: openssl/openssl#11236.

[tskir]

Impact and possible mitigation

In theory, once (and if) OpenSSL developers fix this behaviour, the problem will disappear. However, you may notice that the bug report linked above is more than 2 months old, and on Launchpad it has been designated low priority. So the fix on the OpenSSL side may take very long to arrive.

Ubuntu is a major Linux distribution. The 20.04 was released just a couple of weeks ago, hence not many people have updated yet, but once they gradually do that, I expect you will receive more reports of this problem.

A possible solution is to refresh the server certificate chain so that it does not include a broken certificate. This would be nice to have anyway.

If you'd like to know more details, here is a Launchpad issue opened by someone who faced the same issue with a different server: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1864689

[tskir]

Oh, by the way, if you'll want to reproduce this using Docker images for Ubuntu, I tried that and it won't work because it turns out that the Docker configuration is different from a standard desktop installation. The Docker images use -DOPENSSL_TLS_SECURITY_LEVEL=1, hence no problem. Unfortunately you need an actual Ubuntu 20.04 installation to verify this.

[tuanebi]

Thank you @tskir for reporting the issue.
I can see your point: there is a bug of OpenSSL built with -DOPENSSL_TLS_SECURITY_LEVEL=2, but it can be fixed by removing the redundant certificate from server.

I want to clarify two things:

1. The "broken certificate" you see is the root certificate (self-signed), and actually not broken! The root CA signature is ignored by client because client already has the root CA in its truststore, and thus don't need to check the signature.

2. If you compare certification chains of
   https://www.ssllabs.com/ssltest/analyze.html?d=toodledo.com
   https://www.ssllabs.com/ssltest/analyze.html?d=ensembl.org

you'll see they are the similar and from GoDaddy bundles. I guess many GoDaddy customers are affected, some are within EBI: ensembl.org, europepmc.org, parasite.wormbase.org, uniprot.org. So it will be a web-prod's call to re-generate cert chain for all affected domains, wait for new GoDaddy bundles, or wait for OpenSSL bug fix.

[tskir]

Hi @tuanebi, thank you for your clarifications! What I meant was not the actual certificate being broken, but the RSA-SHA1 cipher being considered broken (which it is, via collision attack).

I notice that this problem still persists. Did you have a chance to triage this with web-prod? Also, do you think this is something which can/should be escalated with GoDaddy? Sounds like they could solve it on their side for all their customers.

[hpages]

Hi,

Facing the same problem here on a fresh Ubuntu 20.04.1 LTS install:

hpages@nebbiolo1:~$ curl https://rest.ensembl.org
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

This breaks some Bioconductor packages trying to access Ensembl REST service e.g.: https://bioconductor.org/checkResults/3.12/bioc-LATEST/MouseFM/nebbiolo1-checksrc.html

Internally, the MouseFM package does something like:

## In an R session:
> library(curl)

> curl::curl_fetch_memory("https://rest.ensembl.org")
Error in curl::curl_fetch_memory("https://rest.ensembl.org") : 
  error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

> sessionInfo()
R version 4.0.2 Patched (2020-08-04 r78971)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.1 LTS

Matrix products: default
BLAS:   /home/hpages/R/R-4.0.r78971/lib/libRblas.so
LAPACK: /home/hpages/R/R-4.0.r78971/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] curl_4.3

loaded via a namespace (and not attached):
[1] compiler_4.0.2

Thanks!
H.

PS: Kudos to @tskir for providing such a detailed and insightful bug report!

[hpages]

Any update on this? Thanks!

[tuanebi]

Hi @hpages @tskir, we are aware of this issue on Ubuntu 20.04 and have liaised with a team who manages our SSL certs. Unfortunately the timeline for fixing it is not clear to me. I would suggest to apply this solution in the meantime https://askubuntu.com/questions/1233186/ubuntu-20-04-how-to-set-lower-ssl-security-level

[hpages]

The temporary fix works for me. Thx @tuanebi!

[hpages]

Hi @tuanebi , have you heard back from the team who manages the SSL certs? Thanks

[grimbough]

For anyone that stumbles here looking for a client-side solution, the following R code worked for me:

library(httr)
url <- "https://rest.ensembl.org/"

## This fails
res <- GET(url)
#> Error in curl::curl_fetch_memory(url, handle = handle): error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

## lower the security level
httr_config <- config(ssl_cipher_list = "DEFAULT@SECLEVEL=1")
res <- with_config(config = httr_config, GET(url))
res
#> Response [https://rest.ensembl.org/]
#>   Date: 2020-10-01 10:32
#>   Status: 200
#>   Content-Type: text/html; charset=utf-8
#>   Size: 31.5 kB
#> 
#> 
#> <!DOCTYPE html>
#> 
#> 
#> 
#> <html lang="en">
#> <head>
#>         <script src="/static/js/20-prettify.js"></script>
#>         <script src="/static/js/30-jquery-1.11.1.min.js"></script>
#> ...

[AsierGonzalez]

I also stumbled upon this problem today but thanks to your suggestions I could fix it. Kudos to you all!

In case anybody is interested, the code that failed for me was a python script using requests to make GET queries. One solution to the issue is to ignore the verification of the certificate setting the verify argument to False as explained in the requests documentation:

requests.get(URL, verify=False)

[jdblischak]

Oh, by the way, if you'll want to reproduce this using Docker images for Ubuntu, I tried that and it won't work because it turns out that the Docker configuration is different from a standard desktop installation. The Docker images use -DOPENSSL_TLS_SECURITY_LEVEL=1, hence no problem. Unfortunately you need an actual Ubuntu 20.04 installation to verify this.

For anyone trying to debug this, I was able to reproduce it using the latest Docker image for Ubuntu Focal.

• The default setting:

  $ docker run --rm ubuntu:focal bash -c "apt-get update -qq; apt-get install -y -qq curl; curl -sS https://rest.ensembl.org > /dev/null"
  curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
  

• Explicitly setting to security level 2:

  $ docker run --rm ubuntu:focal bash -c "apt-get update -qq; apt-get install -y -qq curl; curl -sS --ciphers 'DEFAULT:@SECLEVEL=2' https://rest.ensembl.org > /dev/null"
  curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
  

• Explicitly setting to security level 1 (no error):

  $ docker run --rm ubuntu:focal bash -c "apt-get update -qq; apt-get install -y -qq curl; curl -sS --ciphers 'DEFAULT:@SECLEVEL=1' https://rest.ensembl.org > /dev/null"
  

I omitted output from apt-get. To use the exact same image in the future, you can replace ubuntu:focal with ubuntu@sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c.

$ docker image inspect -f "{{.RepoDigests}}" ubuntu:focal
[ubuntu@sha256:c95a8e48bf88e9849f3e0f723d9f49fa12c5a00cfc6e60d2bc99d87555295e4c]

[grimbough]

Just a heads up that this issue now affects Fedora 33 after that distro also increased the default cryptographic settings details. (Thanks @guidohooiveld for reporting this in grimbough/biomaRt#42)

Unlike Ubuntu this policy seems to apply system wide and affects other applications like Firefox, rather than just those calling OpenSSL.

(Slightly different error message, but I believe the missing cipher is the RSA-SHA1 mentioned in the original post)

This also means individual application workarounds like in #427 (comment) don't work on Fedora. I've so far been unable to get cURL, R or Firefox to access any Ensembl sites without lowing the whole system's security settings.

[tcezard]

Just to add my voice to this list of people reporting this issues.
We've had to stop making call to Ensembl REST API in many of our CI/CD pipeline to prevent it from failing because most of them run on recent version of OS (usually Ubuntu). This is not ideal but we can live with it because it is a development environment.
However, You are probably aware of the imminent switch to Centos 8 in EBI where the default crypto setting are much higher. This will likely mean issues in production and I'd rather avoid that.

[tuanebi]

I've got an update from www-prod that they are developing a solution for all affected EBI certs. The fix will be rolling out asap. I'll update here when I know more.

[hpages]

Hey, this is working today on our Ubuntu 20.04.1 LTS system:

hpages@nebbiolo1:~$ curl https://rest.ensembl.org >test.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 31500  100 31500    0     0  38228      0 --:--:-- --:--:-- --:--:-- 38181

hpages@nebbiolo1:~$ echo $?
0

Yes!

[tskir]

I can confirm everything works for me as well (Ubuntu 20.04.2 LTS), and gnutls-cli probe does not display any broken certificates anymore. Thank you @tuanebi and also everyone who contributed to this discussion. I'm going to close it now; if you're still having any issues on your systems please post the details and I'll reopen.

[guidohooiveld]

Just to add that things are also working fine now on Fedora 33. 👍