Pantheon Recipe - OpenSSL Root Certificate for Let's Encrypt Not Trusted (OpenSSL 1.0.2 is Out of Date)

[GuyPaddock]

Tell us about your setup

v3.4.0 on Ubuntu 20.04.2 LTS (WSL2 on Windows 10.0.19043 Build 19043)

Config Files

.lando.yml

name: my_app
recipe: pantheon
config:
  webroot: web
  drush: 9
  php: '7.3'
  framework: drupal8
  site: REDACTED
  id: REDACTED
services:
  database:
    portforward: 3307
  appserver:
    type: php:7.3
    webroot: web
    xdebug: true

pantheon.yml

api_version: 1
web_docroot: true
php_version: 7.3
enforce_https: transitional

Tell us about the command you were running

# In the example command below, `myapp` is the name of our app
docker exec -it myapp_appserver_1 curl https://mam.inveniem.com/ -vvv

This mimics the behavior that PHP curl calls inside the container also experience.

Tell us about the error you got
With the command above:

*   Trying 52.183.29.73...
* TCP_NODELAY set
* Connected to mam.inveniem.com (52.183.29.73) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: certificate has expired
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

While inside Drupal, making cURL calls with PHP:

 cURL error 60: SSL certificate problem: certificate has expired (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)

Tell us generally about your bug
The problem is that we can't connect to any of our line-of-business services that are secured by Let's Encrypt from PHP in Lando now that the old Let's Encrypt Root Certificate Has Expired. All of our certificates on the server side are up-to-date, but cURL calls from PHP and CLI inside Lando-managed Pantheon containers fail to recognize that ISRG Root X1 is a root certificate, so it's falling over to the old certificate chain that uses the expired DST Root CA X3 root certificate instead.

Consequently, the trusted root certificates inside the container lead the command above to fail.

This issue does not occur on Pantheon itself (especially after last night's updates rolled out to update certificates), so Lando is not accurately replicating the behavior locally that we have in production environments.

Tell us more
It appears that manually entering the app server container and running apt update && apt install does update the root certificates, allowing the CLI cURL command to validate the certificates. However, that has no effect on PHP FPM processes that are already running and does not persist through a restart of containers (as expected).

This prevents us from using Lando without modifying our application to disable SSL verification, which is insecure.

[GuyPaddock]

It looks like a way to workaround this issue is to override the container for the app server with a custom Docker image, as follows:

.lando.yml

name: my_app
# ... lines omitted ...
services:
  # ... lines omitted ...
  appserver:
    # ... lines omitted ...
    overrides:
      build: ./docker/pantheon-appserver-7.3-2
      image: inveniem/pantheon-appserver-with-ssl-updates:7.3-2
# ... lines omitted ...

And then the Docker file:

docker/pantheon-appserver-7.3-2/Dockerfile

FROM devwithlando/pantheon-appserver:7.3-2

RUN apt update && apt install libssl1.0.2

This may mean that the certificates themselves aren't out of date but perhaps there was a bug in the previous version of libSSL 1.0.2. APT reports the new version as libssl1.0.2/oldoldstable 1.0.2u-1~deb9u6 amd64, while the old version was 1.0.2u-1~deb9u4.

For a more reproducible environments, you can pin the libSSL version in the APT command, using the following Dockerfile instead of the one above:

FROM devwithlando/pantheon-appserver:7.3-2

# Update libssl1.0.2 from 1.0.2u-1~deb9u4 to 1.0.2u-1~deb9u6 to work around:
# https://github.com/lando/lando/issues/3162
RUN apt update && apt install libssl1.0.2=1.0.2u-1~deb9u6

[GuyPaddock]

I found a security advisory that explains the problem (was fixed in libssl1.0.2 version 1.0.2u-1~deb9u6):
https://linuxsecurity.com/advisories/deblts/debian-lts-dla-2761-1-openssl1-0-security-update-18-58-09

[pirog]

@GuyPaddock i think the 3.4.0 release should resolve this. You can test the images in that release if you want with:

name: my_app
# ... lines omitted ...
services:
  # ... lines omitted ...
  appserver:
    # ... lines omitted ...
    overrides:
      image: devwithlando/pantheon-appserver:7.3-3
# ... lines omitted ...

[thasmo]

Experiencing similar issues with the platformsh recipe when running lando pull:

➜ lando pull
? Choose relationships to import from platformsh 
? Choose mounts to download from platformsh
? Enter a Platform.sh API token [hidden]
Verifying you are authenticated against platform.sh...
  [RequestException]
  cURL error 60: SSL certificate problem: certificate has expired
  [RingException]
  cURL error 60: SSL certificate problem: certificate has expired

I'm running Lando 3.4.0 and deleted ~/.lando/cache/* and ~/.lando/certs/*.

Is there anything else to do to solve this issue? Thanks!

[pirog]

Platform maintains the images so it would be best to ask them.

…

On Fri, Oct 1, 2021 at 9:23 AM Thomas Deinhamer ***@***.***> wrote: Experiencing similar issues with the platformsh recipe when running lando pull: ➜ lando pull ? Choose relationships to import from platformsh ? Choose mounts to download from platformsh ? Enter a Platform.sh API token [hidden] Verifying you are authenticated against platform.sh... [RequestException] cURL error 60: SSL certificate problem: certificate has expired [RingException] cURL error 60: SSL certificate problem: certificate has expired I'm running Lando 3.4.0 and deleted ~/.lando/cache/* and ~/.lando/certs/*. Is there anything else to do to solve this issue? Thanks! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/lando/lando/issues/3162#issuecomment-932225080>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFOFUCVY4SNBNYRICJD4WDUEWY6NANCNFSM5FDPXB2A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[Hipska]

I'm also having certificate expired problem when adding --remote-url https://sourceforge.net/....

Edit: Indeed fixed with 3.4.0 (brew apparently installed 3.3.x)

[jwbabc]

Issue IS NOT fixed with 3.4.0 using Drupal 8 and the GuzzleHTTP library within the Pantheon recipe.
We still have to disable SSL validation in order to develop within our local environments.

Any tips would be appreciated.

[pirog]

Are you sure about that? Did you do a lando rebuild after you updated?

…

On Fri, Oct 1, 2021 at 10:35 AM Joel Back ***@***.***> wrote: Issue IS NOT fixed with 3.4.0 using Drupal 8 and the GuzzleHTTP library within the Pantheon recipe. We still have to disable SSL validation in order to develop within our local environments. Any tips would be appreciated. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/lando/lando/issues/3162#issuecomment-932285430>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFOFUDKQNXHRXF6QSRF3ALUEXBKBANCNFSM5FDPXB2A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jwbabc]

I did. If you need any information. Let me know...

[GuyPaddock]

3.4.0 on amd64 does not resolve it; I tried that first before reporting this issue. I noted this in my write-up.

3.4.0 still pulls down devwithlando/pantheon-appserver:7.3-2 for me during rebuild.

[GuyPaddock]

Platform maintains the images so it would be best to ask them.

How/where do we go to report those? Do Pantheon recipe issues get reported somewhere else, too?

[GuyPaddock]

@GuyPaddock i think the 3.4.0 release should resolve this. You can test the images in that release if you want with:

name: my_app
# ... lines omitted ...
services:
  # ... lines omitted ...
  appserver:
    # ... lines omitted ...
    overrides:
      image: devwithlando/pantheon-appserver:7.3-3
# ... lines omitted ...

If I explicitly override the image to be devwithlando/pantheon-appserver:7.3-3 that appears to work, but if I just use the 3.4.0 release and do a rebuild, it pulls down devwithlando/pantheon-appserver:7.3-2 which has the issue.

Here's output to show what I mean:

$ lando version
v3.4.0
$ lando rebuild --yes
Rising anew like a fire phoenix from the ashes! Rebuilding app...
Killing my_app_edge_ssl_1        ... done
Killing my_app_edge_1            ... done
Killing my_app_database_1        ... done
Killing my_app_cache_1           ... done
Killing my_app_index_1           ... done
Killing my_app_appserver_nginx_1 ... done
Killing my_app_appserver_1       ... done
Going to remove my_app_edge_ssl_1, my_app_edge_1, my_app_database_1, my_app_cache_1, my_app_index_1, my_app_appserver_nginx_1, my_app_appserver_1
Removing my_app_edge_ssl_1        ... done
Removing my_app_edge_1            ... done
Removing my_app_database_1        ... done
Removing my_app_cache_1           ... done
Removing my_app_index_1           ... done
Removing my_app_appserver_nginx_1 ... done
Removing my_app_appserver_1       ... done
Pulling appserver ... done
Pulling cache     ... done
Pulling index     ... done
Pulling database  ... done
Pulling edge      ... done
index uses an image, skipping
cache uses an image, skipping
appserver uses an image, skipping
appserver_nginx uses an image, skipping
edge uses an image, skipping
database uses an image, skipping
edge_ssl uses an image, skipping
landoproxyhyperion5000gandalfedition_proxy_1 is up-to-date
Creating my_app_appserver_1 ... done
Pantheon pre-run scripting
Generating RSA private key, 2048 bit long modulus
...............+++++
........+++++
e is 65537 (0x010001)
Signature ok
subject=C = US, ST = California, L = San Francisco, O = Lando, OU = Bespin, CN = appserver.my_app.internal
Getting CA Private Key
All settings correct for using Composer
Downloading...

Composer (version 2.0.7) successfully installed to: /usr/local/bin/composer
Use it: php /usr/local/bin/composer

Changed current directory to /var/www/.composer
hirak/prestissimo is not required in your composer.json and has not been removed
./composer.json has been updated
Running composer update hirak/prestissimo
Loading composer repositories with package information
Package "hirak/prestissimo" listed for update is not locked.
Updating dependencies
Nothing to modify in lock file
Installing dependencies from lock file (including require-dev)
Nothing to install, update or remove
Generating autoload files
 Drush Version   :  8.4.2

Attempting to login via terminus...
 [notice] Logging in via machine token.
Logged in as REDACTED
Verifying that you have access to REDACTED...
 ------------------ -------------------------------------------------------------------------------------
  ID                 REDACTED
  Name               my_app-d8-prototype
  Label              my_app-d8-prototype
  Created            2019-06-19 02:26:31
  Framework          drupal8
  Region             United States
  Organization       REDACTED
  Plan               Sandbox
  Max Multidevs      25
  Upstream           4c7176de-e079-eed1-154d-44d5a9945b65: https://github.com/pantheon-systems/empty.git
  Holder Type        user
  Holder ID          REDACTED
  Owner              REDACTED
  Is Frozen?         false
  Date Last Frozen   1970-01-01 00:00:00
 ------------------ -------------------------------------------------------------------------------------
Access confirmed!
Certificate will not expire
Cert is good!
Killing my_app_appserver_1 ...
Killing my_app_appserver_1 ... done
Starting my_app_appserver_1       ... done
Creating my_app_appserver_nginx_1 ... done
Creating my_app_database_1        ... done
Creating my_app_cache_1           ... done
Creating my_app_index_1           ... done
Creating my_app_edge_1            ... done
Creating my_app_edge_ssl_1        ... done
Waiting until database service is ready...
Scanning to determine which services are ready... Please standby...

   ___                      __        __        __     __        ______
  / _ )___  ___  __ _  ___ / /  ___ _/ /_____ _/ /__ _/ /_____ _/ / / /
 / _  / _ \/ _ \/  ' \(_-</ _ \/ _ `/  '_/ _ `/ / _ `/  '_/ _ `/_/_/_/
/____/\___/\___/_/_/_/___/_//_/\_,_/_/\_\\_,_/_/\_,_/_/\_\\_,_(_|_|_)


Your app has started up correctly.
Here are some vitals:

 NAME                  my_app
 LOCATION              REDACTED
 SERVICES              appserver_nginx, appserver, database, cache, edge_ssl, edge, index
 APPSERVER_NGINX URLS  https://localhost:59484
                       http://localhost:59485
 EDGE_SSL URLS         https://localhost:59497
 EDGE URLS             http://localhost:59490
                       http://my_app.lndo.site/
                       https://my_app.lndo.site/

$ docker ps | grep appserver
28ae23189f77   bitnami/nginx:1.16.1-debian-10-r106     "/lando-entrypoint.s…"   2 minutes ago   Up 2 minutes   8080/tcp, 8443/tcp, 127.0.0.1:59485->80/tcp, 127.0.0.1:59484->443/tcp     my_app_appserver_nginx_1
992b03bd1e40   devwithlando/pantheon-appserver:7.3-2   "/lando-entrypoint.s…"   2 minutes ago   Up 2 minutes   9000/tcp                                                                  my_app_appserver_1

Note devwithlando/pantheon-appserver:7.3-2 in that last line.

[pirog]

Ahhh i see the issue @GuyPaddock, we are only using the new -3 variant containers for ARM based laptops primarily Apple Silicon. Guessing we did this so M1 users could try out the new images before we rolled them out to the general audience.

This issue is a good forcing function for just doing that anyway so let me roll a hot fix release.

[jwbabc]

Thanks @GuyPaddock, @pirog and the rest of the lando team!

[reynoldsalec]

@pirog I'm closing this out unless you have further work you wanted to associate with the issue.