Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let's Encrypt certificate verify failed September 30th #69

Closed
2 of 3 tasks
mkj28 opened this issue Oct 1, 2021 · 26 comments
Closed
2 of 3 tasks

Let's Encrypt certificate verify failed September 30th #69

mkj28 opened this issue Oct 1, 2021 · 26 comments

Comments

@mkj28
Copy link

mkj28 commented Oct 1, 2021

Seeing issues related to https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

Pre issue-raising checklist

I have already (please mark the applicable with an x):

  • Upgraded to the latest version of the relevant libraries
  • Checked to see if the issue has already been raised
  • Created an executable example that demonstrates the issue using either:

Software versions

  • pact library: Pact Go CLI v1.6.4, using CLI tools version 1.82.3
  • pact-ruby-standalone: 1.88.72
  • OS: debian buster (golang:1.16.2-buster docker)

Expected behaviour

pact_verifier able to connect to host behind Let's Encrypt cert

Actual behaviour

Hitting SSL issue starting September 30th

Steps to reproduce

TBD

Relevent log files

Not DEBUG, but could be enough?

Error making request - OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed , attempt 1 of 3
Error making request - OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed , attempt 2 of 3
Error making request - OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed , attempt 3 of 3
/opt/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:923:in `connect': SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError)
 from /opt/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:923:in `block in connect'
 from /opt/pact/lib/ruby/lib/ruby/2.2.0/timeout.rb:74:in `timeout'
 from /opt/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:923:in `connect'
 from /opt/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:863:in `do_start'
 from /opt/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:852:in `start'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/hal/http_client.rb:55:in `block in perform_request'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/retry.rb:23:in `until_true'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/hal/http_client.rb:49:in `perform_request'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/hal/http_client.rb:24:in `get'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/hal/link.rb:49:in `get'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/pact_broker/fetch_pact_uris_for_verification.rb:54:in `index'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/pact_broker/fetch_pact_uris_for_verification.rb:39:in `call'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/pact_broker/fetch_pact_uris_for_verification.rb:35:in `call'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/pact_broker.rb:18:in `fetch_pact_uris_for_verification'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/aggregate_pact_configs.rb:46:in `pacts_for_verification'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/aggregate_pact_configs.rb:39:in `pacts_urls_from_broker'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/aggregate_pact_configs.rb:26:in `call'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/aggregate_pact_configs.rb:10:in `call'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/app.rb:207:in `all_pact_urls'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/app.rb:221:in `warn_empty_pact_set'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/app.rb:40:in `call'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/app.rb:35:in `call'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/cli/verify.rb:49:in `verify'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/thor-1.1.0/lib/thor/base.rb:485:in `start'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/cli/custom_thor.rb:17:in `start'
 from /opt/pact/lib/app/pact-provider-verifier.rb:33:in `
@bethesque
Copy link
Member

Until we can get it properly fixed, try setting PACT_DISABLE_SSL_VERIFICATION=true

@mkj28
Copy link
Author

mkj28 commented Oct 1, 2021

Until we can get it properly fixed, try setting PACT_DISABLE_SSL_VERIFICATION=true

you are a life saver! testing now

@mkj28
Copy link
Author

mkj28 commented Oct 1, 2021

would same work for other libraries (node specifically?) - Ill test soon, but let me do go first

@bethesque
Copy link
Member

You can also give it a custom cert with the normal unix SSL_CERT_FILE or SSL_CERT_DIR env vars (though, having briefly read through the issue, that may not help).

@bethesque
Copy link
Member

Anything that uses the pact-ruby-standalone will honour PACT_DISABLE_SSL_VERIFICATION if all the env vars are passed through to the child process.

@mkj28
Copy link
Author

mkj28 commented Oct 1, 2021

@bethesque unfortunately seems to be getting same with PACT_DISABLE_SSL_VERIFICATION=true

I also tried setting SSL_CERT_FILE or SSL_CERT_DIR explicitly - but it made no difference.

@mkj28
Copy link
Author

mkj28 commented Oct 1, 2021

would same work for other libraries (node specifically?) - Ill test soon, but let me do go first

seems to have worked for node, but not pact-go

@mkj28
Copy link
Author

mkj28 commented Oct 1, 2021

with PACT_DISABLE_SSL_VERIFICATION=true (and also PACT_BROKER_DISABLE_SSL_VERIFICATION=true)

still getting:

STDERR:
Error making request - OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed , attempt 1 of 3
Error making request - OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed , attempt 2 of 3
Error making request - OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed , attempt 3 of 3
/opt/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:923:in `connect': SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError)
 from /opt/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:923:in `block in connect'
 from /opt/pact/lib/ruby/lib/ruby/2.2.0/timeout.rb:74:in `timeout'
 from /opt/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:923:in `connect'
 from /opt/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:863:in `do_start'
 from /opt/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:852:in `start'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/hal/http_client.rb:55:in `block in perform_request'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/retry.rb:23:in `until_****'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/hal/http_client.rb:49:in `perform_request'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/hal/http_client.rb:24:in `get'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/hal/link.rb:49:in `get'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/pact_broker/fetch_pact_uris_for_verification.rb:54:in `index'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/pact_broker/fetch_pact_uris_for_verification.rb:39:in `call'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/pact_broker/fetch_pact_uris_for_verification.rb:35:in `call'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.59.0/lib/pact/pact_broker.rb:18:in `fetch_pact_uris_for_verification'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/aggregate_pact_configs.rb:46:in `pacts_for_verification'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/aggregate_pact_configs.rb:39:in `pacts_urls_from_broker'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/aggregate_pact_configs.rb:26:in `call'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/aggregate_pact_configs.rb:10:in `call'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/app.rb:207:in `all_pact_urls'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/app.rb:221:in `warn_empty_pact_set'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/app.rb:40:in `call'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/app.rb:35:in `call'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/cli/verify.rb:49:in `verify'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/thor-1.1.0/lib/thor/base.rb:485:in `start'
 from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/cli/custom_thor.rb:17:in `start'
 from /opt/pact/lib/app/pact-provider-verifier.rb:33:in `

@bethesque
Copy link
Member

What version of the standalone are you on? I only introduced the disabling env var in the last 6 months or so.

@mkj28
Copy link
Author

mkj28 commented Oct 1, 2021

What version of the standalone are you on? I only introduced the disabling env var in the last 6 months or so.

1.88.72

@bethesque
Copy link
Member

@mkj28
Copy link
Author

mkj28 commented Oct 1, 2021

It does seem like not getting picked up by pact-ruby? Maybe pact-foundation/pact-ruby#248 is a better location for the workaround to work there?

@bethesque
Copy link
Member

Yeah, it's only hacked into the standalone. I'll need to put it in separately to pact-ruby.

@mkj28
Copy link
Author

mkj28 commented Oct 1, 2021

Yeah, it's only hacked into the standalone. I'll need to put it in separately to pact-ruby.

And I suspect that will then need to get packaged into new release of standalone?

@mkj28
Copy link
Author

mkj28 commented Oct 1, 2021

Just in case - seems like https://github.com/pact-foundation/pact-ruby-standalone/releases/download/v1.88.73/pact-1.88.73-linux-x86_64.tar.gz still carries 1.59 of pact-ruby rather than 1.60.

@mkj28
Copy link
Author

mkj28 commented Oct 1, 2021

Trying 1.88.74 now

@mefellows
Copy link
Member

mefellows commented Oct 1, 2021

Unfortunately 1.88.75 doesn't seem to resolve this issue:

➜  bin pact-provider-verifier --pact-broker-base-url https://host.with.letsencrypt.cert.com --provider foo --provider-base-url http://localhost:1234
Error making request - OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed , attempt 1 of 3
Error making request - OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed , attempt 2 of 3
Error making request - OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed , attempt 3 of 3
/opt/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:923:in `connect': SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError)
	from /opt/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:923:in `block in connect'
	from /opt/pact/lib/ruby/lib/ruby/2.2.0/timeout.rb:74:in `timeout'
	from /opt/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:923:in `connect'
	from /opt/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:863:in `do_start'
	from /opt/pact/lib/ruby/lib/ruby/2.2.0/net/http.rb:852:in `start'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.60.0/lib/pact/hal/http_client.rb:62:in `block in perform_request'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.60.0/lib/pact/retry.rb:23:in `until_true'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.60.0/lib/pact/hal/http_client.rb:50:in `perform_request'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.60.0/lib/pact/hal/http_client.rb:25:in `get'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.60.0/lib/pact/hal/link.rb:49:in `get'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.60.0/lib/pact/pact_broker/fetch_pact_uris_for_verification.rb:54:in `index'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.60.0/lib/pact/pact_broker/fetch_pact_uris_for_verification.rb:39:in `call'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.60.0/lib/pact/pact_broker/fetch_pact_uris_for_verification.rb:35:in `call'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-1.60.0/lib/pact/pact_broker.rb:18:in `fetch_pact_uris_for_verification'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/aggregate_pact_configs.rb:46:in `pacts_for_verification'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/aggregate_pact_configs.rb:39:in `pacts_urls_from_broker'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/aggregate_pact_configs.rb:26:in `call'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/aggregate_pact_configs.rb:10:in `call'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/app.rb:207:in `all_pact_urls'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/app.rb:221:in `warn_empty_pact_set'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/app.rb:40:in `call'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/app.rb:35:in `call'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/cli/verify.rb:49:in `verify'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/thor-1.1.0/lib/thor/base.rb:485:in `start'
	from /opt/pact/lib/vendor/ruby/2.2.0/gems/pact-provider-verifier-1.36.0/lib/pact/provider_verifier/cli/custom_thor.rb:17:in `start'
	from /opt/pact/lib/app/pact-provider-verifier.rb:33:in `<main>'

Or is the fix to allow disabling of the certificate as per #69 (comment) (i.e. PACT_DISABLE_SSL_VERIFICATION=true, or the SSL_CERT_FILE or SSL_CERT_DIR env vars)

would same work for other libraries (node specifically?) - Ill test soon, but let me do go first

seems to have worked for node, but not pact-go

@mkj28 it should work for Pact Go in the same way, but the env vars may need to be explicitly carried through, i'll take a look now.

@bethesque
Copy link
Member

No, it's not Fixed, the release was to allow the SSL verification to be disabled until we can get a proper fix done.

@bethesque
Copy link
Member

You should also be able to set the SSL_CERT_FILE if you get your certificate chain as per https://docs.pact.io/pact_broker/advanced_topics/using-tls/#extracting-a-copy-of-the-certificate-from-a-running-pact-broker

@mkj28
Copy link
Author

mkj28 commented Oct 2, 2021

ya, just to also post here - what @bethesque said - it's working with 1.88.75 + PACT_DISABLE_SSL_VERIFICATION=true for us

@meringu
Copy link

meringu commented Oct 5, 2021

Hey team, looks like this is due to Pact shipping with an embedded ruby version 2.2. Support for Ruby 2.2 ended 2018: https://www.ruby-lang.org/en/news/2018/06/20/support-of-ruby-2-2-has-ended/

The solution would be to upgrade ruby version Pact ships with, as modern ruby versions have updated bundles. Alternatively as a work around to keep SSL verification on could overwrite the bundle.

Something like:

cp -f /etc/ssl/certs/ca-certificates.crt ./lib/ruby/lib/ca-bundle.crt

Inside the Pact installation directory

@meringu
Copy link

meringu commented Oct 5, 2021

@bethesque, looks like embedded ruby overwrites the SSL_CERT_FILE environment variable.

In an Ubuntu container:

root@a80a1ca49eb1:/tmp/pact/pact# SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt ./lib/ruby/bin/ruby -e 'puts ENV["SSL_CERT_FILE"]'
/tmp/pact/pact/lib/ruby/lib/ca-bundle.crt

@mefellows
Copy link
Member

Yes upgrading is something that will need to happen but I believe we are blocked on an upstream PR to support later versions (via Travelling Ruby).

Thanks for documenting the workaround for others, Beth is also looking at patching it in here for now also.

@meringu
Copy link

meringu commented Oct 5, 2021

Looks like the environment variable is actually working for us sorry. Running embedded ruby directly seems to ignore it.

@m-aciek
Copy link

m-aciek commented Oct 5, 2021

For the record the upgrade of Ruby, required to have this issue solved, is being tracked in #63.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants