SSL certificate verify failed when publishing pacts

[petewarman]

Software versions

• OS: Gitlab CI (docker image used - node:14.17.3)
• Consumer Pact library: @pact-foundation/pact v9.16.3
• Node Version: 14.17.3

Issue Checklist

Please confirm the following:

• I have upgraded to the latest
• I have the read the FAQs in the Readme
• I have triple checked, that there are no unhandled promises in my code and have read the section on intermittent test failures
• I have set my log level to debug and attached a log file showing the complete request/response cycle
• For bonus points and virtual high fives, I have created a reproduceable git repository (see below) to illustrate the problem

Expected behaviour

publishPacts should work as it did before 1st October 2021

Actual behaviour

Since 1st October the tests have been failing with a certificate error

opening connection to [...]
opened
starting SSL for [...] 
SSL established
Conn close because of connect error SSL_connect returned=1 errno=0 state=error: certificate verify failed

Steps to reproduce

I am using Gitlab CI - the node:14.17.3 image. Running apt-get -y upgrade openssl I can successfully curl the broker domain but the pact tests (which have been working perfectly for months) are failing to publish with the certificate error.

Relevant log files

n/a - the tests run fine - it's just the publishPacts step that fails

[mefellows]

See also pact-foundation/pact-js-core#333. I thought we had a tracking issue here but it seems not.

TL;DR
The upstream issue is pact-foundation/pact-ruby-standalone#69, but there are more upstreams to that. The short term resolution is to disable SSL verification with the env var PACT_DISABLE_SSL_VERIFICATION=true or setting one of standard openssl SSL_CERT_FILE or SSL_CERT_DIR env vars.

Impact

1. Provider verifications where it's using an HTTPs endpoint encrypted with Let's Encrypt
2. Any pact-broker command in the embedded CLI tools that communicates to a Pact Broker with a Let's Encrypt cert.

Recommended action

1. Don't do that, it's usually bad practice to test against non-local environments. You should run pact tests in a local environment (dev, CI). If you do this, then HTTPs is not required and provides no additional benefit to your Pact tests. If you can't avoid (1), then...
2. Set one of the above env vars.

[mefellows]

v10.13.9 of Pact Node (a dependency) is now released with upstream fix - can you please check if this resolves the issue? You should no longer need to set environment variables manually. A fresh install of Pact JS should bring in the latest version automatically.

[petewarman]

That seems to have resolved the certificate error problem - thanks!

However, I'm the tests are now failing on a different error

Error: <root-path>/node_modules/@pact-foundation/pact-node/standalone/linux-x64-1.88.77/pact/lib/ruby/lib/ruby/gems/2.2.0/gems/bundler-1.9.9/lib/bundler/shared_helpers.rb:78: warning: Insecure world writable dir <root-path> in PATH, mode 042777

Should I raise a separate issue for this or is there a way to silence this error?

[TimothyJones]

That error is a warning about a misconfiguration on GitHub Agents. You can ignore it.

…

Sent from my mobile

On 11 Oct 2021, at 8:08 pm, Peter Warman ***@***.***> wrote:  That seems to have resolved the certificate error problem - thanks! However, I'm the tests are now failing on a different error Error: ***@***.***/pact-node/standalone/linux-x64-1.88.77/pact/lib/ruby/lib/ruby/gems/2.2.0/gems/bundler-1.9.9/lib/bundler/shared_helpers.rb:78: warning: Insecure world writable dir <root-path> in PATH, mode 042777 Should I raise a separate issue for this or is there a way to silence this error? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[petewarman]

It's giving me a non-zero exit code (ie my CI job is failing) so I can't really ignore it..

[TimothyJones]

Hmm. I suspect something else is the culprit. Can you open another issue with the full log (including logLevel: 'debug') please? How are you running Pact?

The binary doesn't exit when it prints that warning. The error is printed out on standard error from the Ruby binary, which is captured by pact and repeated as an error log (because it came from standard error). The error doesn't generally cause a non-zero exit code - you can see it in several places in our CI. Perhaps there is some place where we are catching it incorrectly, or the presence of the error in the output stream prevents us from parsing the result? I've just had a quick glance at the relevant bits of code, and I don't think that's what's going on.

Older versions of the pact-node wrapper used to fail when the ruby binaries printed anything on standard error. We changed this because of this issue - not all standard error output is fatal. When we first hit it, I wrote a script that corrects (in a very coarse way) the permissions issues. You might like to try that to eliminate the problem:

#!/bin/bash -eu

# This script is a workaround for Ruby complaining about world writeable directories
# and failing the tests because it writes to standard error.

echo "Reducing permissions on github actions' folders to avoid ruby warnings"
sudo chmod 755 -R /home
sudo chmod 755 -R /usr/share
sudo chmod 755 -R /opt

Of course, the real fix for the warning will be with whoever maintains the github runners.

I don't think that this is whatever is failing your build, though. Please open another issue and we'll take a look.

[TimothyJones]

I'll close this issue since the original problem is fixed.

Let's look in to your current problem elsewhere (or just let us know here if the fix was unrelated).

[TimothyJones]

I'm also going to bump the dependency of pact-node, just to ensure that our CI is running the exact version you're trying. This will eliminate any issues from the behaviour of the Ruby binary changing - since the CI line I linked was from a slightly earlier version of pact-node. The build for that is here

[petewarman]

Just FYI I've fixed my tests in CI by wrapping the publisher.publishPacts() call in a try/catch block. They still seem to publish correctly, and the Insecure world writable dir warning is caught and no longer causes the tests to fail.

[TimothyJones]

If you have the chance, it would be great if you could open an issue with more details. It's definitely not supposed to do that, and I can't reproduce it here so far.

[Kampfmoehre]

@TimothyJones We have the same problem and I opened an extra issue with some details and my own debugging experience. See #758