pact-node doesn't error if there is an error in the ruby process

[mefellows]

Software versions

• OS: e.g. Mac OSX 10.11.5
• Pact Node version: e.g. Pact Node v2.6.0
• Node Version: node --version
• Other Versions: Any other versions you think might be relevant (optional)

Expected behaviour

See https://pact-foundation.slack.com/archives/C9VBGLUM9/p1626095803481000

Pact node should throw an error and return a rejected promise.

Actual behaviour

Error is swallowed and visible as DEBUG logs:

➤ YN0000: [2021-07-12 13:53:11.823 +0000] INFO (1219 on runner-uzumdsnp-project-10-concurrent-1gzcgn): pact@9.16.0: Verifying message
➤ YN0000: [2021-07-12 13:53:11.837 +0000] INFO (1219 on runner-uzumdsnp-project-10-concurrent-1gzcgn): pact-node@10.12.2: Verifying Pacts.
➤ YN0000: [2021-07-12 13:53:11.839 +0000] INFO (1219 on runner-uzumdsnp-project-10-concurrent-1gzcgn): pact-node@10.12.2: Verifying Pact Files
➤ YN0000: [2021-07-12 13:53:11.841 +0000] DEBUG (1219 on runner-uzumdsnp-project-10-concurrent-1gzcgn): pact-node@10.12.2: Starting pact binary 'standalone/linux-x64-1.88.49/pact/bin/pact-provider-verifier', with arguments [--log-level debug --provider SomeGameClient --provider-app-version 20210712.135035.4fbe5741 --pact-broker-base-url https://.../ --broker-username ... --broker-password ... --publish-verification-results true --enable-pending true --provider-base-url http://localhost:44261 --verbose true]
➤ YN0000: [2021-07-12 13:53:11.906 +0000] DEBUG (1219 on runner-uzumdsnp-project-10-concurrent-1gzcgn): pact-node@10.12.2: Created 'standalone/linux-x64-1.88.49/pact/bin/pact-provider-verifier' process with PID: 1276
➤ YN0000: PASS src/__tests__/message-provider.spec.ts (7.121 s)
➤ YN0000: [2021-07-12 13:53:13.025 +0000] DEBUG (1219 on runner-uzumdsnp-project-10-concurrent-1gzcgn): pact-node@10.12.2: You cannot provide both a username/password and a bearer token. If your Pact Broker uses a bearer token, please remove the username and password configuration.
➤ YN0000: Deprecation warning: Thor exit with status 0 on errors. To keep this behavior, you must define `exit_on_failure?` in `Pact::ProviderVerifier::CLI::Verify`
➤ YN0000: You can silence deprecations warning by setting the environment variable THOR_SILENCE_DEPRECATION.
➤ YN0000: 
➤ YN0000: [2021-07-12 13:53:13.050 +0000] INFO (1219 on runner-uzumdsnp-project-10-concurrent-1gzcgn): pact-node@10.12.2: Pact Verification succeeded.

Steps to reproduce

Provider both a bearer token in the environment (export PACT_BROKER_TOKEN=xyz) and explicitly set username/password in the verifier.

[mefellows]

YN0000: Deprecation warning: Thor exit with status 0 on errors. To keep this behavior, you must define exit_on_failure? in Pact::ProviderVerifier::CLI::Verify

@bethesque this looks like the Ruby CLI is returning an exit status of 0 despite there being an error.

I've just replicated by running this test with the following env var PACT_BROKER_TOKEN=foo: https://github.com/pact-foundation/pact-js/blob/master/examples/e2e/test/provider.spec.js

It all works, except for when it tries to publish the results back:

Pact Verification Complete!
You cannot provide both a username/password and a bearer token. If your Pact Broker uses a bearer token, please remove the username and password configuration.
Deprecation warning: Thor exit with status 0 on errors. To keep this behavior, you must define `exit_on_failure?` in `Pact::ProviderVerifier::CLI::Verify`
You can silence deprecations warning by setting the environment variable THOR_SILENCE_DEPRECATION.

Unsetting PACT_BROKER_TOKEN results in a successful publish.

[TimothyJones]

I’m fairly sure that Pact-node does fail the promise if it’s non-zero (although possibly one or more of the binaries don’t). Also if there’s any output on standard error it is considered an error log (but does not fail the promise).

(This may not be the behaviour in the last pact-node release. But I think it is, because I touched that code for GitHub actions).

[TimothyJones]

That debug log looks like it’s ours, though. I’ll take a look

[mefellows]

Yep, I'm pretty sure it's in the Ruby CLI (the debug is us, yes)

[bethesque]

That Thor bug has been fixed. Someone submitted a PR to upgrade Thor without addressing that issue, and it slipped through. Update the standalone and it will be fine.

[mefellows]

Ah great, thanks Beth.

[TimothyJones]

I will mark this as fixed in .... some version.