Guzzle SSL Certificate Verification Failure on CircleCI

[evilantnie]

I'm trying to push coverage results from our CircleCI builds, but the test-reporter step fails at the end with the following error:

[Guzzle\Http\Exception\CurlException]                                        
  [curl] 60: server certificate verification failed. CAfile: /home/ubuntu/Sto  
  ck/www/protected/composer_vendor/guzzle/guzzle/src/Guzzle/Http/Resources/ca  
  cert.pem CRLfile: none [url] https://codeclimate.com/test_reports    

I've tried upgrading to the latest guzzle version in our composer.json, as well as ensuring that the box has running update-ca-certs on the box, but I still get this error. Any ideas?

[evilantnie]

I've attempted to upgrade the project to Guzzle 4.0 which has an updated pem file (https://github.com/guzzle/guzzle/blob/master/src/cacert.pem) which fixed the certificate issue. However, the ApiClient code isn't entirely compatible with the newer guzzle client and is failing w/ another error: "Unknown error posting Test coverage data."

At this point I'm giving up on leveraging code climate for php coverage until the feature is better supported.

[pbrisbin]

I was able to upload a coverage file successfully in a test project. This test project is using guzzle/guzzle3@92d9934f which is the release commit for 3.9.1. What version is installed in your case?

[rbait]

I'm getting the same exception on Codeship's CI. Installed version of guzzle is guzzle/guzzle3@92d9934

[munkie]

I have the same exception on Travis-CI. I came up with workaround creating report json with test-rporter --stdout > codeclimate.json and then sending it using console curl:

CODECLIMATE_REPO_TOKEN=<token> bin/test-reporter --stdout > codeclimate.json
curl -X POST -d @codeclimate.json -H 'Content-Type: application/json' -H 'User-Agent: Code Climate (PHP Test Reporter v1.0.1-dev)' https://codeclimate.com/test_reports

[rbait]

Nice, works like a charm 👍

[pbrisbin]

@rbait @munkie I've pushed a branch (pb-upgrade-guzzle) which uses Guzzle 4.0. Could you test this version for me?

I had to add the following to my project's composer.json to try out the branch:

    "repositories": [
      {
        "type": "vcs",
        "url": "https://github.com/codeclimate/php-test-reporter"
      }
    ],
    "require-dev": {
        "codeclimate/php-test-reporter": "dev-pb-upgrade-guzzle"
    }

I'm not very familiar with PHP packaging, so there may be an easier way.

[munkie]

@pbrisbin I tried to build with your branch - https://travis-ci.org/crystalservice/samba/jobs/28482243
Result is the same curl error #60:

CURLE_SSL_CACERT (60)
Peer certificate cannot be authenticated with known CA certificates. 

Also job with PHP 5.3 failed because Guzzle requires php 5.4 and composer install failes to execute.

[pbrisbin]

@munkie Just pushed an update to use Guzzle 4.1 (not 4.0) which has an even newer cert file. Could you try that?

[munkie]

@pbrisbin and exception remains - https://travis-ci.org/crystalservice/samba/jobs/28525032#L289

[sun]

Still an issue (travis-ci). File transfer fails due to CA cert verification error.

Given that cURL works, but Guzzle does not, the most simple fix is probably to configure/instruct Guzzle to not use its own CA file, but use the CA file of the operating system instead.

For Guzzle 3, pass the following config option into Client:

$config = ['ssl.certificate_authority' => 'system'];

For Guzzle 4, pass the following config option into Client:

$config = ['verify' => TRUE];

---

Alternatively, in case the final HTTP file transfer is the only usage of Guzzle in php-test-reporter, then it might be wise to even consider to

1. drop Guzzle altogether (one less dependency), and
2. either change the docs to use @munkie's script lines
   OR
   simply write a few lines of raw/native PHP code to perform a HTTP POST request (e.g., relying on the curl extension, which should be available everywhere), which should be more than sufficient for this use-case.

[pbrisbin]

Thanks for this information!

For now, I'm going to cut a new branch from master and do the (seemingly) quick fix of telling Guzzle 3 to use the system cert.

Ultimately, I like your last suggestion the best -- we really don't need that dependency. I'm not sure how quickly I can get that out the door though, not being a PHP developer myself. Hopefully that change to the client configuration will get things working in the meantime.

[pbrisbin]

OK, the branch pb-system-cert has been pushed and is ready to try.

/cc @munkie

[munkie]

@pbrisbin same sh*t - https://travis-ci.org/crystalservice/samba/jobs/28829337#L283

[sun]

Started some code to skip Guzzle. cURL reports the same error + code, but supplies a more helpful error message (emphasis mine):

Connected to codeclimate.com (199.182.122.73) port 443 (#0)
60: SSL certificate problem: self signed certificate in certificate chain
Closing connection 0

Did you install the wrong SSL certificate on codeclimate.com? Self-signed certs should not be used in production.

[pbrisbin]

/cc @noahd1 can you check on the cert used in production?

@sun thanks again for your efforts. If that were the issue though, why would it work for some and not others? Why would command-line curl work, but not Guzzle configured to use system certificates?

[sun]

@pbrisbin curl may be globally configured to ignore self-signed cert errors.

FWIW, the verbose curl shell output from travis-ci:

$ curl -v -X POST -d @codeclimate.json -H 'Content-Type: application/json' -H 'User-Agent: php-test-reporter' https://codeclimate.com/test_reports
* About to connect() to codeclimate.com port 443 (#0)
*   Trying 199.182.122.73... connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-SHA
* Server certificate:
*    subject: OU=Domain Control Validated; OU=EssentialSSL Wildcard; CN=*.codeclimate.com
*    start date: 2013-06-11 00:00:00 GMT
*    expire date: 2014-08-10 23:59:59 GMT
*    subjectAltName: codeclimate.com matched
*    issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=EssentialSSL CA
*    SSL certificate verify ok.
> POST /test_reports HTTP/1.1
> Host: codeclimate.com
> Accept: */*
> Content-Type: application/json
> User-Agent: php-test-reporter
> Content-Length: 3559
> Expect: 100-continue
> 

< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Transfer-Encoding: chunked
< Status: 200 OK
< Strict-Transport-Security: max-age=31536000
< X-Served-By: cc-app02.c45591.bbg
< X-Revision: 8af27e0c34a0570686f1e85fedacbfdb207b74ed
< X-UA-Compatible: IE=Edge,chrome=1
< ETag: "c8fa519244422aeae605c966a97252a2"
< Cache-Control: max-age=0, private, must-revalidate
< X-Request-Id: 053a7bdb85d0f4e00b86ae9da383b40b
< X-Runtime: 0.021619
< Date: Tue, 01 Jul 2014 14:13:05 GMT
< X-Rack-Cache: invalidate, pass
< X-Powered-By: Phusion Passenger
< Server: nginx + Phusion Passenger
< 

* Connection #0 to host codeclimate.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

[pbrisbin]

The only thing that sticks out:

Not working case (from Travis error):

CAfile: /etc/ssl/certs/ca-certificates.crt
CRLfile: none

Working case (from your curl output):

CAfile: none
CApath: /etc/ssl/certs

[pbrisbin]

We've confirmed that our cert is not self-signed and appears to be setup correctly. I'm not sure how to explain that error.

@sun is your code anywhere I could access it (or are you planning on opening a PR ;)?

In one of our other test reporters, we bundle our own cert so it probably makes sense to give up on more trouble-shooting and just do that.

I can figure out how to have Guzzle use a custom cert, but if your code is close, I'd love to incorporate that into that change.

[pbrisbin]

New branch: pb-bundled-cert ready to try.

[atulatri]

Just tried pb-bundled-cert branch, I am getting same errors.

[pbrisbin]

Heads up to anyone currently testing pb- branches: We've determined these fixes don't work and I'm going to delete the branches soon. Please remove them from any composer.json files since an install would fail to find them.

[enygma]

Okay, so there's something amiss here...when I execute this:

openssl s_client -connect codeclimate.com:443

I get the response:

CONNECTED(00000003)
depth=4 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify error:num=19:self signed certificate in certificate chain
verify return:0

Note the "self signed certificate in certificate chain" there. So, in going through some of this SSL certificate mess, I noticed something that could cause this error. I wonder if the intermediate certificates were installed correctly for the Code Climate wildcard domain. Most browsers apparently already come with the necessary intermediate certs so they don't error. @pbrisbin or @noahd1 - could you guys check on that and see if those were installed correctly?

This site doesn't show any problems though, so I'm not sure what's up here...
https://www.sslshopper.com/ssl-checker.html#hostname=https://codeclimate.com/test_reports:443

[ahmadnassri]

any progress on this? seems like self-signed cert is not the way to go here ... wouldn't you want to use cert with a signing authority backing it?

[enygma]

@ahmadnassri it's not that Code Climate's cert is self-signed (it's signed just fine by EssentialSSL's External CA). There's just something in between that's causing issues...either the intermediate certs or something else. I'm not a SSL guru, so I'm just gathering what I can here. Anyone with more skills at identifying the issue is more that welcome to take a crack at it.

[enygma]

Okay, @ahmadnassri @atulatri @sun @munkie @evilantnie - can you folks do me a favor? Can you report back what this line gives you on the machines it's not working on?

php -i | grep 'cURL Information'

I'm starting to suspect different curl versions as Travis-CI still runs 7.22.0 and I seem to have 7.30.0.

[ahmadnassri]

@enygma problem only on travis for me, local is: cURL Information => 7.35.0

[enygma]

I think this could be a curl issue...I'm going to try to build a box with 7.22 installed and see what happens.

[enygma]

Well, the bad news is that I compiled a VM with PHP using curl 7.22.0 and ran my test script that fails on Travis...and it worked. Back to the drawing board.

UPDATE: I tried it first with PHP 5.6.4 and it worked so I tried it with 5.4.35 and....it worked. I'm at a loss here.

[enygma]

After a bit more research (and more reading of the results) I noticed this in the cert chain from the above openssl command:

 1 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root

The issuer and signer are the same. It looks like AddTrust has a self-signed cert for their External CA. That's where that message comes from but it doesn't seem like it should cause the issue with the Unexpected response: -60 server certificate verification failed message. hmm...

[ahmadnassri]

wonder if poking any one from the travis team for their vms / settings would help give a repeatable path...

[enygma]

So I have a test branch of the php-test-reporter that removes the cURL dependency...I'd be interested to see if it cooperates for anyone else. It works local and on Travis-CI:

https://github.com/enygma/php-test-reporter/tree/remove-curl

I just tried it out on a sample app build and it did indeed report back the coverage data correctly.

[geshan]

after_script:
  - CODECLIMATE_REPO_TOKEN="..." bin/test-reporter --stdout > codeclimate.json
  - "curl -X POST -d @codeclimate.json -H 'Content-Type: application/json' -H 'User-Agent: Code Climate (PHP Test Reporter v0.1.1)' https://codeclimate.com/test_reports"

It there an easy way to run the above commands from a shell script, as the code coverage is just taken into account for master branch I generate the coverage only for master branch. But when I need to post the file I think due to "quotes" the curl does not find the file. How can I call it from a shell script?

[jonathancadepowers]

We have now resolved this issue, so it should no longer be necessary to use the cURL work-around.

Instead, you should now be able to use our standard configuration, by passing your clover.xml file to our PHP test reporter directly. This is true for all testing environments, including CIs.

If anyone else runs into a certificate issue, please let us know via our help form.

[nunorafaelrocha]

@jonathancadepowers I still get the certificate errors, but only on the travis CI. Does anyone still have this?

If this was fixed, shouldn't the workaround be removed from the README.md file?

[jonathancadepowers]

@nunorafaelrocha We have received a few new reports of this recently, so it looks like this issue is unfortunately not fully resolved for all projects, as we had originally thought. Very sorry about that. For now the cURL work-around will continue to work: https://github.com/codeclimate/php-test-reporter#known-issue-ssl-certificate-error