fetching URLs: Received fatal alert: handshake failure

[paregorios]

OpenRefine 2.7 (1) on Mac OSX 10.11.6 with Google Chrome 61.0.3163.100 (Official Build) (64-bit).

Attempting to "Edit column" -> "Add column by fetching URLs ..." one gets: "Received fatal alert: handshake failure" for all URLs tried in the https://pleiades.stoa.org/ domain.

but curling the urls on same platform has no problems, e.g.:

$ curl -I https://pleiades.stoa.org/places/540867/json
HTTP/1.1 200 OK
Date: Thu, 12 Oct 2017 20:48:30 GMT
Server: Zope/(2.13.24, python 2.7.11, linux2) ZServer/1.1
Content-Length: 16430
Vary: Accept
Access-Control-Allow-Origin: *
Content-Type: application/json; charset=utf-8
X-Frame-Options: SAMEORIGIN
X-Varnish: 794267
Age: 0
Via: 1.1 varnish-v4
X-Varnish-Cache: MISS
Accept-Ranges: bytes
Cache-Control: max-age=86400, s-maxage=11988
Expires: Fri, 13 Oct 2017 20:48:30 GMT

[wetneb]

I think this user found a fix - it'd be nice to know if there anything we can do upstream to prevent that problem:
https://twitter.com/27point7/status/918002725048279040

[paregorios]

Hmmm. I installed from DMG per docs. If there's java config that needs doing for non-developer users, I missed it in the docs. Pointer?

[paregorios]

I have Java 1.8.0_144 installed.

[paregorios]

Using keytool to install root cert in question, as implied by the link cited by @wetneb above, has made no difference for me. I am still seeing same errors, even after full restart and ensuring that "cache results" is turned off in OR. I note that the discussion on twitter seems to involve people running OR from the command line, presumably on linux.

I have tried launching OR using the JavaAppLauncher distributed inside the Mac application bundle in the hopes of seeing an error notification or traceback at the command line when the error occurs. OR runs fine this way, and I do see application messages, but the errors still occur without any corresponding output on the command line.

[ostephens]

The problem (or at least part of the problem) seems to be with the java encryption suite. This is different to the solution linked above.

Connecting with:
openssl s_client -connect pleiades.stoa.org:443
I can see the following in the response
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384

This encryption isn't supported in the default Java 8 encryption suite. (a nice way to check what is being used by OpenRefine - add the URL "https://www.howsmyssl.com/a/check" to an OpenRefine cell and use Add column from URL to retrieve the JSON :)

To support this encryption you need to install the JCE - Java Cryptography Extension http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

I'd copied the jars from that download into jre/lib/security/ (replacing the existing versions). I was able to get this working (although by then I'd already installed additional certificates, so I'm not able to confirm whether you need to do both steps, or just the JCE step - I suspect just the JCE will be fine).

At this point I now get a HTTP error 403 : Forbidden from https://pleiades.stoa.org/places/540867/json

I believe this to be down to the due to the User Agent being specified by OpenRefine. If I curl:
curl -I -A "Java/1.7.0_121" "https://pleiades.stoa.org:443/places/540867/json"
curl -I -A "Java/1.8.0_72" https://pleiades.stoa.org:443/places/540867/json
etc.

I also get a 403 forbidden from this website. At the moment it is not possible to change the User Agent used by OpenRefine. See #1217 "The User Agent OpenRefine uses is currently determined by the version of the underlying Java library"

[paregorios]

@ostephens, regarding the 403s: yes, I believe you are correct. We have apache config that rejects certain default and blankish user agent strings because of bad behavior by some bots in the past.

I'm going to undertake to remove (temporarily?) I have removed this restriction from the Pleiades server. I will attempt to follow the steps you outline with the JCE to see if I can get things working. And I'll report back here to inform any follow-on software or documentation actions the OR development community wants to take.

Regarding the apache rewrite rules: If we don't see significant performance degradation tied to anonymous bots, we might leave the exception in place, but ideally I'd like to see automated agents using specific user agents and so will follow the discussion on #1217.

[paregorios]

@ostephens unfortunately I cannot yet duplicate your success. A diagnostic question:

• Are you running the OSX production 2.7 distro of OR, or are you building locally and running from the command-line or an IDE?

[ostephens]

@paregorios I was using the Linux version of the latest code on Mac, running from the command line (and with an additional tweak and rebuild to set a recognised user agent successfully retrieved JSON from Pleiades)

Just tried with the OS X production 2.7 distro and found I got the handshake error again :( So now trying to debug that ...

[ostephens]

@paregorios OK - success. It looks like if you run the Mac version it uses the Java cacerts from /Applications/OpenRefine.app/Contents/PlugIns/jdk1.8.0_60.jdk/Contents/Home/jre/lib/security

(this is mentioned in https://groups.google.com/forum/#!msg/openrefine/6adewVTQh4Y/gsJdepAmAAAJ;context-place=forum/openrefine, although I definitely have $JAVA_HOME set, so I wonder if there is an issue with the Mac build in this respect).

Anyway, so I copied the JCE local_policy.jar and US_export_policy.jar into this directory, but then got the "sun.security.validator.ValidatorException: PKIX path building failed" error, which means that it is missing a certificate.

So I then installed the cert pleiadesstoaorg.crt I got from my browser by visiting pleiades.stoa.org - but that still didn't resolve the issue.

So then I installed the root cert from Lets Encrypt (obtained from https://letsencrypt.org/certificates/) using:
keytool -import -alias isrgrootx1 -keystore /Applications/OpenRefine.app/Contents/PlugIns/jdk1.8.0_60.jdk/Contents/Home/jre/lib/security/cacerts -file /Path/to/isrgrootx1.pem

And Success!

[paregorios]

Yeah, so it looks like the OR 2.7 distro for OSX bundles jdk 1.8.0_60 and ignores the $JAVA_HOME variable. It seems to be hard-coded in OpenRefine.app/Contents/Info.plist.

[paregorios]

According to this, the "DST Root CA X3" certificate, which is the parent to the "Let's Encrypt" certificate was only added to the default JDK/JRE keystore with versions 7u111 and 8u101. So, bundling jdk 1.8.0_101 or later might take care at least of the certificate problem (and not just for Pleiades).

[paregorios]

@ostephens thanks for the roadmap. I am now fetching Pleiades JSON with my hacked version of the OSX OR 2.7 distro. Adding the pleiades cert (which was part of your debugging process, I realize) wasn't necessary, so long as the root cert from Let's Encrypt has been added.

This resolves my immediate issue, but -- if I may be so bold -- would seem to leave open some steps for the OR development community to explore with respect to the (next) OSX release. This workaround is probably too techno-fiddly for most of the users to whom I'd like to be able to recommend OpenRefine for Pleiades reconciliation. I am happy to help troubleshoot and/or test, if that would be useful, but I don't have java development skills.

[jackyq2015]

@paregorios, can I say the root cause is OR 2.7 for OSX ignore the $JAVA_HOME? or OR will not well with Pleiades with jdk 1.8 regardless the OS due to the missing root CA issue ?

[ostephens]

@jackyq2015 don’t think getting OR on Mac to observe $JAVA_HOME will help with this issue specifically- it just pushes the issue back to a local system java issue rather than a bundled java issue.

There are (I think) two issues:

• inclusion of a specific root cert
• use of local_policy.jar and US_export_policy.jar from the JCE

Neither of these are necessarily fixed if we use $JAVA_HOME - it would then just depend on how the local Java was setup.

I wonder if using a bundled java might be a feature rather than a bug. If we can bundle a version of Java with updated cacert and the policy jars - which would fix both issues.

[jackyq2015]

@ostephens , I checked the folder under /Applications/OpenRefine.app/Contents/PlugIns/jdk1.8.0_60.jdk/Contents/Home/jre/lib/security after installing 2.7RC2. I did find the 2 jars you mentioned.
But the cacerts does not include the DST cert indeed.

For the linux box and openjdk, I do see the entry:

root@hp:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security# keytool -list  -keystore cacerts | grep "\.pem" | grep dst
debian:dst_root_ca_x3.pem, 24-Apr-2017, trustedCertEntry, 
debian:dst_aces_ca_x6.pem, 24-Apr-2017, trustedCertEntry, 

Next time we release, we can try to bundle the 1.8.0_101 or later as @ostephens suggested. Hopefully that will solve the issue.

[ostephens]

@jackyq2015 just to note - as far as I know, the local_policy.jar and US_export_policy.jar in the Java distribution is not sufficient for this to work. You need to use the ones from the Java Cryptography Extension (JCE) available at http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html for the decryption to work. The policy jar in the bundled java would need to be updated to be the JCE ones

[thadguidry]

@ostephens I gotta ask... How the hell do other Java applications that run on Mac handle all this ? What do they do now compared to how things were when this article was written by Oracle https://docs.oracle.com/javase/7/docs/technotes/guides/jweb/packagingAppsForMac.html ?

My thoughts are that we don't bundle...but instead provide nice scripts for Mac users to run, to make @paregorios problem... the Cert installation or whatever on Mac for OpenRefine usage... EASIER. Let do that instead. (decouple the issue from our build process, so we don't have to alter our build process)

[jackyq2015]

According to this, there are no AES256 cipher suites offered by current bundle java client. My understanding is that in order to make it work, we have several options:

• Provide the bundled version with (JCE) Unlimited Strength Jurisdiction Policy and dst_root_ca_x3.pem as @ostephens recommenced
• Provide the script to automate the the above process without bundling as @thadguidry recommenced
• Another option could be have the site to provide other more populate cipher which is included to most of Java version. that feasibility has to be left to @paregorios to answer.

[ostephens]

@thadguidry that's a really good question! I don't know, but I think this blog post might describe the same issue https://blog.spaziodati.eu/en/2014/11/17/the-dark-side-of-ssl-on-java-virtual-machine-jvm/ - so if I'm right, we aren't alone in seeing some issues around this.

Having thought about it a bit, on balance I'm in favour of what @thadguidry suggests - avoid bundling the JCE stuff (some slight nagging worry in the back of my mind about legal issues related to this as well), but offer better support.

I think @jackyq2015 point about whether @paregorios could make a change at Pleiades to avoid the problem in this particular case is also a good one.

This Stackoverflow post seems to have a lot of good stuff in - we could even consider testing for the issue and falling back (or just preferring) the suggested Bouncy Castle cryptography library/API that could help avoid the problem https://stackoverflow.com/questions/1179672/how-to-avoid-installing-unlimited-strength-jce-policy-files-when-deploying-an

[msaby]

Hi
I don't understand half of your conversation ;-), but as someone wrote, I spent some time wrangling with java certificates in OR some days ago.
I believe Mac version do NOT ignore $JAVA_HOME. I had issues with missing certificate in cacerts, but my $JAVA_HOME was void. When I put the path of my system java in $JAVA_HOME, the issue was resolved

[ostephens]

@msaby which version of OpenRefine are you using?

[thadguidry]

@ostephens At the end of the day, I'd like you to just close this issue and then provide some good documentation, avoidance, best practice, etc... on our Wiki to help navigate this kind of problem with Server side security when using our Fetch URLs function against a server using HTTPS. We can then point folks to that page to try different options, and advise of security concerns as well. A Wiki page is a much better way to deal with this very niche "area" (its not an issue) in particular. Which really OpenRefine dev / support team should not be getting into the weeds with. Let's call that Wiki page "Fetching URLs with HTTPS" or whatever.

Thanks All.

[ettorerizza]

@ostephens When you say "using a bundled java", would that mean that Open Refine would become totally portable and would not need to install anything else to work? If so, it might interest some users, I believe.

https://twitter.com/parody_bit/status/923931736479883264

[thadguidry]

@ettorerizza The problem still might exist. Users in constrained IT environments might not see OpenRefine being allowed to start because JRE or JDK is caught as suspicious by a corporate security program running on their computer. We cannot help that however, so its a non-issue for us.

I think I would be OK with a bundled JRE, and allowing folks to still override the bundled JRE to that of their $JAVA_HOME

The problem that I don't know about on OSX is about Signing ? Is it a requirement at the OSX level or is it just a requirement for Apple Store ? In other words, does the OpenRefine+JRE bundled app on OSX need to be signed somehow or not ?

[jackyq2015]

@thadguidry self-signed cert is not mandatory from OSX level. Since we are not distribute the software to apple store, that not mandatory either(actually, if you want to distribute to store, you need a developer certificate signed from apple). AFAIK, The reason we self-signed it is to avoid some warning popup for security from the OSX. And for some system, OSX will hide the icon when it is not self-signed. Basically it's poor man's choice to try to align with the security framework from apple. :).

code signing is to sign the distribution as whole to assure the identify of developer does the release, which means it does not matter you bundle something or not.

[ostephens]

I've now added a new page on troubleshooting problems fetching data from URLs, putting in the problems noted in this issue and the solutions at https://github.com/OpenRefine/OpenRefine/wiki/Troubleshooting:-Fetching-data-from-URLs

As noted by @paregorios some of the solutions are technical, and will be challenging for a user without experience of using the command line or similar level of technical knowledge.

I've also confirmed through testing that if you use OpenRefine for Mac, the Java cacerts and encryption policy files that are important are the ones that are in the JRE included in the distribution (e.g. in my current install of OR2.8, this is /Applications/OpenRefine.app/Contents/PlugIns/jdk1.8.0_60.jdk/Contents/Home/jre/lib/security/ although the path can obviously vary across installs and versions)

This is as far as I'm able to take it in terms of the security certs - we haven't identified a good/easy solution for ensuring that users don't encounter problems here. I think @thadguidry's suggestion of a separate script to do the necessary updates is probably the right approach, but I don't know how to start approaching this.

It would be good to update the Java version which is included in the Mac version at some point, but I think that's a connected but separate issue.

Finally #1217 is also related and outstanding - and I think is something we can do something about - I'm planning on having a look at how we might allow the user to specify request headers including user-agent soon.

Closing this specific issue now.

[thadguidry]

@tonero101 Has a Mac and might be able to throw together a script to make this easier. He's working on the JDBC enhancement also. Tony what do you say, can you tackle a script for Mac users to make cacerts installation and management easier ?

@ostephens @jackyq2015 We need to drop Java7 now and go head first into Java8+ Do we have a issue labeled as "task" for that yet ?

[ostephens]

@thadguidry To be clear, the Java version distributed in the Mac package is JDK 1.8.0_60 - so already on Java 8 - just would be good to update this to a more recent 1.8.X at some point which could improve the cacert situation on Mac - but of course Win and Linux (afaik) would depend on the local Java install still, so it's a minor issue I think.

I'm not clear what determines the version of Java included in the Mac build - @jackyq2015 ?