-
-
Notifications
You must be signed in to change notification settings - Fork 2k
fetching URLs: Received fatal alert: handshake failure #1265
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
I think this user found a fix - it'd be nice to know if there anything we can do upstream to prevent that problem: |
Hmmm. I installed from DMG per docs. If there's java config that needs doing for non-developer users, I missed it in the docs. Pointer? |
I have Java 1.8.0_144 installed. |
Using I have tried launching OR using the |
The problem (or at least part of the problem) seems to be with the java encryption suite. This is different to the solution linked above. Connecting with: This encryption isn't supported in the default Java 8 encryption suite. (a nice way to check what is being used by OpenRefine - add the URL "https://www.howsmyssl.com/a/check" to an OpenRefine cell and use Add column from URL to retrieve the JSON :) To support this encryption you need to install the JCE - Java Cryptography Extension http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html I'd copied the jars from that download into jre/lib/security/ (replacing the existing versions). I was able to get this working (although by then I'd already installed additional certificates, so I'm not able to confirm whether you need to do both steps, or just the JCE step - I suspect just the JCE will be fine). At this point I now get a HTTP error 403 : Forbidden from https://pleiades.stoa.org/places/540867/json I believe this to be down to the due to the User Agent being specified by OpenRefine. If I curl: I also get a 403 forbidden from this website. At the moment it is not possible to change the User Agent used by OpenRefine. See #1217 "The User Agent OpenRefine uses is currently determined by the version of the underlying Java library" |
@ostephens, regarding the 403s: yes, I believe you are correct. We have apache config that rejects certain default and blankish user agent strings because of bad behavior by some bots in the past.
Regarding the apache rewrite rules: If we don't see significant performance degradation tied to anonymous bots, we might leave the exception in place, but ideally I'd like to see automated agents using specific user agents and so will follow the discussion on #1217. |
This restriction was causing problems with OpenRefine (see OpenRefine/OpenRefine#1265)
@ostephens unfortunately I cannot yet duplicate your success. A diagnostic question:
|
@paregorios I was using the Linux version of the latest code on Mac, running from the command line (and with an additional tweak and rebuild to set a recognised user agent successfully retrieved JSON from Pleiades) Just tried with the OS X production 2.7 distro and found I got the handshake error again :( So now trying to debug that ... |
@paregorios OK - success. It looks like if you run the Mac version it uses the Java cacerts from /Applications/OpenRefine.app/Contents/PlugIns/jdk1.8.0_60.jdk/Contents/Home/jre/lib/security (this is mentioned in https://groups.google.com/forum/#!msg/openrefine/6adewVTQh4Y/gsJdepAmAAAJ;context-place=forum/openrefine, although I definitely have $JAVA_HOME set, so I wonder if there is an issue with the Mac build in this respect). Anyway, so I copied the JCE local_policy.jar and US_export_policy.jar into this directory, but then got the "sun.security.validator.ValidatorException: PKIX path building failed" error, which means that it is missing a certificate. So I then installed the cert pleiadesstoaorg.crt I got from my browser by visiting pleiades.stoa.org - but that still didn't resolve the issue. So then I installed the root cert from Lets Encrypt (obtained from https://letsencrypt.org/certificates/) using: And Success! |
Yeah, so it looks like the OR 2.7 distro for OSX bundles jdk 1.8.0_60 and ignores the $JAVA_HOME variable. It seems to be hard-coded in |
According to this, the "DST Root CA X3" certificate, which is the parent to the "Let's Encrypt" certificate was only added to the default JDK/JRE keystore with versions 7u111 and 8u101. So, bundling jdk 1.8.0_101 or later might take care at least of the certificate problem (and not just for Pleiades). |
@ostephens thanks for the roadmap. I am now fetching Pleiades JSON with my hacked version of the OSX OR 2.7 distro. Adding the pleiades cert (which was part of your debugging process, I realize) wasn't necessary, so long as the root cert from Let's Encrypt has been added. This resolves my immediate issue, but -- if I may be so bold -- would seem to leave open some steps for the OR development community to explore with respect to the (next) OSX release. This workaround is probably too techno-fiddly for most of the users to whom I'd like to be able to recommend OpenRefine for Pleiades reconciliation. I am happy to help troubleshoot and/or test, if that would be useful, but I don't have java development skills. |
@paregorios, can I say the root cause is OR 2.7 for OSX ignore the $JAVA_HOME? or OR will not well with Pleiades with jdk 1.8 regardless the OS due to the missing root CA issue ? |
@jackyq2015 don’t think getting OR on Mac to observe $JAVA_HOME will help with this issue specifically- it just pushes the issue back to a local system java issue rather than a bundled java issue. There are (I think) two issues:
Neither of these are necessarily fixed if we use $JAVA_HOME - it would then just depend on how the local Java was setup. I wonder if using a bundled java might be a feature rather than a bug. If we can bundle a version of Java with updated cacert and the policy jars - which would fix both issues. |
@ostephens , I checked the folder under /Applications/OpenRefine.app/Contents/PlugIns/jdk1.8.0_60.jdk/Contents/Home/jre/lib/security after installing 2.7RC2. I did find the 2 jars you mentioned. For the linux box and openjdk, I do see the entry:
Next time we release, we can try to bundle the 1.8.0_101 or later as @ostephens suggested. Hopefully that will solve the issue. |
@jackyq2015 just to note - as far as I know, the local_policy.jar and US_export_policy.jar in the Java distribution is not sufficient for this to work. You need to use the ones from the Java Cryptography Extension (JCE) available at http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html for the decryption to work. The policy jar in the bundled java would need to be updated to be the JCE ones |
@ostephens I gotta ask... How the hell do other Java applications that run on Mac handle all this ? What do they do now compared to how things were when this article was written by Oracle https://docs.oracle.com/javase/7/docs/technotes/guides/jweb/packagingAppsForMac.html ? My thoughts are that we don't bundle...but instead provide nice scripts for Mac users to run, to make @paregorios problem... the Cert installation or whatever on Mac for OpenRefine usage... EASIER. Let do that instead. (decouple the issue from our build process, so we don't have to alter our build process) |
According to this, there are no AES256 cipher suites offered by current bundle java client. My understanding is that in order to make it work, we have several options:
|
@thadguidry that's a really good question! I don't know, but I think this blog post might describe the same issue https://blog.spaziodati.eu/en/2014/11/17/the-dark-side-of-ssl-on-java-virtual-machine-jvm/ - so if I'm right, we aren't alone in seeing some issues around this. Having thought about it a bit, on balance I'm in favour of what @thadguidry suggests - avoid bundling the JCE stuff (some slight nagging worry in the back of my mind about legal issues related to this as well), but offer better support. I think @jackyq2015 point about whether @paregorios could make a change at Pleiades to avoid the problem in this particular case is also a good one. This Stackoverflow post seems to have a lot of good stuff in - we could even consider testing for the issue and falling back (or just preferring) the suggested Bouncy Castle cryptography library/API that could help avoid the problem https://stackoverflow.com/questions/1179672/how-to-avoid-installing-unlimited-strength-jce-policy-files-when-deploying-an |
Hi |
@msaby which version of OpenRefine are you using? |
@ostephens At the end of the day, I'd like you to just close this issue and then provide some good documentation, avoidance, best practice, etc... on our Wiki to help navigate this kind of problem with Server side security when using our Fetch URLs function against a server using HTTPS. We can then point folks to that page to try different options, and advise of security concerns as well. A Wiki page is a much better way to deal with this very niche "area" (its not an issue) in particular. Which really OpenRefine dev / support team should not be getting into the weeds with. Let's call that Wiki page "Fetching URLs with HTTPS" or whatever. Thanks All. |
@ostephens When you say "using a bundled java", would that mean that Open Refine would become totally portable and would not need to install anything else to work? If so, it might interest some users, I believe. |
@ettorerizza The problem still might exist. Users in constrained IT environments might not see OpenRefine being allowed to start because JRE or JDK is caught as suspicious by a corporate security program running on their computer. We cannot help that however, so its a non-issue for us. I think I would be OK with a bundled JRE, and allowing folks to still override the bundled JRE to that of their $JAVA_HOME The problem that I don't know about on OSX is about Signing ? Is it a requirement at the OSX level or is it just a requirement for Apple Store ? In other words, does the OpenRefine+JRE bundled app on OSX need to be signed somehow or not ? |
@thadguidry self-signed cert is not mandatory from OSX level. Since we are not distribute the software to apple store, that not mandatory either(actually, if you want to distribute to store, you need a developer certificate signed from apple). AFAIK, The reason we self-signed it is to avoid some warning popup for security from the OSX. And for some system, OSX will hide the icon when it is not self-signed. Basically it's poor man's choice to try to align with the security framework from apple. :). code signing is to sign the distribution as whole to assure the identify of developer does the release, which means it does not matter you bundle something or not. |
I've now added a new page on troubleshooting problems fetching data from URLs, putting in the problems noted in this issue and the solutions at https://github.com/OpenRefine/OpenRefine/wiki/Troubleshooting:-Fetching-data-from-URLs As noted by @paregorios some of the solutions are technical, and will be challenging for a user without experience of using the command line or similar level of technical knowledge. I've also confirmed through testing that if you use OpenRefine for Mac, the Java cacerts and encryption policy files that are important are the ones that are in the JRE included in the distribution (e.g. in my current install of OR2.8, this is /Applications/OpenRefine.app/Contents/PlugIns/jdk1.8.0_60.jdk/Contents/Home/jre/lib/security/ although the path can obviously vary across installs and versions) This is as far as I'm able to take it in terms of the security certs - we haven't identified a good/easy solution for ensuring that users don't encounter problems here. I think @thadguidry's suggestion of a separate script to do the necessary updates is probably the right approach, but I don't know how to start approaching this. It would be good to update the Java version which is included in the Mac version at some point, but I think that's a connected but separate issue. Finally #1217 is also related and outstanding - and I think is something we can do something about - I'm planning on having a look at how we might allow the user to specify request headers including user-agent soon. Closing this specific issue now. |
@tonero101 Has a Mac and might be able to throw together a script to make this easier. He's working on the JDBC enhancement also. Tony what do you say, can you tackle a script for Mac users to make cacerts installation and management easier ? @ostephens @jackyq2015 We need to drop Java7 now and go head first into Java8+ Do we have a issue labeled as "task" for that yet ? |
@thadguidry To be clear, the Java version distributed in the Mac package is JDK 1.8.0_60 - so already on Java 8 - just would be good to update this to a more recent 1.8.X at some point which could improve the cacert situation on Mac - but of course Win and Linux (afaik) would depend on the local Java install still, so it's a minor issue I think. I'm not clear what determines the version of Java included in the Mac build - @jackyq2015 ? |
This restriction was causing problems with OpenRefine (see OpenRefine/OpenRefine#1265)
OpenRefine 2.7 (1) on Mac OSX 10.11.6 with Google Chrome 61.0.3163.100 (Official Build) (64-bit).
Attempting to "Edit column" -> "Add column by fetching URLs ..." one gets: "Received fatal alert: handshake failure" for all URLs tried in the
https://pleiades.stoa.org/
domain.but
curl
ing the urls on same platform has no problems, e.g.:The text was updated successfully, but these errors were encountered: