Investigate php5-curl issues with SNI and redirects

[PVince81]

Whenever an app in the the app store points to a Github URL (or some other servers), some curl on some environments will show:

{"reqId":"zM09Y0\/eIzFoer4ohglK","remoteAddr":"127.0.0.1","app":"core","message":"cURL error 60: SSL certificate problem: unable to get local issuer certificate","level":3,"time":"2015-07-07T13:06:33+00:00","method":"POST","url":"\/owncloud\/index.php\/settings\/ajax\/enableapp.php"}

It happened for me on the following setup:

• openSUSE Tumbleweed
• curl-7.42.1-1.1.x86_64
• php5-5.6.9-1.1.x86_64
• php5-curl-5.6.9-1.1.x86_64

The problem was with the calendar app which redirected to the following URL: https://github.com/owncloud/calendar/releases/download/v0.7.0/calendar.zip which itself seems to redirect to an amazon S3 link.

See owncloud-archive/documentation#1290 and owncloud/calendar#826 (comment)

@LukasReschke

[PolishedStaple]

So do I need to wait until 8.2 is released before my very basic installation of ownCloud with a shared calendar will work? Right now I'm getting the same error as mentioned by PVince81 when I try to enable the calendar app.

Thank you.

[DeepDiver1975]

@nestsman in case you have ssh access to the server you can also manually install apps.

• wget the app from the appstore
• extract the zip/tar into the apps folder
• enable the app in the apps management

[gamanakis]

I face the same problem with FreeBSD 10.1. Could this have to do with an older OpenSSL version? Mine is 1.0.1l with cURL 7.43.

[gamanakis]

I tried with OpenSSL 1.0.2c from ports, recompiled cURL and php5-curl (5.4.42), however this doesn't resolve it.

[ghost]

@DeepDiver1975 tried this with the Calendar app; still getting the cURL error 60 when trying to enable from the web interface.

[josh4trunks]

I thought this had to do with PHP not being able to find a certificate store but my hypothesis was wrong. Playing with php.ini's curl.cainfo, openssl.cafile, and openssl.capath didn't solve the issue.

@gamanakis seems it doens't have to do with old versions of openssl either, as what your testing with is very current.

[PVince81]

My distro where the problem appeared has openssl-1.0.2a-1.1.x86_64
My productive server where it worked properlty has openssl-1.0.1k-11.72.1.x86_64

@Entyrion maybe it's trying to update the app. Did you get the correct version of the calendar app from Github ? It should be 0.7.0.
Have a look in the file "apps/calendar/appinfo/version"

[gamanakis]

I did a git bisect.
Known good commit: 8.1 RC1 d5ef1f2
Known bad commit: 8.1 RC2 e7a30e0
So:
ae3d402 is the first bad commit
commit ae3d402
Author: Joas Schilling nickvergessen@owncloud.com
Date: Mon Jun 29 11:15:56 2015 +0200

Send OC version to appstore so it can reply with the right links

:040000 040000 d3b7bb600bc8ec37ab5afb2f1c22905cc37e898f 2a78d56b5cdc508527d93638b4884ff1720046de M lib
:040000 040000 aeb5c93adbac570d85c97bd11a32efb8ea43c8f1 b93b29d4137a1abdb0e5c034ac166919a9911c73 M settings
:040000 040000 898d6d580c7917f818698d8154be41ec51a6679f 210fae7a1328a18fdb00ab3b8657564b63de14b6 M tests

Could someone verify this?

[PVince81]

I don't see anything related to SSL verification in this commit.
But the parameters seem to have changed. Maybe without the parameters the link that the app store delivered was not from Github, which made it work ? And the new link (for 8.1) points to Github and causes the redirect issue ?

[gamanakis]

False alarm. On my system (FreeBSD 10.1, see above) reversing selectively the above on the official 8.1.0 doesn't produce the error anymore but also doesn't install the apps (contacts and calendar), meaning probably that no connection at all is made to the apps-server.

[ruspantini]

unsuccessful update 8.0.4 to 8.1 #17534 false alarm too. Reinstall worked fine without apps (contacts and calendar). They can't be reinstalled : cURL error 60

[ruspantini]

I just manually installed the calendar and it works !!!
I downloaded here Calendar 0.6.5 for ownCloud 8.1 then Extracted the contents from the downloaded archive to the apps folder. Then navigate to "Apps" with an administrative user, and enabled the app again. This time it worked.
I have to try with Contacts and will report ...

[jankkm]

Yes that works. Not really a solution to the problem though.

[ruspantini]

Well I'm not a developer ... kind of a newbie and I was quite proud of having found a solution and maybe helping you all ;-) still trying with contacts ...

[ruspantini]

It works with Contacts too. I forgot to mention ownCloud 8.1 asks for reinstallation each time. After a quick reinstallation contacts and calendar show up. Have a great evening everyone. I'll be back tomorrow

[PolishedStaple]

I prefer to wait for 8.1.2 to fix the problem. Apparently there is not a QA team in place to catch this kind of stuff before it's released?

[gamanakis]

I think I resolved it for FreeBSD 10.1. In addition to curl, php5-curl and php5-openssl also apache24 has to be compiled against OpenSSL 1.0.2c from ports.

There seems to be a problem with the OpenSSL 1.0.1l installed in the base system regarding SNI. For example:
OpenSSL 1.0.1l base system
/usr/bin/openssl s_client -servername chrismeller.com -connect chrismeller.com:443
returns "Verify return code: 20 (unable to get local issuer certificate)"

while OpenSSL 1.0.2c from ports
/usr/local/bin/openssl s_client -servername chrismeller.com -connect chrismeller.com:443
returns "Verify return code: 0 (ok)"

[josh4trunks]

@gamanakis
thanks, I'll try recompiling everything with "WITH_OPENSSL_PORT=yes" and see if that fixes my use cases.

[gamanakis]

@PVince81

My distro where the problem appeared has openssl-1.0.2a-1.1.x86_64

See: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=201446
"This is a bug with OpenSSL since 1998 and fixed in 1.0.1n..."

[josh4trunks]

I can confirm recompiling everything with openssl 1.0.2d worked for me. I'm not sure what package needs to be compiled against the latest version, but I assume php.

[gamanakis]

Compiling only apache24 against OpenSSL 1.0.2d in FreeBSD 10.1 resolves the issue. I think it has to do with the underlying OpenSSL installation.

[ghost]

@b1ggjoe You need to update OpenSSL and make sure PHP and curl is using that new version as already posted above.

[b1ggjoe]

Yep, I updated and recompiled Apache to 2.4, OpenSSL to OpenSSL 1.0.2d 9 Jul 2015 and cURL to curl-7.43.0-2.0.cf.rhel5.x86_64.rpm. The latest and greatest for all of them.

[ghost]

PHP is missing here as already written above. PHP and also the curl PHP module needs to be linked against the new OpenSSL version as also written above.

[b1ggjoe]

RealRancor,

I think (or I know) I must be doing something wrong. Ok, why is it that when I type in 'OpenSSL version' I get this: OpenSSL 1.0.2d 9 Jul 2015. However, when I take a look at PHPInfo I get this: SSL Version OpenSSL/1.0.1e?

Also, when I type in: curl -V I get this: curl 7.43.0. However, according to PHPInfo I see this: cURL Information 7.38.0.

I know I'm close, but I'm missing something.

BJ

[ghost]

Your PHP and curl PHP module are not linked against your newer (from source built) OpenSSL version as already written (twice) above.

[b1ggjoe]

RealRancor,

That makes sense, thank you. Any ideas on how I can get this corrected?

Is there a step-by-step example that someone can provide? I know this goes outside the scope of ownCloud, but I think such a guide would be greatly beneficial.

This issue seems to be greatly affecting those running on CentOS. CentOS is the OS of choice for many Shared Hosting accounts and etc.

I think if we can get some more direction or some sort of a guide, that would be a huge win for many of us.

BJ.

[ghost]

I know this goes outside the scope of ownCloud, but I think such a guide would be greatly beneficial.

Yes, i think this is far out of the scope of oC and also of this bugtracker. Also most people shouldn't build their stuff from sources and circumvent the updates from their distro, especially not on CentOS/RHEL etc.

[b1ggjoe]

Make sense, but how else can we fix this issue? If someone who knows PHP can whip something together and post it somewhere, this would be awesome. Otherwise, we're reduced to manually installing apps. I guess that isn't so bad...but it just means that we're going to be limping along with a somewhat broken ownCloud Server.

[ghost]

Make sense, but how else can we fix this issue?

Upgrading the system wide OpenSSL library to 1.0.1d+ / 1.0.2b+ via some distro packages (maybe via 3rdparty repos?) or upgrade your base system to a Major Version which is shipping those versions.

[b1ggjoe]

RealRancor,

Understood. I wonder if we should dedicated a new bug or area for this? The issue is further complicated by the fact that my CentOS v6.6 is actually running on a Server with WHM & cPanel, rather than just a plain vanilla Linux Distro install.

This was specifically installed via the 'Softaculous' script., (WHM/cPanel).

Normally, when you're on a Server with WHM/cPanel, unless it's your VPS or Dedicated Server, there aren't many options for you to customize your setup.

When I tried updating/upgrading my various libraries via the traditional route, it came back that there were no updates available.

For example, earlier in this bug tracker it talked about the versions that ship with updated OpenSSL libraries. It states:

CentOS / RHEL

✅ 6 ships with 1.0.1e
✅ 7 ships with 1.0.1e

Well, I already had CentOS version 6.6 with OpenSSL 1.0.1e (as stated above) with the bug fixes.

I then updated to the latest version of OpenSSL: OpenSSL 1.0.2d 9 Jul 2015, yet I'm still having issues.

So I'm not sure what else I can do.

[b1ggjoe]

@PVince81, @josh4trunks, @karlitschek,

Given my specific situation, do you guys have any other ideas or suggestions?

Thank you,

J.

[josh4trunks]

Sorry I don't know anything about CentOS.

On FreeBSD it was as easy as recompiling apache, PHP, and it's extensions using the latest openssl.

[LukasReschke]

CentOS also uses NSS in a buggy version. This is another bug. I'm on my mobile at the moment and can't reference the issue. But basically there is since 1 year an issue open that nobody cares to fix ;-)

Well. There is a fix. Just all those Enterprise distributions are using totally old versions and won't backport fixes.

So either get a new NSS version and recompile it, install apps yourself without the app store or wait until we deployed the app store to a non SNI setup.

[LukasReschke]

Or get a RHEL Subscription and throw enough money at RedHat ;-)

[b1ggjoe]

@LukasReschke,

Thank you very much for the info. I totally forgot about the whole NSS issue and CentOS. Ok, I will try to get a new NSS version and recompile and see if that fixes it.

If not, then Manual App install it is for the time being.

More to come...

BJ

[b1ggjoe]

@LukasReschke,

Well it looks like I'm out of luck:

nss-softokn-freebl-3.14.3-22.el6_6.x86_64 already installed and latest version

So for now, I will just park the issue until there are further developments or someone else finds a way to fix this.

In the mean time, I'll just do manual installation of apps.

BJ

[Siedlerchr]

I don't know which linux my hoster (shared hosting) exactly uses but I am also affected by this problem.
This are the details which are displayed in cPanel:

Apache-Version 2.2.29
PHP-Version 5.4.40
MySQL-Version 5.6.23
Architektur x86_64
Betriebssystem linux
Kernel-Version 2.6.32-042stab108

[PolishedStaple]

I just installed 8.1.1 thinking it would resolve the issue, "cURL error 60: See http://curl.haxx.se/libcurl/c/libcurl-errors.html", but it did not. The change log seemed to indicate this was one of the fixes. I do use centOS on my VPS server. Is there anything that can be done besides manually installing apps?

Thank you.

[ghost]

@nestsman ownCloud 8.1.1 isn't fixing an issue you're having with outdated/buggy libs on CentOS. It only shows in your admin backend if you're affected as also described here:

https://doc.owncloud.org/server/8.1/admin_manual/configuration_server/security_setup_warnings.html#outdated-nss-openssl-version

There you can also see the info what to do to fix this issue.

[ergillette]

Wow. . .well this is a drag. I just ran into the same issue. Perhaps I need to start a company that pulls app developers and OS developers together. I'm sure I'm not the only one that finds it odd that basically OwnCloud is saying the problem is CentOS and then CentOS says OwnCloud can fix it.

That leaves the people who use OwnCloud in a fundamentally unresolved situation. . .that is the fastest way to lose your user base.

I mean heaven forbid someone comes along and develops an app that does what OwnCloud does but doesn't have the shortcomings of OwnCloud. . .I suspect that would make for a very interesting situation.

I wonder if OwnCloud Enterprise has this problem. . .

[ghost]

@ergillette please scan through this complete issue and you will see that the whole issue is caused by your distro shipping buggy libraries. oC itself only could fix this by using either insecure transport protection (http) or don't use a common used technology (SNI). None of those should be needed if you're using a distro shipping current non-buggy libraries.

[ergillette]

@RealRancor

I did that already.

I don't disagree with you -- but my point is this is surely something you guys can easily resolve rather than pointing people to CentOS and saying: "Let them fix it."

I mean call me crazy if you'd like, but there are like 1,000 ways to grab data without using CURL.

Like the one guy majority of my machines and clients run WHM/cPanel which compounds the problem.

I used to wonder why OwnCloud doesn't have the same traction as DropBox or Box.com for example, but when I see issues like this it becomes glaringly obvious.

Sure I can download and install the apps manually using SSH, but what about the other folks who are NOT technical enough to do that -- they pick up and go with another solution without even taking the time to look up the error message or even write a message here to let anyone know there was a problem.

It really just comes down to a development team's preference of obtaining data (i.e. curl versus PHP's built-in file streaming capabilities with other libraries besides curl) I mean. . .I'm not even sure why an app has to be downloaded using a secure connection in the first place.

Sounds like a cannon was used to kill the mosquito when the flyswatter would have worked just fine.

But OwnCloud wouldn't be the first solution that's been plagued with simple issues like this that could be easily corrected and ends up causing it to become an obscure application that gets beat out by an application that introduces the practicality that the majority of people expect.

Personally. . .if you guys want to leave it broken, that's your prerogative, but you should probably at least consider having the app provide a more detailed error message rather than the one it does that led me here.

I researched because I'm a server administrator -- imagine the users who won't do that and simply abandon the app because they get stuck.

Just some food for thought. . .you folks do what you wanna do.

Trust me it'll be a matter of time before an app comes along that does what OwnCloud does and provides better error reporting, or takes a more concerned approach to solving the problem -- and just like that OwnCloud will lose traction to that application nearly overnight.

It always happens that way. I'd love to see it not happen, but when this is the approach you guys take, it's bound to go the way other applications that were plagued by similar issues have gone. ;-)

Just my 3 cents. . .for now I'll just install the apps manually via SSH for the countless other people who didn't even make it to this page and just abandoned the app instead.

[chrisice]

I can't say if it's a good workaround or not, but I was able to move the config/ca-bundle.crt file that is provided with OwnCloud out of the way, then get the one that is in a Wordpress installation in the wp-includes/certificates directory and put it in it's place. That allows it to work and I haven't noticed any issues from it yet

[andrewnazarov]

@chrisice
Same story here(fully updated openSUSE 13.2, owncloud 8.1.3). The only difference is that I got ca-bundle.crt file from https://github.com/bagder/ca-bundle/blob/e9175fec5d0c4d42de24ed6d84a06d504d5e5a09/ca-bundle.crt. Then just replaced ca-bundle.crt provided by owncloud(located here: /srv/www/htdocs/owncloud/config) with the one from github. That solved the problem with curl error 60. Downloaded file is the bundle that had been converted before "weak" certificates(that were still using RSA-1024 keys) was removed.