Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

longjmp causes uninitialized stack frame ***: sidekiq 2.11.1 diaspora [25 of 25 busy] terminated #4202

Closed
movilla opened this issue Jun 5, 2013 · 42 comments

Comments

Projects
None yet
@movilla
Copy link
Contributor

commented Jun 5, 2013

@jhass

This comment has been minimized.

Copy link
Member

commented Jun 6, 2013

Mmh, looks like you want to rebuild curl (and then reinstall the ethon gem) with c-ares support: http://stackoverflow.com/questions/9191668/error-longjmp-causes-uninitialized-stack-frame

@movilla

This comment has been minimized.

Copy link
Contributor Author

commented Jun 6, 2013

Oks. Solved ;]

I onlyy install c-ares http://curl.haxx.se/dev/readme-ares.html

@movilla movilla closed this Jun 6, 2013

@dmorley

This comment has been minimized.

Copy link
Contributor

commented Jul 5, 2013

@movilla You just installed e-ares and not recompile curl and that was the fix? I have this same issue now, seems debian pulled c-ares out of curl in repos.

@groovehunter

This comment has been minimized.

Copy link

commented Aug 21, 2013

I had the same issue and I did (on ubuntu precise)
apt-get purge libcurl3 (removes also libcurl4-openssl-dev)
installed c-ares ,
in libcurl added symlink as stated in README
configured libcurl with
./configure --prefix=/usr --enable-ares
make && make install
and then
dpkg -i --ignore-depends=libcurl3 /var/cache/apt/archives/libcurl4-openssl-dev_7.22.0-3ubuntu4.2_amd64.deb

now sidekiq running for >10 minutes ... seems fine

@jhass

This comment has been minimized.

Copy link
Member

commented Aug 27, 2013

For reference with curl 7.32 Debian enabled the threaded resolver which also solves this problem. It is currently in unstable. For more informations see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=570436#67 and the following message.

A more in depth explanation of the issue can be found here: http://stackoverflow.com/questions/9191668/error-longjmp-causes-uninitialized-stack-frame

@ghost

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2013

Today I installed a new wheezy box and problem still appears on wheezy with curl 7.26 and old libcurl - we should add this information to https://wiki.diasporafoundation.org/Installation/Debian/Wheezy

@ghost

This comment has been minimized.

Copy link
Contributor

commented Aug 30, 2013

@MrZyx On wheezy I tried curl 7.32 from jessie but without success.

Workaround for Debian Wheezy:

Install c-ares

  1. unpack the c-ares archive
  2. cd c-ares-dir
  3. ./configure
  4. make
  5. make install

Remove Debian packages

  1. apt-get purge curl
  2. apt-get purge libcurl3 with dependencies

Install curl with c-ares

  1. unpack the curl archive
  2. cd curl-dir
  3. ./configure --enable-ares=/usr/local/lib
  4. make
  5. make install

Check curl version
curl --version
curl 7.32.0 libcurl/7.32.0 OpenSSL/1.0.1e zlib/1.2.7 c-ares/1.10.0 libidn/1.25 libssh2/1.4.2 librtmp/2.3

@jaywink

This comment has been minimized.

Copy link
Contributor

commented Aug 30, 2013

Got hit by this too :P There is also an Ubuntu bug on Launchpad (https://bugs.launchpad.net/ubuntu/+source/curl/+bug/613274) - hopefully 7.32 will be packaged there too soon to some PPA.

@jaywink

This comment has been minimized.

Copy link
Contributor

commented Aug 30, 2013

Btw, why is this closed? Following the current installation instructions this bug will affect afaik many distros at least - I don't think having a workaround is enough to have it closed until the issue is fixed upstream and our instructions take this bug in to account.

@jaywink

This comment has been minimized.

Copy link
Contributor

commented Aug 31, 2013

OK I pushed the upstream debian 7.32.0 into a PPA since none existed yet, so if you want to test the new curl version on Ubuntu 12.04 onwards do this;

sudo apt-add-repository ppa:jaywink/curldebian
sudo apt-get update && sudo apt-get upgrade
@Raven24

This comment has been minimized.

Copy link
Member

commented Aug 31, 2013

reopening for tracking the upstream issue, but there is not much we can do from the D* side, code-wise

@Raven24 Raven24 reopened this Aug 31, 2013

@dbaber

This comment has been minimized.

Copy link

commented Dec 7, 2013

I haven't made it all the way through the install (yet), but if you follow the guide for Debian Wheezy, you should already have a version of libc-ares2 and libc-ares-dev installed? (or you can install them if needed) and just be able to build curl from source against it:

Remove Debian packages

  1. apt-get purge curl
  2. apt-get purge libcurl3

Install curl with c-ares

  1. wget http://curl.haxx.se/download/curl-7.33.0.tar.gz
  2. tar zxvf curl-7.33.0.tar.gz
  3. cd curl-7.33.0.tar.gz
  4. ./configure --enable-ares
  5. make
  6. make install
  7. run ldconfig -v to cache libcurl.so.4

Check curl version
curl --version
curl 7.33.0 (x86_64-unknown-linux-gnu) libcurl/7.33.0 OpenSSL/1.0.1e zlib/1.2.7 c-ares/1.9.1 libidn/1.25 libssh2/1.4.2 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP

===========UPDATE==========
I have it all working now as well and see no issues with sidekiq

@jaywink

This comment has been minimized.

Copy link
Contributor

commented Feb 20, 2014

I think we should make d* warn podmins on launch if curl version is lower than 7.32? This bug will cause confusion for a while yet.

For example this long thread involving many podmins suffering from the problem.

@goobertron

This comment has been minimized.

Copy link

commented Feb 20, 2014

Should I do a DHQ announcement? If so, please give me some draft text because I don't really understand the issues and don't want to make an inaccurate announcement.

@jaywink

This comment has been minimized.

Copy link
Contributor

commented Feb 20, 2014

@goobertron actually what I meant was a "**** Warning! Your curl version sucks" kind of thingy at the start of ./script/server, with a link to this issue - if the system has an old curl. Maybe with a slightly different text than sucks ;)

Can do a pull when I have time.

@ghost

This comment has been minimized.

Copy link

commented Feb 20, 2014

This is a quick fix for Debian 7/Wheezy. It is not a long term solution, but should get us past the version issue.

This uses Jason Robinson's PPA/Repo, and the Debian Sid Repo in combination to get the correct files.

Add the following to your /etc/apt/sources.list :

deb http://ppa.launchpad.net/jaywink/curldebian/ubuntu precise main
deb-src http://ppa.launchpad.net/jaywink/curldebian/ubuntu precise main
deb http://ftp.us.debian.org/debian/ sid main
deb-src http://ftp.us.debian.org/debian/ sid main

Then do :

apt-get update

Ignore any errors.

Then do this :

apt-get install libc6

Finally, this :

apt-get install curl libcurl3 libcurl3-gnutls libcurl4-openssl-dev

Then remove the lines from your /etc/apt/sources.list

That got me to version 7.35.0

@goobertron

This comment has been minimized.

Copy link

commented Feb 20, 2014

@jaywink, sure for the future. I'm just thinking about helping out any podmins who are having problems now. I've seen several pods report federation problems this evening, and one of them fixed it by updating curl. See https://joindiaspora.com/posts/3669087 . Perhaps we should alert all podmins who might be affected as soon as possible, unless I have misunderstood the issue.

If it only affects pods running on Debian/Wheezy I could link to this thread for Star's quick fix.

@Flaburgan

This comment has been minimized.

Copy link
Member

commented Feb 21, 2014

@goobertron I'll do a post with dhq tomorrow ;)

@jaywink

This comment has been minimized.

Copy link
Contributor

commented Feb 21, 2014

Well I'm not entirely sure we need to post via the DHQ account ... or if you do make sure to make it clear that only podmins need to read it :P

Also it's not only Debian/Wheezy - AFAIK it's any system with an older curl. For example the current Ubuntu LTS 12.04 has an old curl. But I don't know enough of the upstream issue whether everybody with an older curl will be affected or just some systems with an older curl.

If you make a post, just make sure to point that podmins should be aware, don't say "federation will not work unless" :) Federation works, sidekiq just is unstable.

@Flaburgan

This comment has been minimized.

Copy link
Member

commented Feb 21, 2014

Is that okay? (From an english point of view too :p)

Many podmins encountered difficulties with sidekiq suddently stopping to work. If this is your case, please check that you are using at least curl 7.32 (the default version on Ubuntu 12.04 and Debian Wheezy is 7.22 and is too old). We observed many instabilities with old version of curl.
To upgrade from Ubuntu, you can use a ppa @jaywink created and run

sudo apt-add-repository ppa:jaywink/curldebian sudo apt-get update && sudo apt-get upgrade

For debian please referer to this comment

@goobertron

This comment has been minimized.

Copy link

commented Feb 21, 2014

How about:

Notice for podmins only:
Sidekiq requires curl 7.32 or later to function properly. If you have been experiencing any performance issues recently, please check your version of curl. Some distros, for example Debian Wheezy, still include an older version.
If you need to update curl, [then add the instructions]

ps: in English, it's 'please refer' not 'referer'.

@jaywink

This comment has been minimized.

Copy link
Contributor

commented Feb 21, 2014

Awesome, +1

@goobertron

This comment has been minimized.

Copy link

commented Feb 21, 2014

I suppose as we know it's been affecting federation, it should read:

If you have been experiencing any performance issues recently (especially concerning federation), please check your version of curl.

@jaywink

This comment has been minimized.

Copy link
Contributor

commented Feb 21, 2014

Well the real issue is actually federation stopping completely :) Since sidekiq dies. So it's neither performance or federation issues ;)

@ghost

This comment has been minimized.

Copy link

commented Feb 21, 2014

Well, I think perhaps its safe to say it has been affecting federation, though its not the federation itself at fault. Mine and at least 2 others have been affected. When sidekiq crashes, there is no federation ;)

Perhaps word it something like this:

curl versions lower than 7.35 are known to cause issues with sidekiq.
This in turn affects your pods ability to federate.
It is highly advised that you upgrade as soon as possible, to maintain
the smooth running of your pod.
For Ubuntu, the PPA is here: 
https://launchpad.net/~jaywink/+archive/curldebian
For Debian, the instructions are here:
https://github.com/diaspora/diaspora/issues/4202#issuecomment-35676203
@ghost

This comment has been minimized.

Copy link

commented Feb 21, 2014

If this fixes such a large problem, I think it would a good idea to really push this to the podmins. They really need to know.

A side note : This problem does not affect Debian Sid - I double checked.

@Flaburgan

This comment has been minimized.

Copy link
Member

commented Feb 21, 2014

When sidekiq crashes, there is no federation ;)

There is also no notification, etc... :p

@Flaburgan

This comment has been minimized.

Copy link
Member

commented Feb 21, 2014

@goobertron I post the message with dhq

@jaywink

This comment has been minimized.

Copy link
Contributor

commented Feb 21, 2014

The message from @starblessed is good imho - wanna send it out @Flaburgan ?

@Flaburgan

This comment has been minimized.

Copy link
Member

commented Apr 14, 2014

I just wanted to point that I received today an update for curl on Ubuntu 13.10, we now have the 7.32 version by default! If someone with an Ubuntu 12.04 can see if the update is available there too...

curl --version
curl 7.32.0 (x86_64-pc-linux-gnu) libcurl/7.32.0 OpenSSL/1.0.1e zlib/1.2.8 libidn/1.28 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP

@jaywink

This comment has been minimized.

Copy link
Contributor

commented Apr 15, 2014

@Flaburgan 13.10 has always had 7.32 IMHO, I'm quite sure about that.

But below 13.10 - it's older and you need to use for example the PPA to get 7.32. Check here: http://packages.ubuntu.com/search?keywords=curl

Btw, if no one objects, I'm closing this issue since 1) upstream has fixed it, it's just that some distros still package old curl and 2) in #4885 we've added a warning to ./script/server to warn of missing AsynchDNS feature, pointing to this issue.

@jaywink jaywink closed this Apr 15, 2014

@Flaburgan

This comment has been minimized.

Copy link
Member

commented May 5, 2014

The current version in wheezy is 7.26, what's the cleanest way to upgrade to at least 7.32 in the solution listed here?

@jaywink

This comment has been minimized.

Copy link
Contributor

commented May 7, 2014

Would this help?
#4202 (comment)

@Flaburgan

This comment has been minimized.

Copy link
Member

commented May 7, 2014

@jaywink #4202 (comment) would help too, I didn't try them now to know which one I should do.

@lazzarello

This comment has been minimized.

Copy link

commented Sep 27, 2014

Dang, building an unofficial backport on Wheezy isn't easy because the version in sid depends on a new libc6. I guess we're stuck compiling for now or upgrading to jessie.

@lazzarello

This comment has been minimized.

Copy link

commented Sep 28, 2014

hey everyone. I made a test backport of the curl_7.38.0-2.dsc source package from Sid on Wheezy but I had to change some of the dependencies in debian/control. The packages compile clean and the version looks good, and explicitly reports the AsynchDNS feature. worth noting that this took a really long time (over an hour) to build on a newly created dual core 2Ghz Linux container with 2GB RAM.

Proof:

# cat /etc/debian_version 
7.6
# /usr/bin/curl --version
curl 7.38.0 (x86_64-pc-linux-gnu) libcurl/7.38.0 OpenSSL/1.0.1e zlib/1.2.7 libidn/1.25 libssh2/1.4.2 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API SPNEGO NTLM NTLM_WB SSL libz TLS-SRP

Build artifacts directory listing

drwxr-xr-x 1 root root     632 Sep 27 11:38 curl-7.38.0
-rw-r--r-- 1 root root  205384 Sep 27 11:38 curl_7.38.0-2_amd64.deb
-rw-r--r-- 1 root root   26568 Sep 23 15:33 curl_7.38.0-2.debian.tar.xz
-rw-r--r-- 1 root root    2625 Sep 23 15:33 curl_7.38.0-2.dsc
-rw-r--r-- 1 root root 4094034 Sep 10 19:09 curl_7.38.0.orig.tar.gz
-rw-r--r-- 1 root root  289754 Sep 27 11:38 libcurl3_7.38.0-2_amd64.deb
-rw-r--r-- 1 root root 3808724 Sep 27 11:38 libcurl3-dbg_7.38.0-2_amd64.deb
-rw-r--r-- 1 root root  280416 Sep 27 11:38 libcurl3-gnutls_7.38.0-2_amd64.deb
-rw-r--r-- 1 root root  293510 Sep 27 11:38 libcurl3-nss_7.38.0-2_amd64.deb
-rw-r--r-- 1 root root 1116674 Sep 27 11:38 libcurl4-doc_7.38.0-2_all.deb
-rw-r--r-- 1 root root  408508 Sep 27 11:38 libcurl4-gnutls-dev_7.38.0-2_amd64.deb
-rw-r--r-- 1 root root  423100 Sep 27 11:38 libcurl4-nss-dev_7.38.0-2_amd64.deb
-rw-r--r-- 1 root root  419620 Sep 27 11:38 libcurl4-openssl-dev_7.38.0-2_amd64.deb

I'm not sure if I should publish these binaries since they haven't been tested. I'm going to install them on my production pod at https://diaspora.openhosting.com and cross my fingers. Here's the Build-Depends of my control file. This is the only modification I made other than installing the build deps.

Build-Depends: debhelper (>= 9),
 autoconf,
 automake,
 ca-certificates,
 groff-base,
 libgnutls-dev,
 libidn11-dev,
 libkrb5-dev,
 libldap2-dev,
 libnss3-dev,
 librtmp-dev,
 libssh2-1-dev,
 libssl-dev,
 libtool,
 openssh-server,
 python,
 quilt,
 stunnel4,
 zlib1g-dev

If you want to build and test this backport yourself, it's pretty easy but don't do the build on your production server. I suggest creating a VM or Linux Container (Docker is quite useful for something like this). Taken from the Debian simple backports guide.

Add the following to /etc/apt/sources.list

deb-src http://ftp.us.debian.org/debian sid main contrib non-free

Install package build toolchain. Update and download the source package. Check for build deps and install if necessary.

# apt-get update
# apt-get install devscripts build-essential debian-keyring dpkg-dev
# apt-get source curl
# cd curl-7.38.0
# dpkg-checkbuilddeps

If you get output, apt-get install all the packages listed.

# debian/control binary
# cd ..
# apt-get remove --purge curl libcurl3
# dpkg -i libcurl3_7.38.0-2_amd64.deb libcurl3-gnutls_7.38.0-2_amd64.deb libcurl3-nss_7.38.0-2_amd64.deb curl_7.38.0-2_amd64.deb
# /usr/bin/curl --version

Now you can copy these .deb files to any Wheezy system and install them with the same dpkg command above.

Good luck!

@lazzarello

This comment has been minimized.

Copy link

commented Sep 28, 2014

Update, I ran into some snags on my production system. I didn't have two required packages installed thus I couldn't manually install libcurl3-nss_7.38.0-2_amd64.deb through dpkg. I fixed it with the following.

# apt-get remove libcurl3-nss
# apt-get install libnspr4 libnss3
# dpkg -i libcurl3-nss_7.38.0-2_amd64.deb

that should get you through the dependency conflicts. Since you have kind of messed with your package database in a weird way, you should probably put these packages on hold so you don't accidentally break them with an apt-get upgrade. That's easy, with the following.

# echo "curl hold" | dpkg --set-selections
# echo "libcurl3 hold" | dpkg --set-selections
# echo "libcurl3-gnutls hold" | dpkg --set-selections
# echo "libcurl3-nss hold" | dpkg --set-selections

To remove them from hold replace the word "hold" with "install".

@Raven24

This comment has been minimized.

Copy link
Member

commented Oct 4, 2014

just for the record,
I just came across the longjmp issue on a CentOS 6.5 machine with curl 7.19.7 (from the repos). compiling the latest curl 7.38.0 from source seems to works ok.

@ldidry

This comment has been minimized.

Copy link

commented Oct 11, 2014

@lazzarello How's your backported curl working? 14 days after its installation, do you think it may be good enough for releasing?

@lazzarello

This comment has been minimized.

Copy link

commented Oct 11, 2014

Heya, yeah I was thinking about this last night. Nothing looks weird on my server but I haven't had time to set up a proper APT repo. In the mean time, the instructions I posted above will get you a binary. So you don't have to wait on me.

@ldidry

This comment has been minimized.

Copy link

commented Oct 12, 2014

Thank you 😄

What I wanted the most is if it was worthwhile to backport the packages. Since it is, I will do it.

@norayr

This comment has been minimized.

Copy link

commented Sep 4, 2015

in my funtoo i had this warning, after system upgrade curl was recompiled with "threads" flag which forces "-adns". well i recompiled curl with "adns -threads" and i hope it will solve the issue. thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.