Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error in `Passenger core': malloc(): memory corruption #2089

Closed
HimanshuM opened this issue Jun 12, 2018 · 18 comments

Comments

@HimanshuM
Copy link

commented Jun 12, 2018

Open source Passenger 5.1.2/Nginx, installed through Rubygems with RVM
Linux 4.4.0-128-generic #154-Ubuntu SMP
Ruby 2.1.8
Rails 4

While restarting Nginx, Passenger core seems to be reluctant to start. It gets stuck in a loop of attempting to restart the core. After maybe, an hour or so, the Passenger is up and stable. But as soon as I restart Nginx, the process repeats.
Tried looking for it over the internet, could not find the exact same issue anywhere.

This started since today morning, apparently randomly. And restarting the server is not helping either.
I have some 1200+ crash dumps on my server by now...

I have attached the log, logged at level 7. Hope it helps figuring out my issue...
passenger-crash-log.1528827885.txt

@sixfeetover

This comment has been minimized.

Copy link

commented Jun 12, 2018

This is happening for us as well, across multiple different servers. Our stack trace implies it is related to the phone home for security updates as it fails establishing a connection to 159.69.16.82, which appears to be Passenger's security check service. Our log looks effectively the same as @HimanshuM's.

Edit: This started just today, after the release of 5.3.2.

@CamJN

This comment has been minimized.

Copy link
Contributor

commented Jun 12, 2018

This is likely due to an issue fixed in 5.1.6 where once enough security advisories had built up and the total length of the response from the security update check server exceeded ~500B then Passenger would crash. You can update to the latest version of Passenger to receive this fix, or disable the security update checker if you aren't going to use it.

edited to fix: version where this was addressed.

@tsu-shiuan

This comment has been minimized.

Copy link

commented Jun 12, 2018

Disabling the security update checker fixed this for us. We're on 5.1.2. Thanks @CamJN !

@sixfeetover

This comment has been minimized.

Copy link

commented Jun 12, 2018

Most of our instances are using 5.1.5, including the ones experiencing this issue. However, we will go ahead and disable it and work on deploying 5.3.

@kotcity

This comment has been minimized.

Copy link

commented Jun 12, 2018

There's going to be a lot of crashing Passengers out in the wild. Any chances of keeping the security info short to avoid crashing and then supply a new URL for new Passenger versions?

@HimanshuM

This comment has been minimized.

Copy link
Author

commented Jun 12, 2018

Hi @CamJN , thank you for your quick response. Disabling the security update check fixed the problem for now. I guess, we needed a push to perform the upgrade to a newer version :)

@HimanshuM HimanshuM closed this Jun 12, 2018

@kotcity

This comment has been minimized.

Copy link

commented Jun 12, 2018

@CamJN is there the possibility of having the securitycheck server NOT send large output to these older affected versions? not everyone has the ability to upgrade at the drop of a hat.

@sixfeetover

This comment has been minimized.

Copy link

commented Jun 12, 2018

@HimanshuM This should probably stay open until a permanent solution is implemented.

@shanedroid

This comment has been minimized.

Copy link

commented Jun 13, 2018

Can confirm we were bit by this as well running on 5.1.3 installed through Rubygems with RVM.

Linux 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27 10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
ruby 2.3.4p301 (2017-03-30 revision 58214) [x86_64-linux]
Apache/2.4.7 (Ubuntu)
Passanger 5.1.3
Rails 4.2.10 

This could have been messy for us had we not caught it in time - thankfully disabling the security update as per @CamJN worked for now and we will be updating to 5.3

Agree with @sixfeetover that this issue should not be left closed unless the default setting is changed or docs can be updated to reflect that in order to leave security updated enabled you must be running ~> 5.1.6 to hopefully help others from getting hosed by this.

@bongole

This comment has been minimized.

Copy link

commented Jun 13, 2018

Hi, same error happened on passenger-5.1.1.

May be this commit fixed the error.
986776d

@FooBarWidget

This comment has been minimized.

Copy link
Member

commented Jun 13, 2018

Keeping this issue open for now for visibility.

@rakuhei

This comment has been minimized.

Copy link

commented Jun 13, 2018

Hi , I see the same bug.
I want to know is the occurrence reason.

Apache/2.4.6
passenger 5.1.2
CentOS Linux release 7.3.1611
ruby 2.3.3p222

I found the securitycheck URL on the SecurityUpdateChecker.h

$cat SecurityUpdateChecker.h
#define CHECK_HOST_DEFAULT "securitycheck.phusionpassenger.com"
#define CHECK_URL_DEFAULT "https://" CHECK_HOST_DEFAULT ":443/v1/check.json"

And,check the URL.
Command:
curl -k https://securitycheck.phusionpassenger.com:443/v1/check.json

Response:
This is not the way to access the security update check. For more information, see: https://www.phusionpassenger.com/library/indepth/security_update_check.html.

It's not JSON response.
I think this bug has been caused by changing the response.
It's so right ?

@Insood

This comment has been minimized.

Copy link

commented Jun 13, 2018

+1 - another 2 broken servers on Passenger 5.1.2

Will be updating to 5.3.2 tonight (as I should've... a long time ago)

@GUI

This comment has been minimized.

Copy link

commented Jun 13, 2018

One small note, but I think this was actually fixed in 5.1.6, not 5.1.5 as earlier mentioned:

passenger/CHANGELOG

Lines 1 to 4 in 320d397

Release 5.1.6
-------------------------------
* Fixes a typo that causes a looping crash when long security update information is sent by the server. In practice we will keep the messages shorter to avoid triggering this until there has been ample time to upgrade.

So I just thought that might help clear up confusion for people that were already running 5.1.5 and also hitting this.

@FooBarWidget

This comment has been minimized.

Copy link
Member

commented Jun 13, 2018

We've now updated the security check server to send a custom message for Passenger < 5.1.6 so that they don't crash. Of course, people should still upgrade in order to get the latest security fixes.

@aaronlifton

This comment has been minimized.

Copy link

commented Jun 13, 2018

@FooBarWidget Still crashing. Doesn't seem like that worked. Had to disable security check

kewubenduben added a commit to caregiverasia/opsworks-cookbooks that referenced this issue Jun 15, 2018

@tadams42

This comment has been minimized.

Copy link

commented Jun 15, 2018

I can confirm this on Ruby 2.2, Passenger 5.1.4. Upgrade to Passenger 5.2.3 solved it.

It was a pretty weird issue since our servers that hadn't been touched for months started segfaulting one after another.

Man I'm glad to have found this issue because we were unable to figure out root cause and that was quite demoralizing.

@CamJN

This comment has been minimized.

Copy link
Contributor

commented Jul 17, 2018

Closing since this should be done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.