New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[???] Segfault in phalcon.so #2104

Closed
temuri416 opened this Issue Feb 26, 2014 · 24 comments

Comments

Projects
None yet
4 participants
@temuri416
Contributor

temuri416 commented Feb 26, 2014

With the latest Phalcon v1.3.0 I started seeing this error in kern.log:

kernel: [608773.583588] php-fpm[20398]: segfault at 7f46a37b1a68 ip 00007f46a8292223 sp 00007fff6d579348 error 4 in phalcon.so[7f46a8256000+275000]

There are no errors/warnings dumped to php_errors.log or anywhere else.

The strange part is only one out of four HTTP requests fails.

I am quite certain that my code does not generate errors or warnings.

How can this be debugged?

Thanks!


Want to back this issue? Place a bounty on it! We accept bounties via Bountysource.

@temuri416

This comment has been minimized.

Show comment
Hide comment
@temuri416

temuri416 Feb 26, 2014

Contributor

I am seeing this:

root@projectQ:/tmp# gdb php /tmp/core-php-fpm.28363
GNU gdb (GDB) 7.5.91.20130417-cvs-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /var/web/php/bin/php...done.

warning: core file may not match specified executable file.
[New LWP 28363]

warning: Error reading shared library list entry at 0x31ff3100000002be

warning: Error reading shared library list entry at 0x5f72158d48008b48

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff8ff7000
Core was generated by `php-fpm: pool www                                                             '.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fd2a920d354 in ?? ()

(gdb) bt full
#0  0x00007fd2a920d354 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.
Contributor

temuri416 commented Feb 26, 2014

I am seeing this:

root@projectQ:/tmp# gdb php /tmp/core-php-fpm.28363
GNU gdb (GDB) 7.5.91.20130417-cvs-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /var/web/php/bin/php...done.

warning: core file may not match specified executable file.
[New LWP 28363]

warning: Error reading shared library list entry at 0x31ff3100000002be

warning: Error reading shared library list entry at 0x5f72158d48008b48

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff8ff7000
Core was generated by `php-fpm: pool www                                                             '.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fd2a920d354 in ?? ()

(gdb) bt full
#0  0x00007fd2a920d354 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Feb 26, 2014

Do you have the code on which this can be reproduced?

Or you may want to try to rebuild Phalcon in debug mode:

cd ext
phpize
./configure CFLAGS="-O0 -g3"
make -j16
sudo make install

ghost commented Feb 26, 2014

Do you have the code on which this can be reproduced?

Or you may want to try to rebuild Phalcon in debug mode:

cd ext
phpize
./configure CFLAGS="-O0 -g3"
make -j16
sudo make install
@temuri416

This comment has been minimized.

Show comment
Hide comment
@temuri416

temuri416 Feb 26, 2014

Contributor
  1. I did compile Phalcon in debug mode, as it was indicated in the article
  2. I have an HTTP location crashing inside my app, there's no way to reproduce it from CLI.
  3. Why can't I generate the proper dump? Am I missing a step?
  4. Never had this problem with one month-old revision of Phalcon...
Contributor

temuri416 commented Feb 26, 2014

  1. I did compile Phalcon in debug mode, as it was indicated in the article
  2. I have an HTTP location crashing inside my app, there's no way to reproduce it from CLI.
  3. Why can't I generate the proper dump? Am I missing a step?
  4. Never had this problem with one month-old revision of Phalcon...
@temuri416

This comment has been minimized.

Show comment
Hide comment
@temuri416

temuri416 Feb 27, 2014

Contributor

I am going to check out an earlier commits to find the troublemaker.

How do I now disable core dumps?

Contributor

temuri416 commented Feb 27, 2014

I am going to check out an earlier commits to find the troublemaker.

How do I now disable core dumps?

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Feb 27, 2014

ulimit -S -c 0

I have an HTTP location crashing inside my app, there's no way to reproduce it from CLI.

You can try this: create a file router.php and place it where your index.php is:

<?php
<?php

if (isset($_SERVER['PATH_INFO'])) {
        $_GET['_url'] = $_SERVER['PATH_INFO'];
        require 'index.php';
}
else {
        return false;
}
?>

Then start php-cli in the server mode:

php -S 127.0.0.1:10001

Navigate to http://127.0.0.1:10001/ and try to reproduce the crash.

If you have valgrind installed, then

USE_ZEND_ALLOC=0 valgrind /path/to/php -S 127.0.0.1:10001

will be more helpful.

Why can't I generate the proper dump? Am I missing a step?

The dump is OK, it just means that the error happened somewhere in the PHP core.

Never had this problem with one month-old revision of Phalcon

I wish I knew how to write programs without any bugs at all :-)

ghost commented Feb 27, 2014

ulimit -S -c 0

I have an HTTP location crashing inside my app, there's no way to reproduce it from CLI.

You can try this: create a file router.php and place it where your index.php is:

<?php
<?php

if (isset($_SERVER['PATH_INFO'])) {
        $_GET['_url'] = $_SERVER['PATH_INFO'];
        require 'index.php';
}
else {
        return false;
}
?>

Then start php-cli in the server mode:

php -S 127.0.0.1:10001

Navigate to http://127.0.0.1:10001/ and try to reproduce the crash.

If you have valgrind installed, then

USE_ZEND_ALLOC=0 valgrind /path/to/php -S 127.0.0.1:10001

will be more helpful.

Why can't I generate the proper dump? Am I missing a step?

The dump is OK, it just means that the error happened somewhere in the PHP core.

Never had this problem with one month-old revision of Phalcon

I wish I knew how to write programs without any bugs at all :-)

@temuri416

This comment has been minimized.

Show comment
Hide comment
@temuri416

temuri416 Feb 27, 2014

Contributor

:) thanks, I'll post more results as soon as I have them.

Contributor

temuri416 commented Feb 27, 2014

:) thanks, I'll post more results as soon as I have them.

@temuri416

This comment has been minimized.

Show comment
Hide comment
@temuri416

temuri416 Feb 27, 2014

Contributor

What I think is happening is that somewhere in my app a non-fatal error occurs. A warning or notice is issued. I cannot find it in the logs, but maybe it's because my logging level is different.

Do you think that may somehow crash Phalcon?

Contributor

temuri416 commented Feb 27, 2014

What I think is happening is that somewhere in my app a non-fatal error occurs. A warning or notice is issued. I cannot find it in the logs, but maybe it's because my logging level is different.

Do you think that may somehow crash Phalcon?

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Feb 27, 2014

You will need to run this as root:

ulimit -c unlimited
service php-fpm stop
service php-fpm start

(or whatever commands are used to start and stop php-fpm).

ghost commented Feb 27, 2014

You will need to run this as root:

ulimit -c unlimited
service php-fpm stop
service php-fpm start

(or whatever commands are used to start and stop php-fpm).

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Feb 27, 2014

Do you think that may somehow crash Phalcon?

Not sure. Do you have any idea as to what code causes crash?

ghost commented Feb 27, 2014

Do you think that may somehow crash Phalcon?

Not sure. Do you have any idea as to what code causes crash?

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost commented Feb 27, 2014

@temuri416

This comment has been minimized.

Show comment
Hide comment
@temuri416

temuri416 Feb 27, 2014

Contributor

I've been verifying my code and found that there was a PHP warning being generated during the execution of the HTTP request. It was not being dumped to error log, that's why I could not catch it earlier.

Having fixed it, Phalcon has become stable.

So, just FYI, in a situation like this, Phalcon becomes unstable and crashes randomly. You may keep that in mind, perhaps you find an explanation one day.

Thanks!

Contributor

temuri416 commented Feb 27, 2014

I've been verifying my code and found that there was a PHP warning being generated during the execution of the HTTP request. It was not being dumped to error log, that's why I could not catch it earlier.

Having fixed it, Phalcon has become stable.

So, just FYI, in a situation like this, Phalcon becomes unstable and crashes randomly. You may keep that in mind, perhaps you find an explanation one day.

Thanks!

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Feb 28, 2014

perhaps you find an explanation one day

Without the code to reproduce this on, this will be very difficult 😉

ghost commented Feb 28, 2014

perhaps you find an explanation one day

Without the code to reproduce this on, this will be very difficult 😉

@temuri416

This comment has been minimized.

Show comment
Hide comment
@temuri416

temuri416 Feb 28, 2014

Contributor

How about doing screensharing? I can guide you through the app and will show you any code that might be suspicious.

I have installed latest Ubuntu, recompiled everything, tried various config files - it is still crashing. Spent the whole day, no result.

Contributor

temuri416 commented Feb 28, 2014

How about doing screensharing? I can guide you through the app and will show you any code that might be suspicious.

I have installed latest Ubuntu, recompiled everything, tried various config files - it is still crashing. Spent the whole day, no result.

@ianbytchek

This comment has been minimized.

Show comment
Hide comment
@ianbytchek

ianbytchek Feb 28, 2014

Contributor

Might sound like a silly question, but did you try setting a breakpoint in index.php and following to the point where it crashes?

Contributor

ianbytchek commented Feb 28, 2014

Might sound like a silly question, but did you try setting a breakpoint in index.php and following to the point where it crashes?

@temuri416

This comment has been minimized.

Show comment
Hide comment
@temuri416

temuri416 Feb 28, 2014

Contributor

I am using PHPed and the script does not crash. It always finished successfully, I traced it up until the moment of session destructor - no warnings, no errors.

I started checking out earlier commits and I see that they are stable. So, I'm gonna find the one that broke it.

Contributor

temuri416 commented Feb 28, 2014

I am using PHPed and the script does not crash. It always finished successfully, I traced it up until the moment of session destructor - no warnings, no errors.

I started checking out earlier commits and I see that they are stable. So, I'm gonna find the one that broke it.

@ianbytchek

This comment has been minimized.

Show comment
Hide comment
@ianbytchek

ianbytchek Feb 28, 2014

Contributor

I've been having a lot of pain lately with 1.3.0, too. Try recompiling it from the source, I've put some suggestions. That might help if you're after the latest commit.

Contributor

ianbytchek commented Feb 28, 2014

I've been having a lot of pain lately with 1.3.0, too. Try recompiling it from the source, I've put some suggestions. That might help if you're after the latest commit.

@temuri416

This comment has been minimized.

Show comment
Hide comment
@temuri416

temuri416 Mar 1, 2014

Contributor

@sjinks

OK, I FOUND IT.

The last stable commit is Round 1, 7db1056 , from 02/12/2014 03:51.

The next commit, Round 2, fa43e8f, from 02/12/2014 07:01 is crashing it.

I was compiling from ext using the following command:

phpize .clean && phpize && ./configure CFLAGS="-O0 -g3" CC=gcc && make -j75 && sudo make install

Again, there are no warnings or errors generated by my script.

I can send you over my source files as a last resort, but hopefully, we'll find other way to catch error.

Thank you!

Contributor

temuri416 commented Mar 1, 2014

@sjinks

OK, I FOUND IT.

The last stable commit is Round 1, 7db1056 , from 02/12/2014 03:51.

The next commit, Round 2, fa43e8f, from 02/12/2014 07:01 is crashing it.

I was compiling from ext using the following command:

phpize .clean && phpize && ./configure CFLAGS="-O0 -g3" CC=gcc && make -j75 && sudo make install

Again, there are no warnings or errors generated by my script.

I can send you over my source files as a last resort, but hopefully, we'll find other way to catch error.

Thank you!

@ianbytchek

This comment has been minimized.

Show comment
Hide comment
@ianbytchek

ianbytchek Mar 1, 2014

Contributor

Did you figure out how to get a dump by the way? Can you post it?

Contributor

ianbytchek commented Mar 1, 2014

Did you figure out how to get a dump by the way? Can you post it?

@ianbytchek

This comment has been minimized.

Show comment
Hide comment
@ianbytchek
Contributor

ianbytchek commented Mar 1, 2014

@temuri416

This comment has been minimized.

Show comment
Hide comment
@temuri416

temuri416 Mar 1, 2014

Contributor

I got the dump, but sjinks said this:

"The dump is OK, it just means that the error happened somewhere in the PHP core."

Contributor

temuri416 commented Mar 1, 2014

I got the dump, but sjinks said this:

"The dump is OK, it just means that the error happened somewhere in the PHP core."

@temuri416

This comment has been minimized.

Show comment
Hide comment
@temuri416

temuri416 Mar 1, 2014

Contributor

@ianbytchek

oh, sorry misread your question. That's the link where I got the working solution from.

Contributor

temuri416 commented Mar 1, 2014

@ianbytchek

oh, sorry misread your question. That's the link where I got the working solution from.

@ianbytchek

This comment has been minimized.

Show comment
Hide comment
@ianbytchek

ianbytchek Mar 1, 2014

Contributor

😐 missed that. Sorry. Can you post piece of code where it crashes? You said you narrowed it down to session being destroyed?

Contributor

ianbytchek commented Mar 1, 2014

😐 missed that. Sorry. Can you post piece of code where it crashes? You said you narrowed it down to session being destroyed?

@ianbytchek

This comment has been minimized.

Show comment
Hide comment
@ianbytchek

ianbytchek Mar 1, 2014

Contributor

I'm somehow confident it's has something to do with Phalcon, especially if other commits are working.

Contributor

ianbytchek commented Mar 1, 2014

I'm somehow confident it's has something to do with Phalcon, especially if other commits are working.

@temuri416

This comment has been minimized.

Show comment
Hide comment
@temuri416

temuri416 Mar 1, 2014

Contributor

You're confusing two issues that I'm currently having.

  1. After the mentioned commit Phalcon starts crashing on 1 out of 5 HTTP requests. I cannot post the code, because I cannot pinpoint it to any line and the execution stack is very long (my app also requires presence of a valid Google OAuth2 user, which does not make it easier to debug). The script does not crash. My code is perfectly valid, as it works and had worked for about 7 months with all previous commits on branch 1.3.0.
  2. There's another issue that's resurfaced - #1624
Contributor

temuri416 commented Mar 1, 2014

You're confusing two issues that I'm currently having.

  1. After the mentioned commit Phalcon starts crashing on 1 out of 5 HTTP requests. I cannot post the code, because I cannot pinpoint it to any line and the execution stack is very long (my app also requires presence of a valid Google OAuth2 user, which does not make it easier to debug). The script does not crash. My code is perfectly valid, as it works and had worked for about 7 months with all previous commits on branch 1.3.0.
  2. There's another issue that's resurfaced - #1624
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Mar 1, 2014

Collaborator

Could you please try again using the current 1.3.0?

Collaborator

ghost commented Mar 1, 2014

Could you please try again using the current 1.3.0?

@ianbytchek

This comment has been minimized.

Show comment
Hide comment
@ianbytchek

ianbytchek Mar 1, 2014

Contributor

Got you. I had similar "floater" recently #2085. There still must be someway to pinpoint it either through dump or through php -S 127.0.0.1:10001 suggester above.

Contributor

ianbytchek commented Mar 1, 2014

Got you. I had similar "floater" recently #2085. There still must be someway to pinpoint it either through dump or through php -S 127.0.0.1:10001 suggester above.

@temuri416

This comment has been minimized.

Show comment
Hide comment
@temuri416

temuri416 Mar 1, 2014

Contributor

@phalcon

It is stable now!

BTW - what do I have to do to convince you to get rid of that sage of Phalcon\DI to store non-objects is deprecated, please use Phalcon\Registry instead error? I am pulling my hair out..

Thanks.

Contributor

temuri416 commented Mar 1, 2014

@phalcon

It is stable now!

BTW - what do I have to do to convince you to get rid of that sage of Phalcon\DI to store non-objects is deprecated, please use Phalcon\Registry instead error? I am pulling my hair out..

Thanks.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Mar 1, 2014

Collaborator

We could add a php.ini option to keep backwards compatibility with the old behavior, but you must know you're giving an incorrect use to Phalcon\DI that's why you're getting that warning.

Collaborator

ghost commented Mar 1, 2014

We could add a php.ini option to keep backwards compatibility with the old behavior, but you must know you're giving an incorrect use to Phalcon\DI that's why you're getting that warning.

@temuri416

This comment has been minimized.

Show comment
Hide comment
@temuri416

temuri416 Mar 1, 2014

Contributor

I do not think I am using the DI incorrectly. Let me explain using the code from the application I'm working on.

This is the WAMP Topic user-specific prefix generator that lives in the shared DI:

'WampPrefix' => function($app) {
    if ($app->getDI()->has('auth')) {
        $auth = $app->getDI()->get('auth');
        if ($auth->isAuthenticated()) {
            return $app->getDI()->get('params')['WampBasePrefix'] . $auth->getAccount()->getId();
        }
    }
    throw new \Exception('WAMP Prefix cannot be generated without authenticated User account.');
},

It returns a string and should only be executed once within the lifetime of a request. That string does not change, as it is specific to the currently signed in user. However, to generate that string, I need access to current $app object and its DI and some other components from DI.

I do not see anything wrong with the above approach. I don't want to use Phalcon\Registry as it will compose that prefix every time $registry->WampPrefix() is called. In addition, Phalcon\Registry does not extend Phalcon\DI\Injectable, and actually, that is where it all ends for me.

Is there a rational explanation of the dangers/problems that my approach has to returning strings from the DI?

Thanks.

Contributor

temuri416 commented Mar 1, 2014

I do not think I am using the DI incorrectly. Let me explain using the code from the application I'm working on.

This is the WAMP Topic user-specific prefix generator that lives in the shared DI:

'WampPrefix' => function($app) {
    if ($app->getDI()->has('auth')) {
        $auth = $app->getDI()->get('auth');
        if ($auth->isAuthenticated()) {
            return $app->getDI()->get('params')['WampBasePrefix'] . $auth->getAccount()->getId();
        }
    }
    throw new \Exception('WAMP Prefix cannot be generated without authenticated User account.');
},

It returns a string and should only be executed once within the lifetime of a request. That string does not change, as it is specific to the currently signed in user. However, to generate that string, I need access to current $app object and its DI and some other components from DI.

I do not see anything wrong with the above approach. I don't want to use Phalcon\Registry as it will compose that prefix every time $registry->WampPrefix() is called. In addition, Phalcon\Registry does not extend Phalcon\DI\Injectable, and actually, that is where it all ends for me.

Is there a rational explanation of the dangers/problems that my approach has to returning strings from the DI?

Thanks.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Mar 1, 2014

It is stable now!

It is not. fcall cache corruption is the result of another bug where the same variable is destructed more than once or when a function incorrectly uses parent function's memory frame.

Turning off the cache just masks the issue but does not fix it.

ghost commented Mar 1, 2014

It is stable now!

It is not. fcall cache corruption is the result of another bug where the same variable is destructed more than once or when a function incorrectly uses parent function's memory frame.

Turning off the cache just masks the issue but does not fix it.

@lantian lantian referenced this issue Mar 1, 2014

Closed

Phalcon 1.3.0 #2066

@temuri416

This comment has been minimized.

Show comment
Hide comment
@temuri416

temuri416 Mar 10, 2014

Contributor

Is this still an open issue?

Contributor

temuri416 commented Mar 10, 2014

Is this still an open issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment