Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Seg. faults at login process (TYPO3) #2377

Closed
khaefeli opened this issue Apr 9, 2014 · 21 comments
Closed

Seg. faults at login process (TYPO3) #2377

khaefeli opened this issue Apr 9, 2014 · 21 comments

Comments

@khaefeli
Copy link

khaefeli commented Apr 9, 2014

Hi,

I've the following setup:

  • Debian wheezy
  • Apache 2.2 / nginx 1.2
  • fastcgi
  • hhvm 3.0.1
  • TYPO3 6.2 LTS:

loginSecurityLevel' => 'rsa'
loginSecurityLevel' => 'normal' (both not working)

I try to login into the TYPO3 6.2 backend and hhvm crashes.
Credentials are correct - frontend is working well.

HHVM stacktrace

#0  bt_handler at /tmp/tmp.5Swo76XRcG/hphp/runtime/base/crash-reporter.cpp:63
#1  killpg at /'lib/x86_64-linux-gnu/libc.so.6:0
#2  memcpy at /tmp/tmp.5Swo76XRcG/hphp/runtime/vm/jit/memcpy-x64.cpp:41
#3  get_tty_password at /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18:0
#4  mysql_stmt_fetch at /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18:0
#5  HPHP::MySQLStmt::fetch() at /tmp/tmp.5Swo76XRcG/hphp/runtime/ext/mysql/mysql_common.cpp:1077
#6  c_mysqli_stmt_ni_fetch at /tmp/tmp.5Swo76XRcG/hphp/runtime/ext/mysql/ext_mysqli.cpp:526
#7  long HPHP::Native::NativeFuncCaller<long, 0ul, 0ul>::call<HPHP::TypedValue*, HPHP::Value*>(HPHP::Func const*, HPHP::TypedValue*, HPHP::TypedValue*, HPHP::Value*) at /tmp/tmp.5Swo76XRcG/hphp/runtime/vm/native.cpp:97
#8  makeNativeRefCall<HPHP::TypedValue> at /tmp/tmp.5Swo76XRcG/hphp/runtime/vm/native.cpp:229
#9  HPHP::Native::callFunc(HPHP::Func const*, HPHP::TypedValue*, HPHP::TypedValue*, int, HPHP::TypedValue&) at /tmp/tmp.5Swo76XRcG/hphp/runtime/vm/native.cpp:318
#10 HPHP::Native::methodWrapper(HPHP::ActRec*) at /tmp/tmp.5Swo76XRcG/hphp/runtime/vm/native.cpp:432
#11 HPHP::ExecutionContext::iopNativeImpl(unsigned char const*&) at /tmp/tmp.5Swo76XRcG/hphp/runtime/vm/bytecode.cpp:6951
#12 void HPHP::ExecutionContext::dispatchImpl<4>(int) at /tmp/tmp.5Swo76XRcG/hphp/runtime/vm/bytecode.cpp:7626
#13 HPHP::ExecutionContext::dispatch() at /tmp/tmp.5Swo76XRcG/hphp/runtime/vm/bytecode.cpp:7633
#14 HPHP::ExecutionContext::enterVMAtFunc(HPHP::ActRec*) at /tmp/tmp.5Swo76XRcG/hphp/runtime/vm/bytecode.cpp:1554
#15 HPHP::ExecutionContext::enterVM(HPHP::ActRec*) at /tmp/tmp.5Swo76XRcG/hphp/runtime/vm/bytecode.cpp:1600
#16 HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue*, HPHP::Func const*, HPHP::Variant const&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags) at /tmp/tmp.5Swo76XRcG/hphp/runtime/vm/bytecode.cpp:1807
#17 HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue*, HPHP::Unit*) at /tmp/tmp.5Swo76XRcG/hphp/runtime/vm/bytecode.cpp:1960
#18 HPHP::invoke_file_impl(HPHP::Variant&, HPHP::String const&, bool, char const*) at /tmp/tmp.5Swo76XRcG/hphp/runtime/base/builtin-functions.cpp:855
#19 HPHP::invoke_file(HPHP::String const&, bool, char const*) at /tmp/tmp.5Swo76XRcG/hphp/runtime/base/builtin-functions.cpp:836
#20 HPHP::include_impl_invoke(HPHP::String const&, bool, char const*) at /tmp/tmp.5Swo76XRcG/hphp/runtime/base/builtin-functions.cpp:819
#21 HPHP::hphp_invoke(HPHP::ExecutionContext*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, HPHP::Array const&, HPHP::VRefParamValue const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool, bool, bool) at /tmp/tmp.5Swo76XRcG/hphp/runtime/base/program-functions.cpp:1691
#22 HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /tmp/tmp.5Swo76XRcG/hphp/runtime/server/http-request-handler.cpp:360
#23 HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /tmp/tmp.5Swo76XRcG/hphp/runtime/server/http-request-handler.cpp:289
#24 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /tmp/tmp.5Swo76XRcG/hphp/runtime/server/server-worker.h:104
#25 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /tmp/tmp.5Swo76XRcG/hphp/runtime/server/server-worker.h:57
#26 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /tmp/tmp.5Swo76XRcG/hphp/util/job-queue.h:409
#27 HPHP::AsyncFunc<HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits> >::run_(void*) at /tmp/tmp.5Swo76XRcG/hphp/util/async-func.h:206
#28 HPHP::AsyncFuncImpl::threadFuncImpl() at /tmp/tmp.5Swo76XRcG/hphp/util/async-func.cpp:140
#29 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /tmp/tmp.5Swo76XRcG/hphp/util/async-func.cpp:52
#30 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
#31 __clone at /lib/x86_64-linux-gnu/libc.so.6:0

PHP Stacktrace:

#0  mysqli_stmt->fetch() called at [/var/www/demo/www/TYPO3.CMS/typo3/sysext/core/Classes/Database/PreparedStatement.php:425]
#1  TYPO3\CMS\Core\Database\PreparedStatement->fetch() called at [/var/www/demo/www/TYPO3.CMS/typo3/sysext/core/Classes/Authentication/AbstractUserAuthentication.php:907]
#2  TYPO3\CMS\Core\Authentication\AbstractUserAuthentication->fetchUserSession() called at [/var/www/demo/www/TYPO3.CMS/typo3/sysext/core/Classes/Authentication/AbstractUserAuthentication.php:648]
#3  TYPO3\CMS\Core\Authentication\AbstractUserAuthentication->checkAuthentication() called at [/var/www/demo/www/TYPO3.CMS/typo3/sysext/core/Classes/Authentication/AbstractUserAuthentication.php:431]
#4  TYPO3\CMS\Core\Authentication\AbstractUserAuthentication->start() called at [/var/www/demo/www/TYPO3.CMS/typo3/sysext/core/Classes/Core/Bootstrap.php:1003]
#5  TYPO3\CMS\Core\Core\Bootstrap->initializeBackendUser() called at [/var/www/demo/www/TYPO3.CMS/typo3/init.php:71]
#6  include(/var/www/demo/www/TYPO3.CMS/typo3/init.php) called at [/var/www/demo/www/TYPO3.CMS/typo3/backend.php:27]

reproduction case in comment #2377 (comment)

@alexmalyshev
Copy link
Contributor

Thanks for opening this issue, and for attaching the stack traces. Feel free to dig deeper into this (you, or anyone else) and send us a PR; we'd appreciate it.

@khaefeli
Copy link
Author

Hi @alexmalyshev, you're welcome.
I'm sysadmin and not a developer - so I keep away from C++ code ;-)
FYI: the issue still exists in:
HipHop VM 3.2.0-dev+2014.05.27 (dbg)
TYPO3 6.2.4-dev

@Aatch
Copy link
Contributor

Aatch commented Jun 12, 2014

@khaefeli do you mind sharing the version of libmysqlclient you're using?

@khaefeli
Copy link
Author

@Aatch sure:

libmysqlclient18:amd64 - 5.5.37-0+wheezy1

hope it helps.

@r3wt
Copy link

r3wt commented Jun 12, 2014

is there any hope for us?

@khaefeli
Copy link
Author

@Aatch & @r3wt regarding the refernce #2921 - I dont think it's related to this bug.
I expect that TYPO3 is doing that the right way with valid php code and it's also working if I fallback it to php-fpm over nginx:

error_page 500 501 502 503 504  @fpm;

anyway: but for sure, we should get a usable error message instead of a seg. fault / crash

@ptarjan is there any chance for a higher priority here?
TYPO3 is running on over 500'000 websites (enterprise) websites worldwide ;)

@SiebelsTim
Copy link
Contributor

@khaefeli This will be fixed in no time if you got a small reproduction case. :P

@khaefeli
Copy link
Author

@SiebelsTim g
sorry, I thought the description is clear ;-)

  1. Install hhvm 3.1 with nginx or apache and a mysql server (5.5 for example)
  2. download and install TYPO3 6.2 LTS http://typo3.org/download/ or git clone and install https://github.com/TYPO3/TYPO3.CMS
    2.1) TYPO3 install howto: https://github.com/TYPO3/TYPO3.CMS/blob/master/INSTALL.md#installation
  3. login into the backend over http://yourhost/typo3/ ends in a segfault

@SiebelsTim
Copy link
Contributor

@khaefeli
hehe, yes of course. I meant a piece of code that can reproduce it. Typo3 is a complex CMS. That makes it harder to find the line of code what breaks it.

@khaefeli
Copy link
Author

@SiebelsTim I tried to break it down to a small code snippet. But as I wrote in #2377 (comment) I'm sysadmin and no php dev ;-)
https://github.com/TYPO3/TYPO3.CMS/blob/master/typo3/sysext/core/Classes/Database/PreparedStatement.php#L428

        /**
     * Fetches a row from a result set associated with a \TYPO3\CMS\Core\Database\PreparedStatement object.
     *
     * @param integer $fetch_style Controls how the next row will be returned to the caller. This value must be one of the \TYPO3\CMS\Core\Database\PreparedStatement::FETCH_* constants. If omitted, default fetch mode for this prepared query will be used.
     * @return array Array of rows or FALSE if there are no more rows.
     * @api
     */
public function fetch($fetch_style = 0) {
    if ($fetch_style == 0) {
        $fetch_style = $this->defaultFetchMode;
    }

    if ($this->statement instanceof \mysqli_stmt) {
        if ($this->buffer === NULL) {
            $variables = array();
            $this->buffer = array();
            foreach ($this->fields as $field) {
                $this->buffer[$field] = NULL;
                $variables[] = &$this->buffer[$field];
            }

            call_user_func_array(array($this->statement, 'bind_result'), $variables);
        }
        $success = $this->statement->fetch();
        $columns = $this->buffer;
    } else {
        $columns = $this->statement->fetch();
        $success = is_array($columns);
    }

    if ($success) {
        $row = array();
        foreach ($columns as $key => $value) {
            switch ($fetch_style) {
                case self::FETCH_ASSOC:
                    $row[$key] = $value;
                    break;
                case self::FETCH_NUM:
                    $row[] = $value;
                    break;
                default:
                    throw new \InvalidArgumentException('$fetch_style must be either TYPO3\\CMS\\Core\\Database\\PreparedStatement::FETCH_ASSOC or TYPO3\\CMS\\Core\\Database\\PreparedStatement::FETCH_NUM', 1281646455);
            }
        }
    } else {
        $row = FALSE;
    }

    return $row;
}

var_dump of $this->statement with php5-fpm when it's called in the TYPO3 login process:

object(mysqli_stmt)#464 (10) {
  ["affected_rows"]=>
  int(1)
  ["insert_id"]=>
  int(0)
  ["num_rows"]=>
  int(1)
  ["param_count"]=>
  int(1)
  ["field_count"]=>
  int(1)
  ["errno"]=>
  int(0)
  ["error"]=>
  string(0) ""
  ["error_list"]=>
  array(0) {
  }
  ["sqlstate"]=>
  string(5) "00000"
  ["id"]=>
  int(1)
}

HHVM:

object(mysqli_stmt)#493 (2) {
  ["__stmt":"mysqli_stmt":private]=>
  resource(5) of type (mysql stmt)  
  ["__link":"mysqli_stmt":private]=>
  object(mysqli)#443 (1) {
    ["__connection":"mysqli":private]=>
    resource(4) of type (mysql link)
  }
}

@cgerold
Copy link

cgerold commented Jun 21, 2014

It seems that the problem is caused by mysql datatype longtext.
Within the be_session table the field ses_data is of the type longtext.
When you change the type to text the backend login works.

CREATE TABLE `be_sessions` (
  `ses_id` varchar(32) NOT NULL DEFAULT '',
  `ses_data` longtext,
  PRIMARY KEY (`ses_id`),
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Here is a test script provided. When you change the data type of the above given attribute ses_data to TEXT it works fine.

<?php
$mysqli = new mysqli("localhost", "username", "password" "database");

if (mysqli_connect_errno()) {
    printf("Connect failed: %s\n", mysqli_connect_error());
    exit();
}

$query = "SELECT ses_id, ses_data FROM be_sessions";

if ($stmt = $mysqli->prepare($query)) {
    $stmt->execute();

    $stmt->bind_result($ses_id, $ses_data);

    while ($stmt->fetch()) {
        printf ("%s (%s)\n", $ses_id, $ses_data);
    }

    $stmt->close();
}

$mysqli->close();
?>

Other data types it works not with: longblob, longtext
Other data types it works with: text, mediumtext, blob, mediumblob

Contribution to @beachmachine, @rewi, @sascha-egerer

@jaguerra
Copy link

Problem seems to be on libmysqlclient, the following test fails either on hhvm and on vanilla php:

<?php

/*
 * Create a table with a LONGTEXT and pull in some data
 */

$mysqli = new mysqli("127.0.0.1", "root", "root", "test_hhvm");
if ($mysqli->connect_errno) {
    echo "Failed to connect to MySQL: (" . $mysqli->connect_errno . ") " . $mysqli->connect_error;
}

if (!$mysqli->query("DROP TABLE IF EXISTS test") || !$mysqli->query("CREATE TABLE test(label LONGTEXT)")) {
    echo "Table creation failed: (" . $mysqli->errno . ") " . $mysqli->error;
}

$mysqli->query('INSERT INTO test(label) values ("test")');

/*
 * Hit the bug, expect a segfault on hhvm
 */

$stmt = $mysqli->prepare("SELECT label FROM test");
$stmt->execute();
$stmt->bind_result($label);
while($stmt->fetch()) {
        echo "$label" . PHP_EOL;
}
$stmt->close();
$mysqli->close();

?>

Difference is hhvm dies on a segmentation fault and vanilla php tries to allocate all the system memory so a fatal error is thrown.

HHVM output

$ hhvm testLongTextBug.php  
Segmentation fault (core dumped)

PHP output

$ php testLongTextBug.php 
PHP Fatal error:  Out of memory (allocated 262144) (tried to allocate 4294967296 bytes) in /home/vagrant/testLongTextBug.php on line 24

Tested on Ubuntu 14.04 64bits

@SiebelsTim
Copy link
Contributor

@jaguerra What libmysqlclient and hhvm version do you have?
That test fails in php for me but works in hhvm

I use Ubuntu 14.04, mysql 5.5.37 and hhvm 3.1.0

@jaguerra
Copy link

@SiebelsTim Using hhvm-nightly

vagrant@vagrant-ubuntu-trusty-64:~$ hhvm --version
HipHop VM 3.2.0-dev+2014.06.20 (rel)
Compiler: heads/master-0-g3b5e28c1f1f9562c8c7313b5ad1fc4d46c5d190f
Repo schema: f0695d484a1bd52cac118659ee68c6e7d0027122


vagrant@vagrant-ubuntu-trusty-64:~$ mysql --version
mysql  Ver 14.14 Distrib 5.5.37, for debian-linux-gnu (x86_64) using readline 6.3


vagrant@vagrant-ubuntu-trusty-64:~$ dpkg -l *libmysqlclient*
||/ Name                              Version               Architecture          Description
+++-=================================-=====================-=====================-========================================================================
ii  libmysqlclient18:amd64            5.5.37-0ubuntu0.14.04 amd64                 MySQL database client library

@jaguerra
Copy link

@SiebelsTim Also breaks on HHVM 3.1.0

vagrant@vagrant-ubuntu-trusty-64:~$ hhvm --version
HipHop VM 3.1.0 (rel)
Compiler: tags/HHVM-3.1.0-0-g71ecbd8fb5e94b2a008387a2b5e9a8df5c6f5c7b
Repo schema: 88ae0db264d72ec2e2eb22ab25d717214aee568b

@r3wt
Copy link

r3wt commented Jun 21, 2014

yep. my column it was failing on was a longtext

@SiebelsTim
Copy link
Contributor

@jaguerra I also tried with 3.2.0. It's not failing for me.

However I can reproduce this in a virtual machine.

@SiebelsTim
Copy link
Contributor

@jaguerra I get a segfault on my host machine as well when I am low on memory. Are we running out of memory?

How much memory do you have? I have 4GB.
How much memory is free when you run it?

@SiebelsTim
Copy link
Contributor

It's probably this call that fails, because b->buffer_length is 4294967295 (2^32) for me returning null.
https://github.com/facebook/hhvm/blob/master/hphp/runtime/ext/mysql/mysql_common.cpp#L842
That value later is used in mysql to memcpy the row's value.
buffer_length is probably is 2^32 because that's the max_length for a LONGTEXT.
LONGTEXT should be a variable-length type. But that seems to be not working.

(gdb) p *fields
$5 = {name = 0x7fffe856beb0 "label", org_name = 0x7fffe856beb8 "label", 
  table = 0x7fffe856bea0 "test", org_table = 0x7fffe856bea8 "test", 
  db = 0x7fffe856be98 "test", catalog = 0x7fffe856be90 "def", def = 0x0, 
  length = 4294967295, max_length = 0, name_length = 5, org_name_length = 5, 
  table_length = 4, org_table_length = 4, db_length = 4, catalog_length = 3, 
  def_length = 0, flags = 16, decimals = 0, charsetnr = 8, 
  type = MYSQL_TYPE_BLOB, extension = 0x0}

This might be a mysql bug (regarding php fails as well)

Reproduce like this:

  1. Fire up gdb
  2. break on HPHP::MySQLStmtVariables::bind_result
  3. step through fields or the calloc call

SiebelsTim added a commit to SiebelsTim/hhvm that referenced this issue Jun 26, 2014
PHP calls STMT_ATTR_UPDATE_MAX_LENGTH in bind_result

This results in LONGTEXTs being variable length rather than 4GB

Fixes facebook#2377
Probably Fixes facebook#2921
@fredemmott
Copy link
Contributor

There's a fix being reviewed/tested internally; FB: D1407344

@khaefeli
Copy link
Author

I can confirm the bugfix. It working with HipHop VM 3.2.0-dev+2014.07.17 (rel)
thx to all!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants