New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

if addServers too much server, and OPT_LIBKETAMA_COMPATIBLE is true, core dump will appear #407

Open
khadgarmage opened this Issue Aug 9, 2018 · 4 comments

Comments

Projects
None yet
3 participants
@khadgarmage
Copy link

khadgarmage commented Aug 9, 2018

hi, @sodabrew if addServers too much server, and OPT_LIBKETAMA_COMPATIBLE is true, core dump will appear
PHP env:
PHP 5.3.29 or PHP 5.5.15
Reproduce:

<?php
$mem=new Memcached();
$servers = array (
        array ( 'host' => '10.13.81.227', 'port' => 11211 ),
        array ( 'host' => '10.13.81.227', 'port' => 11212 ),
        array ( 'host' => '10.13.81.227', 'port' => 11213 ),
        array ( 'host' => '10.13.81.155', 'port' => 11211 ),
        array ( 'host' => '10.13.81.155', 'port' => 11212 ),
        array ( 'host' => '10.13.81.155', 'port' => 11213 ),
        array ( 'host' => '10.13.81.231', 'port' => 11211 ),
        array ( 'host' => '10.13.81.231', 'port' => 11212 ),
        array ( 'host' => '10.13.81.231', 'port' => 11213 ),
        array ( 'host' => '10.13.81.125', 'port' => 11211 ),
        array ( 'host' => '10.13.81.125', 'port' => 11212 ),
        array ( 'host' => '10.13.81.125', 'port' => 11213 ),
        array ( 'host' => '10.13.81.13', 'port' => 11211 ),
        array ( 'host' => '10.13.81.13', 'port' => 11212 ),
        array ( 'host' => '10.13.81.13', 'port' => 11213 ),
        array ( 'host' => '10.13.81.92', 'port' => 11211 ),
        array ( 'host' => '10.13.81.92', 'port' => 11212 ),
        array ( 'host' => '10.13.81.92', 'port' => 11213 ),
        array ( 'host' => '10.13.81.209', 'port' => 11211 ),
        array ( 'host' => '10.13.81.209', 'port' => 11212 ),
        array ( 'host' => '10.13.81.209', 'port' => 11213 ),
        array ( 'host' => '10.13.81.177', 'port' => 11211 ),
        array ( 'host' => '10.13.81.177', 'port' => 11212 ),
        array ( 'host' => '10.13.81.177', 'port' => 11213 ),
        array ( 'host' => '10.13.81.70', 'port' => 11211 ),
        array ( 'host' => '10.13.81.70', 'port' => 11212 ),
        array ( 'host' => '10.13.81.70', 'port' => 11213 ),
);
$mem->addServers($servers);
$mem->setOption(Memcached::OPT_DISTRIBUTION,Memcached::DISTRIBUTION_CONSISTENT);
$mem->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true);
$mem->set('name',"testdfdsafdsafdsafda",15);
echo $mem->get('name');

Result On cli:

[ffs-amazon-web root@ip-10-13-80-98 ~]# php t.php
*** glibc detected *** php: double free or corruption (out): 0x000000000132c210 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3fca275e76]
/lib64/libc.so.6[0x3fca2789b3]
/lib64/libc.so.6(qsort_r+0x2a8)[0x3fca234f38]
/usr/lib/libmemcached.so.11(+0x15812)[0x7fd6743c4812]
/usr/lib64/php/modules/memcached.so(+0x79ef)[0x7fd6745e79ef]
/usr/lib64/php/modules/memcached.so(+0x7bc7)[0x7fd6745e7bc7]
php[0x63eef3]
php(execute_ex+0x38)[0x6307a8]
php(zend_execute_scripts+0x18c)[0x5bbbdc]
php(php_execute_script+0x1f0)[0x55c5c0]
php[0x66d233]
php[0x66da38]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3fca21ed5d]
php[0x422939]
======= Memory map: ========
00400000-00754000 r-xp 00000000 ca:01 148069                             /usr/bin/php
00954000-009e8000 rw-p 00354000 ca:01 148069                             /usr/bin/php
009e8000-00a03000 rw-p 00000000 00:00 0
00de7000-00df2000 rw-p 003e7000 ca:01 148069                             /usr/bin/php
01033000-01353000 rw-p 00000000 00:00 0                                  [heap]
391fa00000-391fa1d000 r-xp 00000000 ca:01 237783                         /lib64/libselinux.so.1
391fa1d000-391fc1c000 ---p 0001d000 ca:01 237783                         /lib64/libselinux.so.1
391fc1c000-391fc1d000 r--p 0001c000 ca:01 237783                         /lib64/libselinux.so.1
391fc1d000-391fc1e000 rw-p 0001d000 ca:01 237783                         /lib64/libselinux.so.1
391fc1e000-391fc1f000 rw-p 00000000 00:00 0
391fe00000-391fe22000 r-xp 00000000 ca:01 239044                         /lib64/libncurses.so.5.7
391fe22000-3920021000 ---p 00022000 ca:01 239044                         /lib64/libncurses.so.5.7
3920021000-3920022000 rw-p 00021000 ca:01 239044                         /lib64/libncurses.so.5.7
@khadgarmage

This comment has been minimized.

Copy link
Author

khadgarmage commented Aug 9, 2018

if the code like as followed, the bug will not appear:

<?php
$mem=new Memcached();
$servers = array (
        array ( 'host' => '10.13.81.227', 'port' => 11211 ),
        array ( 'host' => '10.13.81.227', 'port' => 11212 ),
        array ( 'host' => '10.13.81.227', 'port' => 11213 ),
);
$mem->addServers($servers);
$mem->setOption(Memcached::OPT_DISTRIBUTION,Memcached::DISTRIBUTION_CONSISTENT);
$mem->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true);
$mem->set('name',"testdfdsafdsafdsafda",15);
echo $mem->get('name');

@sodabrew sodabrew added this to the 2.3.0 milestone Dec 22, 2018

@sodabrew

This comment has been minimized.

Copy link
Contributor

sodabrew commented Dec 22, 2018

I am not able to reproduce this with memcached 3.1.x with PHP 7.x.

@crrodriguez

This comment has been minimized.

Copy link

crrodriguez commented Jan 11, 2019

I can reproduce this with this slightly shorter code

<?php
$mem=new Memcached();
$servers = array (
        array ( 'host' => '10.13.81.227', 'port' => 11211 ), 
        array ( 'host' => '10.13.81.227', 'port' => 11212 ), 
        array ( 'host' => '10.13.81.227', 'port' => 11213 ), 
        array ( 'host' => '10.13.81.155', 'port' => 11211 ), 
        array ( 'host' => '10.13.81.155', 'port' => 11212 ), 
        array ( 'host' => '10.13.81.155', 'port' => 11213 ), 
        array ( 'host' => '10.13.81.231', 'port' => 11211 ), 
        array ( 'host' => '10.13.81.231', 'port' => 11212 ), 
        array ( 'host' => '10.13.81.231', 'port' => 11213 ), 
        array ( 'host' => '10.13.81.125', 'port' => 11211 ), 
        array ( 'host' => '10.13.81.125', 'port' => 11212 ), 
        array ( 'host' => '10.13.81.125', 'port' => 11213 ), 
        array ( 'host' => '10.13.81.13', 'port' => 11211 ), 
        array ( 'host' => '10.13.81.13', 'port' => 11212 ), 
        array ( 'host' => '10.13.81.13', 'port' => 11213 ), 
        array ( 'host' => '10.13.81.92', 'port' => 11211 ), 
        array ( 'host' => '10.13.81.92', 'port' => 11212 ), 
);
$mem->addServers($servers);
$mem->setOption(Memcached::OPT_DISTRIBUTION,Memcached::DISTRIBUTION_CONSISTENT);
$mem->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true);

removing one less server makes the issue go away, here the backtrace..

malloc: top chunk is corrupt
Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff65f34e9 in __GI_abort () at abort.c:79
#2  0x00007ffff664c9a7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff67564c2 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007ffff6652e9c in malloc_printerr (str=str@entry=0x7ffff67545f1 "malloc: top chunk is corrupt") at malloc.c:5350
#4  0x00007ffff6652efe in top_check () at hooks.c:242
#5  0x00007ffff6656642 in malloc_check (sz=21760, caller=<optimized out>) at hooks.c:257
#6  0x00007ffff660bb9d in __GI___qsort_r (b=0x5555560ffb20, n=2720, s=8, cmp=0x7ffff2b14d70 <continuum_item_cmp(void const*, void const*)>, arg=0x0) at msort.c:222
#7  0x00007ffff2b154f1 in update_continuum (ptr=0x5555560ea7a0) at libmemcached/hosts.cc:336
#8  0x00007ffff2b3f02c in php_memc_set_option (intern=0x7ffff5e73080, option=<optimized out>, value=0x7ffff5e1d110)
    at /usr/src/debug/php7-memcached-3.1.3-1.1.x86_64/php_memcached.c:3029
#9  0x00007ffff2b3f4c6 in zim_Memcached_setOption (execute_data=<optimized out>, return_value=0x7fffffffaa20)
    at /usr/src/debug/php7-memcached-3.1.3-1.1.x86_64/php_memcached.c:3281
#10 0x0000555555bb9b2a in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at /usr/src/debug/php7-7.3.0-3.1.x86_64/Zend/zend_vm_execute.h:980
#11 execute_ex (ex=0x2) at /usr/src/debug/php7-7.3.0-3.1.x86_64/Zend/zend_vm_execute.h:55438
#12 0x0000555555bbbf43 in zend_execute (op_array=op_array@entry=0x7ffff5e1d030, return_value=return_value@entry=0x0)
    at /usr/src/debug/php7-7.3.0-3.1.x86_64/Zend/zend_vm_execute.h:60834
#13 0x0000555555b325b0 in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3)
    at /usr/src/debug/php7-7.3.0-3.1.x86_64/Zend/zend.c:1568
#14 0x0000555555aca8a8 in php_execute_script (primary_file=<optimized out>) at /usr/src/debug/php7-7.3.0-3.1.x86_64/main/main.c:2630
#15 0x0000555555bbe70a in do_cli (argc=2, argv=0x555555e33fd0) at /usr/src/debug/php7-7.3.0-3.1.x86_64/sapi/cli/php_cli.c:997
#16 0x000055555597cd3f in main (argc=2, argv=0x555555e33fd0) at /usr/src/debug/php7-7.3.0-3.1.x86_64/sapi/cli/php_cli.c:1389`

It crashes on calling memcached_behavior_set(intern->memc, flag, (uint64_t)lval); with flag MEMCACHED_BEHAVIOR_KETAMA_WEIGHTED and lval 1
afaik.

@sodabrew

This comment has been minimized.

Copy link
Contributor

sodabrew commented Jan 11, 2019

Thank you! I’ll investigate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment