Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

if addServers too much server, and OPT_LIBKETAMA_COMPATIBLE is true, core dump will appear #407

Open
khadgarmage opened this issue Aug 9, 2018 · 4 comments
Milestone

Comments

@khadgarmage
Copy link

@khadgarmage khadgarmage commented Aug 9, 2018

hi, @sodabrew if addServers too much server, and OPT_LIBKETAMA_COMPATIBLE is true, core dump will appear
PHP env:
PHP 5.3.29 or PHP 5.5.15
Reproduce:

<?php
$mem=new Memcached();
$servers = array (
        array ( 'host' => '10.13.81.227', 'port' => 11211 ),
        array ( 'host' => '10.13.81.227', 'port' => 11212 ),
        array ( 'host' => '10.13.81.227', 'port' => 11213 ),
        array ( 'host' => '10.13.81.155', 'port' => 11211 ),
        array ( 'host' => '10.13.81.155', 'port' => 11212 ),
        array ( 'host' => '10.13.81.155', 'port' => 11213 ),
        array ( 'host' => '10.13.81.231', 'port' => 11211 ),
        array ( 'host' => '10.13.81.231', 'port' => 11212 ),
        array ( 'host' => '10.13.81.231', 'port' => 11213 ),
        array ( 'host' => '10.13.81.125', 'port' => 11211 ),
        array ( 'host' => '10.13.81.125', 'port' => 11212 ),
        array ( 'host' => '10.13.81.125', 'port' => 11213 ),
        array ( 'host' => '10.13.81.13', 'port' => 11211 ),
        array ( 'host' => '10.13.81.13', 'port' => 11212 ),
        array ( 'host' => '10.13.81.13', 'port' => 11213 ),
        array ( 'host' => '10.13.81.92', 'port' => 11211 ),
        array ( 'host' => '10.13.81.92', 'port' => 11212 ),
        array ( 'host' => '10.13.81.92', 'port' => 11213 ),
        array ( 'host' => '10.13.81.209', 'port' => 11211 ),
        array ( 'host' => '10.13.81.209', 'port' => 11212 ),
        array ( 'host' => '10.13.81.209', 'port' => 11213 ),
        array ( 'host' => '10.13.81.177', 'port' => 11211 ),
        array ( 'host' => '10.13.81.177', 'port' => 11212 ),
        array ( 'host' => '10.13.81.177', 'port' => 11213 ),
        array ( 'host' => '10.13.81.70', 'port' => 11211 ),
        array ( 'host' => '10.13.81.70', 'port' => 11212 ),
        array ( 'host' => '10.13.81.70', 'port' => 11213 ),
);
$mem->addServers($servers);
$mem->setOption(Memcached::OPT_DISTRIBUTION,Memcached::DISTRIBUTION_CONSISTENT);
$mem->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true);
$mem->set('name',"testdfdsafdsafdsafda",15);
echo $mem->get('name');

Result On cli:

[ffs-amazon-web root@ip-10-13-80-98 ~]# php t.php
*** glibc detected *** php: double free or corruption (out): 0x000000000132c210 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3fca275e76]
/lib64/libc.so.6[0x3fca2789b3]
/lib64/libc.so.6(qsort_r+0x2a8)[0x3fca234f38]
/usr/lib/libmemcached.so.11(+0x15812)[0x7fd6743c4812]
/usr/lib64/php/modules/memcached.so(+0x79ef)[0x7fd6745e79ef]
/usr/lib64/php/modules/memcached.so(+0x7bc7)[0x7fd6745e7bc7]
php[0x63eef3]
php(execute_ex+0x38)[0x6307a8]
php(zend_execute_scripts+0x18c)[0x5bbbdc]
php(php_execute_script+0x1f0)[0x55c5c0]
php[0x66d233]
php[0x66da38]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3fca21ed5d]
php[0x422939]
======= Memory map: ========
00400000-00754000 r-xp 00000000 ca:01 148069                             /usr/bin/php
00954000-009e8000 rw-p 00354000 ca:01 148069                             /usr/bin/php
009e8000-00a03000 rw-p 00000000 00:00 0
00de7000-00df2000 rw-p 003e7000 ca:01 148069                             /usr/bin/php
01033000-01353000 rw-p 00000000 00:00 0                                  [heap]
391fa00000-391fa1d000 r-xp 00000000 ca:01 237783                         /lib64/libselinux.so.1
391fa1d000-391fc1c000 ---p 0001d000 ca:01 237783                         /lib64/libselinux.so.1
391fc1c000-391fc1d000 r--p 0001c000 ca:01 237783                         /lib64/libselinux.so.1
391fc1d000-391fc1e000 rw-p 0001d000 ca:01 237783                         /lib64/libselinux.so.1
391fc1e000-391fc1f000 rw-p 00000000 00:00 0
391fe00000-391fe22000 r-xp 00000000 ca:01 239044                         /lib64/libncurses.so.5.7
391fe22000-3920021000 ---p 00022000 ca:01 239044                         /lib64/libncurses.so.5.7
3920021000-3920022000 rw-p 00021000 ca:01 239044                         /lib64/libncurses.so.5.7
@khadgarmage
Copy link
Author

@khadgarmage khadgarmage commented Aug 9, 2018

if the code like as followed, the bug will not appear:

<?php
$mem=new Memcached();
$servers = array (
        array ( 'host' => '10.13.81.227', 'port' => 11211 ),
        array ( 'host' => '10.13.81.227', 'port' => 11212 ),
        array ( 'host' => '10.13.81.227', 'port' => 11213 ),
);
$mem->addServers($servers);
$mem->setOption(Memcached::OPT_DISTRIBUTION,Memcached::DISTRIBUTION_CONSISTENT);
$mem->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true);
$mem->set('name',"testdfdsafdsafdsafda",15);
echo $mem->get('name');
@sodabrew sodabrew added this to the 2.3.0 milestone Dec 22, 2018
@sodabrew
Copy link
Contributor

@sodabrew sodabrew commented Dec 22, 2018

I am not able to reproduce this with memcached 3.1.x with PHP 7.x.

@crrodriguez
Copy link

@crrodriguez crrodriguez commented Jan 11, 2019

I can reproduce this with this slightly shorter code

<?php
$mem=new Memcached();
$servers = array (
        array ( 'host' => '10.13.81.227', 'port' => 11211 ), 
        array ( 'host' => '10.13.81.227', 'port' => 11212 ), 
        array ( 'host' => '10.13.81.227', 'port' => 11213 ), 
        array ( 'host' => '10.13.81.155', 'port' => 11211 ), 
        array ( 'host' => '10.13.81.155', 'port' => 11212 ), 
        array ( 'host' => '10.13.81.155', 'port' => 11213 ), 
        array ( 'host' => '10.13.81.231', 'port' => 11211 ), 
        array ( 'host' => '10.13.81.231', 'port' => 11212 ), 
        array ( 'host' => '10.13.81.231', 'port' => 11213 ), 
        array ( 'host' => '10.13.81.125', 'port' => 11211 ), 
        array ( 'host' => '10.13.81.125', 'port' => 11212 ), 
        array ( 'host' => '10.13.81.125', 'port' => 11213 ), 
        array ( 'host' => '10.13.81.13', 'port' => 11211 ), 
        array ( 'host' => '10.13.81.13', 'port' => 11212 ), 
        array ( 'host' => '10.13.81.13', 'port' => 11213 ), 
        array ( 'host' => '10.13.81.92', 'port' => 11211 ), 
        array ( 'host' => '10.13.81.92', 'port' => 11212 ), 
);
$mem->addServers($servers);
$mem->setOption(Memcached::OPT_DISTRIBUTION,Memcached::DISTRIBUTION_CONSISTENT);
$mem->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true);

removing one less server makes the issue go away, here the backtrace..

malloc: top chunk is corrupt
Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff65f34e9 in __GI_abort () at abort.c:79
#2  0x00007ffff664c9a7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff67564c2 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007ffff6652e9c in malloc_printerr (str=str@entry=0x7ffff67545f1 "malloc: top chunk is corrupt") at malloc.c:5350
#4  0x00007ffff6652efe in top_check () at hooks.c:242
#5  0x00007ffff6656642 in malloc_check (sz=21760, caller=<optimized out>) at hooks.c:257
#6  0x00007ffff660bb9d in __GI___qsort_r (b=0x5555560ffb20, n=2720, s=8, cmp=0x7ffff2b14d70 <continuum_item_cmp(void const*, void const*)>, arg=0x0) at msort.c:222
#7  0x00007ffff2b154f1 in update_continuum (ptr=0x5555560ea7a0) at libmemcached/hosts.cc:336
#8  0x00007ffff2b3f02c in php_memc_set_option (intern=0x7ffff5e73080, option=<optimized out>, value=0x7ffff5e1d110)
    at /usr/src/debug/php7-memcached-3.1.3-1.1.x86_64/php_memcached.c:3029
#9  0x00007ffff2b3f4c6 in zim_Memcached_setOption (execute_data=<optimized out>, return_value=0x7fffffffaa20)
    at /usr/src/debug/php7-memcached-3.1.3-1.1.x86_64/php_memcached.c:3281
#10 0x0000555555bb9b2a in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at /usr/src/debug/php7-7.3.0-3.1.x86_64/Zend/zend_vm_execute.h:980
#11 execute_ex (ex=0x2) at /usr/src/debug/php7-7.3.0-3.1.x86_64/Zend/zend_vm_execute.h:55438
#12 0x0000555555bbbf43 in zend_execute (op_array=op_array@entry=0x7ffff5e1d030, return_value=return_value@entry=0x0)
    at /usr/src/debug/php7-7.3.0-3.1.x86_64/Zend/zend_vm_execute.h:60834
#13 0x0000555555b325b0 in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3)
    at /usr/src/debug/php7-7.3.0-3.1.x86_64/Zend/zend.c:1568
#14 0x0000555555aca8a8 in php_execute_script (primary_file=<optimized out>) at /usr/src/debug/php7-7.3.0-3.1.x86_64/main/main.c:2630
#15 0x0000555555bbe70a in do_cli (argc=2, argv=0x555555e33fd0) at /usr/src/debug/php7-7.3.0-3.1.x86_64/sapi/cli/php_cli.c:997
#16 0x000055555597cd3f in main (argc=2, argv=0x555555e33fd0) at /usr/src/debug/php7-7.3.0-3.1.x86_64/sapi/cli/php_cli.c:1389`

It crashes on calling memcached_behavior_set(intern->memc, flag, (uint64_t)lval); with flag MEMCACHED_BEHAVIOR_KETAMA_WEIGHTED and lval 1
afaik.

@sodabrew
Copy link
Contributor

@sodabrew sodabrew commented Jan 11, 2019

Thank you! I’ll investigate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants