New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FastCGI / Crash with Jemalloc: Failed assertion: "pageind >= map_bias" #221

Closed
J-Siu opened this Issue Dec 9, 2015 · 12 comments

Comments

Projects
None yet
2 participants
@J-Siu

J-Siu commented Dec 9, 2015

I have ran into fast-cgi core dump issue with monkey 1.6.6. After research I notice there is a fix for it already, so I cloned the master branch and compiled it, but I ran into another issue:

Monkey HTTP Server v1.7.0
Built : Dec  9 2015 10:44:08 (/usr/bin/cc 5.2.1)
Home  : http://monkey-project.com
[2015/12/09 12:28:37] [Warning] [fastcgi] configuration error/missing, plugin disabled.
[2015/12/09 12:28:37] [  Error] Plugin 'fastcgi' was not loaded

But I do have the fastcgi.conf. Following are related conf files:

conf
├── CMakeFiles
│   ├── CMakeDirectoryInformation.cmake
│   └── progress.marks
├── cmake_install.cmake
├── Makefile
├── monkey.conf
├── monkey.mime
├── plugins
│   ├── cheetah
│   │   └── cheetah.conf
│   ├── dirlisting
│   │   ├── dirhtml.conf
│   │   └── themes
│   │       ├── bootstrap
│   │       │   ├── entry.theme
│   │       │   ├── footer.theme
│   │       │   └── header.theme
│   │       └── guineo
│   │           ├── entry.theme
│   │           ├── footer.theme
│   │           └── header.theme
│   ├── fastcgi
│   │   └── fastcgi.conf
│   ├── logger
│   │   └── logger.conf
│   └── mandril
│       └── mandril.conf
├── plugins.load
└── sites
    └── default

plugins.load

# Monkey Plugins Loader
# =====================
# Monkey plugins are extended functionalities for Monkey,
# the main directive to load a plugin is LoadPlugin plus
# the absolute path for the desired plugin.
#
# Please check the following list of available plugins:

[PLUGINS]
    # Load /home/john/code/monkey/build/lib/monkey/monkey-auth.so
    # Load /home/john/code/monkey/build/lib/monkey/monkey-cgi.so
    # Load /home/john/code/monkey/build/lib/monkey/monkey-cheetah.so
    # Load /home/john/code/monkey/build/lib/monkey/monkey-dirlisting.so
    Load /home/john/code/monkey/build/lib/monkey/monkey-fastcgi.so
    # Load /home/john/code/monkey/build/lib/monkey/monkey-logger.so
    # Load /home/john/code/monkey/build/lib/monkey/monkey-mandril.so

fastcgi.conf

# FastCGI
# =======
# To enable this plugin you'll need at least one [FASTCGI_SERVER].
#
# This configuration handles php scripts using php5-fpm running on
# localhost or over the network.

[FASTCGI_SERVER]
    # Each server must have a unique name, this is mandatory.
    ServerName php5-fpm

    # Depending on your version of php5-fpm, one of these should be
    # enabled.
    #
    # ServerAddr 127.0.0.1:9000
    ServerPath /var/run/php5-fpm-js.sock
@edsiper

This comment has been minimized.

Show comment
Hide comment
@edsiper

edsiper Dec 9, 2015

Member

thanks for reporting the problem:

  1. "I have ran into fast-cgi core dump issue with monkey 1.6.6"

    would you please specify how can I reproduce this problem ?

  2. Configuration error

    I was able to reproduce the problem, research underway.

Member

edsiper commented Dec 9, 2015

thanks for reporting the problem:

  1. "I have ran into fast-cgi core dump issue with monkey 1.6.6"

    would you please specify how can I reproduce this problem ?

  2. Configuration error

    I was able to reproduce the problem, research underway.

@J-Siu

This comment has been minimized.

Show comment
Hide comment
@J-Siu

J-Siu Dec 9, 2015

For (1), I follow instruction in monkey website http://monkey-project.com/documentation/1.6/getting_started/debian_ubuntu.html to install 1.6.6.
Then I configure fast-cgi as above. Start monkey from command line, then point my browser to the php page. The php page will load correctly, but monkey will core dump at the same time. I am running it in a Virtualbox with Ubuntu 15.10. I did a quick test with seige on a 200k static file and it works fine. Only show issue when I try access a php page using fastcgi.

J-Siu commented Dec 9, 2015

For (1), I follow instruction in monkey website http://monkey-project.com/documentation/1.6/getting_started/debian_ubuntu.html to install 1.6.6.
Then I configure fast-cgi as above. Start monkey from command line, then point my browser to the php page. The php page will load correctly, but monkey will core dump at the same time. I am running it in a Virtualbox with Ubuntu 15.10. I did a quick test with seige on a 200k static file and it works fine. Only show issue when I try access a php page using fastcgi.

@edsiper

This comment has been minimized.

Show comment
Hide comment
@edsiper

edsiper Dec 9, 2015

Member

I was unable to reproduce the segfault problem on v1.6.6. Would you please share the PHP script content ?

Member

edsiper commented Dec 9, 2015

I was unable to reproduce the segfault problem on v1.6.6. Would you please share the PHP script content ?

@J-Siu

This comment has been minimized.

Show comment
Hide comment
@J-Siu

J-Siu Dec 9, 2015

I will try again later today and get back on (1). I uninstalled it last night to make sure it won't conflict with the compile version. The test php is very simple:

<?php
echo 'Hello!'
?>

J-Siu commented Dec 9, 2015

I will try again later today and get back on (1). I uninstalled it last night to make sure it won't conflict with the compile version. The test php is very simple:

<?php
echo 'Hello!'
?>
@J-Siu

This comment has been minimized.

Show comment
Hide comment
@J-Siu

J-Siu Dec 9, 2015

Hmm, actually if I use instruction from that link and do apt-get, I get 1.6.5 as below. So the monkey-project.com repo hasn't update yet.

root@U64S04:/etc/monkey# monkey -b
Monkey HTTP Server v1.6.5
Built : Oct 30 2015 10:28:38 (/usr/bin/cc 4.9.2)
Home  : http://monkey-project.com

[system: Linux]
Linux-4.1.6-040106-generic

[configure]


[setup]
configuration dir: /etc/monkey/


[built-in plugins]
liana               Liana Network Layer
tls                 SSL/TLS Network Layer

Run from command line

root@U64S04:/etc/monkey# monkey 
Monkey HTTP Server v1.6.5
Built : Oct 30 2015 10:28:38 (/usr/bin/cc 4.9.2)
Home  : http://monkey-project.com
[+] Process ID is 17792
[+] Server listening on 0.0.0.0:80
[+] 4 threads, may handle up to 1024 client connections
[+] Loaded Plugins: liana fastcgi 
[+] Linux Features: TCP_FASTOPEN SO_REUSEPORT 
[2015/12/09 15:29:50] [   Info] HTTP Server started
[2015/12/09 15:31:16] [  Error] Segmentation fault (11), code=1, addr=0x7f7a3fffffb0
Aborted (core dumped)

uname info:

root@U64S04:/etc/monkey# uname -a
Linux U64S04 4.2.0-19-generic #23-Ubuntu SMP Wed Nov 11 11:39:30 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@U64S04:/etc/monkey# 

Installed package info:

root@U64S04:/etc/monkey# dpkg -l | grep monkey
ii  monkey                                 1.6.5-1                          amd64        Fast and lightweight Web Server for Linux.

J-Siu commented Dec 9, 2015

Hmm, actually if I use instruction from that link and do apt-get, I get 1.6.5 as below. So the monkey-project.com repo hasn't update yet.

root@U64S04:/etc/monkey# monkey -b
Monkey HTTP Server v1.6.5
Built : Oct 30 2015 10:28:38 (/usr/bin/cc 4.9.2)
Home  : http://monkey-project.com

[system: Linux]
Linux-4.1.6-040106-generic

[configure]


[setup]
configuration dir: /etc/monkey/


[built-in plugins]
liana               Liana Network Layer
tls                 SSL/TLS Network Layer

Run from command line

root@U64S04:/etc/monkey# monkey 
Monkey HTTP Server v1.6.5
Built : Oct 30 2015 10:28:38 (/usr/bin/cc 4.9.2)
Home  : http://monkey-project.com
[+] Process ID is 17792
[+] Server listening on 0.0.0.0:80
[+] 4 threads, may handle up to 1024 client connections
[+] Loaded Plugins: liana fastcgi 
[+] Linux Features: TCP_FASTOPEN SO_REUSEPORT 
[2015/12/09 15:29:50] [   Info] HTTP Server started
[2015/12/09 15:31:16] [  Error] Segmentation fault (11), code=1, addr=0x7f7a3fffffb0
Aborted (core dumped)

uname info:

root@U64S04:/etc/monkey# uname -a
Linux U64S04 4.2.0-19-generic #23-Ubuntu SMP Wed Nov 11 11:39:30 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@U64S04:/etc/monkey# 

Installed package info:

root@U64S04:/etc/monkey# dpkg -l | grep monkey
ii  monkey                                 1.6.5-1                          amd64        Fast and lightweight Web Server for Linux.
@J-Siu

This comment has been minimized.

Show comment
Hide comment
@J-Siu

J-Siu Dec 9, 2015

Compiled 1.6.6 and test, it crashed also:

john@U64S04:~/code/monkey-1.6.6/build$ ./monkey -b
Monkey HTTP Server v1.6.6
Built : Dec  9 2015 15:55:58 (/usr/bin/cc 5.2.1)
Home  : http://monkey-project.com

[system: Linux]
Linux-4.2.0-19-generic

[configure]


[setup]
configuration dir: /home/john/code/monkey-1.6.6/build/conf/


[built-in plugins]
liana               Liana Network Layer

john@U64S04:~/code/monkey-1.6.6/build$ ./monkey 
Monkey HTTP Server v1.6.6
Built : Dec  9 2015 15:55:58 (/usr/bin/cc 5.2.1)
Home  : http://monkey-project.com
[+] Process ID is 21261
[+] Server listening on 0.0.0.0:2001
[+] 4 threads, may handle up to 1024 client connections
[+] Loaded Plugins: liana fastcgi 
[+] Linux Features: TCP_FASTOPEN SO_REUSEPORT 
[2015/12/09 16:04:55] [   Info] HTTP Server started
[2015/12/09 16:06:33] [  Error] Segmentation fault (11), code=1, addr=0x7fefc3ffffb0
Aborted (core dumped)

J-Siu commented Dec 9, 2015

Compiled 1.6.6 and test, it crashed also:

john@U64S04:~/code/monkey-1.6.6/build$ ./monkey -b
Monkey HTTP Server v1.6.6
Built : Dec  9 2015 15:55:58 (/usr/bin/cc 5.2.1)
Home  : http://monkey-project.com

[system: Linux]
Linux-4.2.0-19-generic

[configure]


[setup]
configuration dir: /home/john/code/monkey-1.6.6/build/conf/


[built-in plugins]
liana               Liana Network Layer

john@U64S04:~/code/monkey-1.6.6/build$ ./monkey 
Monkey HTTP Server v1.6.6
Built : Dec  9 2015 15:55:58 (/usr/bin/cc 5.2.1)
Home  : http://monkey-project.com
[+] Process ID is 21261
[+] Server listening on 0.0.0.0:2001
[+] 4 threads, may handle up to 1024 client connections
[+] Loaded Plugins: liana fastcgi 
[+] Linux Features: TCP_FASTOPEN SO_REUSEPORT 
[2015/12/09 16:04:55] [   Info] HTTP Server started
[2015/12/09 16:06:33] [  Error] Segmentation fault (11), code=1, addr=0x7fefc3ffffb0
Aborted (core dumped)
@J-Siu

This comment has been minimized.

Show comment
Hide comment
@J-Siu

J-Siu Dec 9, 2015

For 1.6.6, if I do "./configure --malloc-libc --local", then it won't crash when I load php page with fastcgi.
If I do "./configure --local", then it crash.

J-Siu commented Dec 9, 2015

For 1.6.6, if I do "./configure --malloc-libc --local", then it won't crash when I load php page with fastcgi.
If I do "./configure --local", then it crash.

@edsiper

This comment has been minimized.

Show comment
Hide comment
@edsiper

edsiper Dec 10, 2015

Member

Thanks for catching that. I was able to reproduce the problem, it only happens when using the Jemalloc memory allocator.

I will try to find the root cause...

Member

edsiper commented Dec 10, 2015

Thanks for catching that. I was able to reproduce the problem, it only happens when using the Jemalloc memory allocator.

I will try to find the root cause...

@edsiper

This comment has been minimized.

Show comment
Hide comment
@edsiper

edsiper Dec 10, 2015

Member

research

with jemalloc enabled and debug symbols, valgrind reports the following:

$ valgrind  build/monkey -w 1
==21238== Memcheck, a memory error detector
==21238== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==21238== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==21238== Command: build/monkey -w 1
==21238== 
Monkey HTTP Server v1.6.6
Built : Dec  9 2015 19:38:18 (/usr/bin/cc 5.2.1)
Home  : http://monkey-project.com
[+] Process ID is 21238
[+] Server listening on 0.0.0.0:2001
[+] 1 threads, may handle up to 1024 client connections
[+] Loaded Plugins: liana fastcgi 
[+] Linux Features: TCP_FASTOPEN SO_REUSEPORT 
[2015/12/09 19:39:27] [   Info] HTTP Server started
==21238== Thread 3 monkey: wrk/0:
==21238== Invalid read of size 8
==21238==    at 0x42349C: je_arena_mapbitsp_read (arena.h:525)
==21238==    by 0x42349C: je_arena_mapbits_get (arena.h:532)
==21238==    by 0x42349C: je_arena_mapbits_binind_get (arena.h:573)
==21238==    by 0x42349C: je_arena_salloc (arena.h:991)
==21238==    by 0x42349C: je_isalloc (jemalloc_internal.h:849)
==21238==    by 0x42349C: ifree (jemalloc.c:1228)
==21238==    by 0x42349C: je_free (jemalloc.c:1308)
==21238==    by 0x419646: mk_mem_free (mk_memory.h:96)
==21238==    by 0x419A2B: mk_iov_free_marked (mk_iov.c:143)
==21238==    by 0x4199D6: mk_iov_free (mk_iov.c:126)
==21238==    by 0x6402C19: fcgi_exit (fcgi_handler.c:404)
==21238==    by 0x640309A: cb_fastcgi_on_read (fcgi_handler.c:520)
==21238==    by 0x4126A7: mk_server_worker_loop (mk_server.c:455)
==21238==    by 0x40AB7A: mk_sched_launch_worker_loop (mk_scheduler.c:394)
==21238==    by 0x4E3F6A9: start_thread (pthread_create.c:333)
==21238==    by 0x5360EEC: clone (clone.S:109)
==21238==  Address 0x6000100 is 192 bytes inside a block of size 552 free'd
==21238==    at 0x4C2CE2B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C87C2: fclose@@GLIBC_2.2.5 (iofclose.c:85)
==21238==    by 0x41A9DF: mk_rconf_create (mk_rconf.c:233)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238==  Block was alloc'd at
==21238==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C911C: __fopen_internal (iofopen.c:69)
==21238==    by 0x41A4F2: mk_rconf_create (mk_rconf.c:106)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238== 
==21238== Invalid read of size 8
==21238==    at 0x42376B: je_arena_mapbitsp_read (arena.h:525)
==21238==    by 0x42376B: je_arena_mapbits_get (arena.h:532)
==21238==    by 0x42376B: je_arena_mapbits_binind_get (arena.h:573)
==21238==    by 0x42376B: je_arena_salloc (arena.h:991)
==21238==    by 0x42376B: je_isalloc (jemalloc_internal.h:849)
==21238==    by 0x42376B: je_p2rz (jemalloc_internal.h:884)
==21238==    by 0x42376B: ifree (jemalloc.c:1232)
==21238==    by 0x42376B: je_free (jemalloc.c:1308)
==21238==    by 0x419646: mk_mem_free (mk_memory.h:96)
==21238==    by 0x419A2B: mk_iov_free_marked (mk_iov.c:143)
==21238==    by 0x4199D6: mk_iov_free (mk_iov.c:126)
==21238==    by 0x6402C19: fcgi_exit (fcgi_handler.c:404)
==21238==    by 0x640309A: cb_fastcgi_on_read (fcgi_handler.c:520)
==21238==    by 0x4126A7: mk_server_worker_loop (mk_server.c:455)
==21238==    by 0x40AB7A: mk_sched_launch_worker_loop (mk_scheduler.c:394)
==21238==    by 0x4E3F6A9: start_thread (pthread_create.c:333)
==21238==    by 0x5360EEC: clone (clone.S:109)
==21238==  Address 0x6000100 is 192 bytes inside a block of size 552 free'd
==21238==    at 0x4C2CE2B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C87C2: fclose@@GLIBC_2.2.5 (iofclose.c:85)
==21238==    by 0x41A9DF: mk_rconf_create (mk_rconf.c:233)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238==  Block was alloc'd at
==21238==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C911C: __fopen_internal (iofopen.c:69)
==21238==    by 0x41A4F2: mk_rconf_create (mk_rconf.c:106)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238== 
==21238== Invalid read of size 8
==21238==    at 0x44E8D5: je_arena_mapbitsp_read (arena.h:525)
==21238==    by 0x44E8D5: je_arena_mapbits_get (arena.h:532)
==21238==    by 0x44E8D5: je_arena_mapbits_binind_get (arena.h:573)
==21238==    by 0x44E8D5: je_arena_salloc (arena.h:991)
==21238==    by 0x44E8D5: je_isalloc (jemalloc_internal.h:849)
==21238==    by 0x44E8D5: je_quarantine (quarantine.c:103)
==21238==    by 0x4235E7: je_iqalloct (jemalloc_internal.h:915)
==21238==    by 0x4235E7: je_iqalloc (jemalloc_internal.h:924)
==21238==    by 0x4235E7: ifree (jemalloc.c:1233)
==21238==    by 0x4235E7: je_free (jemalloc.c:1308)
==21238==    by 0x419646: mk_mem_free (mk_memory.h:96)
==21238==    by 0x419A2B: mk_iov_free_marked (mk_iov.c:143)
==21238==    by 0x4199D6: mk_iov_free (mk_iov.c:126)
==21238==    by 0x6402C19: fcgi_exit (fcgi_handler.c:404)
==21238==    by 0x640309A: cb_fastcgi_on_read (fcgi_handler.c:520)
==21238==    by 0x4126A7: mk_server_worker_loop (mk_server.c:455)
==21238==    by 0x40AB7A: mk_sched_launch_worker_loop (mk_scheduler.c:394)
==21238==    by 0x4E3F6A9: start_thread (pthread_create.c:333)
==21238==    by 0x5360EEC: clone (clone.S:109)
==21238==  Address 0x6000100 is 192 bytes inside a block of size 552 free'd
==21238==    at 0x4C2CE2B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C87C2: fclose@@GLIBC_2.2.5 (iofclose.c:85)
==21238==    by 0x41A9DF: mk_rconf_create (mk_rconf.c:233)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238==  Block was alloc'd at
==21238==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C911C: __fopen_internal (iofopen.c:69)
==21238==    by 0x41A4F2: mk_rconf_create (mk_rconf.c:106)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238== 
==21238== Invalid read of size 8
==21238==    at 0x44EF77: je_idalloct (jemalloc_internal.h:898)
==21238==    by 0x44EF77: je_idalloc (jemalloc_internal.h:907)
==21238==    by 0x44EF77: je_quarantine (quarantine.c:157)
==21238==    by 0x4235E7: je_iqalloct (jemalloc_internal.h:915)
==21238==    by 0x4235E7: je_iqalloc (jemalloc_internal.h:924)
==21238==    by 0x4235E7: ifree (jemalloc.c:1233)
==21238==    by 0x4235E7: je_free (jemalloc.c:1308)
==21238==    by 0x419646: mk_mem_free (mk_memory.h:96)
==21238==    by 0x419A2B: mk_iov_free_marked (mk_iov.c:143)
==21238==    by 0x4199D6: mk_iov_free (mk_iov.c:126)
==21238==    by 0x6402C19: fcgi_exit (fcgi_handler.c:404)
==21238==    by 0x640309A: cb_fastcgi_on_read (fcgi_handler.c:520)
==21238==    by 0x4126A7: mk_server_worker_loop (mk_server.c:455)
==21238==    by 0x40AB7A: mk_sched_launch_worker_loop (mk_scheduler.c:394)
==21238==    by 0x4E3F6A9: start_thread (pthread_create.c:333)
==21238==    by 0x5360EEC: clone (clone.S:109)
==21238==  Address 0x6000000 is in a rwx anonymous segment
==21238== 
==21238== Invalid read of size 8
==21238==    at 0x44EF8C: je_arena_mapbitsp_read (arena.h:525)
==21238==    by 0x44EF8C: je_arena_mapbits_get (arena.h:532)
==21238==    by 0x44EF8C: je_arena_dalloc (arena.h:1036)
==21238==    by 0x44EF8C: je_idalloct (jemalloc_internal.h:898)
==21238==    by 0x44EF8C: je_idalloc (jemalloc_internal.h:907)
==21238==    by 0x44EF8C: je_quarantine (quarantine.c:157)
==21238==    by 0x4235E7: je_iqalloct (jemalloc_internal.h:915)
==21238==    by 0x4235E7: je_iqalloc (jemalloc_internal.h:924)
==21238==    by 0x4235E7: ifree (jemalloc.c:1233)
==21238==    by 0x4235E7: je_free (jemalloc.c:1308)
==21238==    by 0x419646: mk_mem_free (mk_memory.h:96)
==21238==    by 0x419A2B: mk_iov_free_marked (mk_iov.c:143)
==21238==    by 0x4199D6: mk_iov_free (mk_iov.c:126)
==21238==    by 0x6402C19: fcgi_exit (fcgi_handler.c:404)
==21238==    by 0x640309A: cb_fastcgi_on_read (fcgi_handler.c:520)
==21238==    by 0x4126A7: mk_server_worker_loop (mk_server.c:455)
==21238==    by 0x40AB7A: mk_sched_launch_worker_loop (mk_scheduler.c:394)
==21238==    by 0x4E3F6A9: start_thread (pthread_create.c:333)
==21238==    by 0x5360EEC: clone (clone.S:109)
==21238==  Address 0x6000100 is 192 bytes inside a block of size 552 free'd
==21238==    at 0x4C2CE2B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C87C2: fclose@@GLIBC_2.2.5 (iofclose.c:85)
==21238==    by 0x41A9DF: mk_rconf_create (mk_rconf.c:233)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238==  Block was alloc'd at
==21238==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C911C: __fopen_internal (iofopen.c:69)
==21238==    by 0x41A4F2: mk_rconf_create (mk_rconf.c:106)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238== 
==21238== Invalid read of size 4
==21238==    at 0x4E41C84: pthread_mutex_lock (pthread_mutex_lock.c:67)
==21238==    by 0x43016E: je_malloc_mutex_lock (mutex.h:77)
==21238==    by 0x43016E: je_arena_dalloc_large (arena.c:1978)
==21238==    by 0x4235E7: je_iqalloct (jemalloc_internal.h:915)
==21238==    by 0x4235E7: je_iqalloc (jemalloc_internal.h:924)
==21238==    by 0x4235E7: ifree (jemalloc.c:1233)
==21238==    by 0x4235E7: je_free (jemalloc.c:1308)
==21238==    by 0x419646: mk_mem_free (mk_memory.h:96)
==21238==    by 0x419A2B: mk_iov_free_marked (mk_iov.c:143)
==21238==    by 0x4199D6: mk_iov_free (mk_iov.c:126)
==21238==    by 0x6402C19: fcgi_exit (fcgi_handler.c:404)
==21238==    by 0x640309A: cb_fastcgi_on_read (fcgi_handler.c:520)
==21238==    by 0x4126A7: mk_server_worker_loop (mk_server.c:455)
==21238==    by 0x40AB7A: mk_sched_launch_worker_loop (mk_scheduler.c:394)
==21238==    by 0x4E3F6A9: start_thread (pthread_create.c:333)
==21238==    by 0x5360EEC: clone (clone.S:109)
==21238==  Address 0x3ffff8 is not stack'd, malloc'd or (recently) free'd
==21238== 
[2015/12/09 19:39:29] [  Error] Segmentation fault (11), code=1, addr=0x3ffff8

Member

edsiper commented Dec 10, 2015

research

with jemalloc enabled and debug symbols, valgrind reports the following:

$ valgrind  build/monkey -w 1
==21238== Memcheck, a memory error detector
==21238== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==21238== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==21238== Command: build/monkey -w 1
==21238== 
Monkey HTTP Server v1.6.6
Built : Dec  9 2015 19:38:18 (/usr/bin/cc 5.2.1)
Home  : http://monkey-project.com
[+] Process ID is 21238
[+] Server listening on 0.0.0.0:2001
[+] 1 threads, may handle up to 1024 client connections
[+] Loaded Plugins: liana fastcgi 
[+] Linux Features: TCP_FASTOPEN SO_REUSEPORT 
[2015/12/09 19:39:27] [   Info] HTTP Server started
==21238== Thread 3 monkey: wrk/0:
==21238== Invalid read of size 8
==21238==    at 0x42349C: je_arena_mapbitsp_read (arena.h:525)
==21238==    by 0x42349C: je_arena_mapbits_get (arena.h:532)
==21238==    by 0x42349C: je_arena_mapbits_binind_get (arena.h:573)
==21238==    by 0x42349C: je_arena_salloc (arena.h:991)
==21238==    by 0x42349C: je_isalloc (jemalloc_internal.h:849)
==21238==    by 0x42349C: ifree (jemalloc.c:1228)
==21238==    by 0x42349C: je_free (jemalloc.c:1308)
==21238==    by 0x419646: mk_mem_free (mk_memory.h:96)
==21238==    by 0x419A2B: mk_iov_free_marked (mk_iov.c:143)
==21238==    by 0x4199D6: mk_iov_free (mk_iov.c:126)
==21238==    by 0x6402C19: fcgi_exit (fcgi_handler.c:404)
==21238==    by 0x640309A: cb_fastcgi_on_read (fcgi_handler.c:520)
==21238==    by 0x4126A7: mk_server_worker_loop (mk_server.c:455)
==21238==    by 0x40AB7A: mk_sched_launch_worker_loop (mk_scheduler.c:394)
==21238==    by 0x4E3F6A9: start_thread (pthread_create.c:333)
==21238==    by 0x5360EEC: clone (clone.S:109)
==21238==  Address 0x6000100 is 192 bytes inside a block of size 552 free'd
==21238==    at 0x4C2CE2B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C87C2: fclose@@GLIBC_2.2.5 (iofclose.c:85)
==21238==    by 0x41A9DF: mk_rconf_create (mk_rconf.c:233)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238==  Block was alloc'd at
==21238==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C911C: __fopen_internal (iofopen.c:69)
==21238==    by 0x41A4F2: mk_rconf_create (mk_rconf.c:106)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238== 
==21238== Invalid read of size 8
==21238==    at 0x42376B: je_arena_mapbitsp_read (arena.h:525)
==21238==    by 0x42376B: je_arena_mapbits_get (arena.h:532)
==21238==    by 0x42376B: je_arena_mapbits_binind_get (arena.h:573)
==21238==    by 0x42376B: je_arena_salloc (arena.h:991)
==21238==    by 0x42376B: je_isalloc (jemalloc_internal.h:849)
==21238==    by 0x42376B: je_p2rz (jemalloc_internal.h:884)
==21238==    by 0x42376B: ifree (jemalloc.c:1232)
==21238==    by 0x42376B: je_free (jemalloc.c:1308)
==21238==    by 0x419646: mk_mem_free (mk_memory.h:96)
==21238==    by 0x419A2B: mk_iov_free_marked (mk_iov.c:143)
==21238==    by 0x4199D6: mk_iov_free (mk_iov.c:126)
==21238==    by 0x6402C19: fcgi_exit (fcgi_handler.c:404)
==21238==    by 0x640309A: cb_fastcgi_on_read (fcgi_handler.c:520)
==21238==    by 0x4126A7: mk_server_worker_loop (mk_server.c:455)
==21238==    by 0x40AB7A: mk_sched_launch_worker_loop (mk_scheduler.c:394)
==21238==    by 0x4E3F6A9: start_thread (pthread_create.c:333)
==21238==    by 0x5360EEC: clone (clone.S:109)
==21238==  Address 0x6000100 is 192 bytes inside a block of size 552 free'd
==21238==    at 0x4C2CE2B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C87C2: fclose@@GLIBC_2.2.5 (iofclose.c:85)
==21238==    by 0x41A9DF: mk_rconf_create (mk_rconf.c:233)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238==  Block was alloc'd at
==21238==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C911C: __fopen_internal (iofopen.c:69)
==21238==    by 0x41A4F2: mk_rconf_create (mk_rconf.c:106)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238== 
==21238== Invalid read of size 8
==21238==    at 0x44E8D5: je_arena_mapbitsp_read (arena.h:525)
==21238==    by 0x44E8D5: je_arena_mapbits_get (arena.h:532)
==21238==    by 0x44E8D5: je_arena_mapbits_binind_get (arena.h:573)
==21238==    by 0x44E8D5: je_arena_salloc (arena.h:991)
==21238==    by 0x44E8D5: je_isalloc (jemalloc_internal.h:849)
==21238==    by 0x44E8D5: je_quarantine (quarantine.c:103)
==21238==    by 0x4235E7: je_iqalloct (jemalloc_internal.h:915)
==21238==    by 0x4235E7: je_iqalloc (jemalloc_internal.h:924)
==21238==    by 0x4235E7: ifree (jemalloc.c:1233)
==21238==    by 0x4235E7: je_free (jemalloc.c:1308)
==21238==    by 0x419646: mk_mem_free (mk_memory.h:96)
==21238==    by 0x419A2B: mk_iov_free_marked (mk_iov.c:143)
==21238==    by 0x4199D6: mk_iov_free (mk_iov.c:126)
==21238==    by 0x6402C19: fcgi_exit (fcgi_handler.c:404)
==21238==    by 0x640309A: cb_fastcgi_on_read (fcgi_handler.c:520)
==21238==    by 0x4126A7: mk_server_worker_loop (mk_server.c:455)
==21238==    by 0x40AB7A: mk_sched_launch_worker_loop (mk_scheduler.c:394)
==21238==    by 0x4E3F6A9: start_thread (pthread_create.c:333)
==21238==    by 0x5360EEC: clone (clone.S:109)
==21238==  Address 0x6000100 is 192 bytes inside a block of size 552 free'd
==21238==    at 0x4C2CE2B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C87C2: fclose@@GLIBC_2.2.5 (iofclose.c:85)
==21238==    by 0x41A9DF: mk_rconf_create (mk_rconf.c:233)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238==  Block was alloc'd at
==21238==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C911C: __fopen_internal (iofopen.c:69)
==21238==    by 0x41A4F2: mk_rconf_create (mk_rconf.c:106)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238== 
==21238== Invalid read of size 8
==21238==    at 0x44EF77: je_idalloct (jemalloc_internal.h:898)
==21238==    by 0x44EF77: je_idalloc (jemalloc_internal.h:907)
==21238==    by 0x44EF77: je_quarantine (quarantine.c:157)
==21238==    by 0x4235E7: je_iqalloct (jemalloc_internal.h:915)
==21238==    by 0x4235E7: je_iqalloc (jemalloc_internal.h:924)
==21238==    by 0x4235E7: ifree (jemalloc.c:1233)
==21238==    by 0x4235E7: je_free (jemalloc.c:1308)
==21238==    by 0x419646: mk_mem_free (mk_memory.h:96)
==21238==    by 0x419A2B: mk_iov_free_marked (mk_iov.c:143)
==21238==    by 0x4199D6: mk_iov_free (mk_iov.c:126)
==21238==    by 0x6402C19: fcgi_exit (fcgi_handler.c:404)
==21238==    by 0x640309A: cb_fastcgi_on_read (fcgi_handler.c:520)
==21238==    by 0x4126A7: mk_server_worker_loop (mk_server.c:455)
==21238==    by 0x40AB7A: mk_sched_launch_worker_loop (mk_scheduler.c:394)
==21238==    by 0x4E3F6A9: start_thread (pthread_create.c:333)
==21238==    by 0x5360EEC: clone (clone.S:109)
==21238==  Address 0x6000000 is in a rwx anonymous segment
==21238== 
==21238== Invalid read of size 8
==21238==    at 0x44EF8C: je_arena_mapbitsp_read (arena.h:525)
==21238==    by 0x44EF8C: je_arena_mapbits_get (arena.h:532)
==21238==    by 0x44EF8C: je_arena_dalloc (arena.h:1036)
==21238==    by 0x44EF8C: je_idalloct (jemalloc_internal.h:898)
==21238==    by 0x44EF8C: je_idalloc (jemalloc_internal.h:907)
==21238==    by 0x44EF8C: je_quarantine (quarantine.c:157)
==21238==    by 0x4235E7: je_iqalloct (jemalloc_internal.h:915)
==21238==    by 0x4235E7: je_iqalloc (jemalloc_internal.h:924)
==21238==    by 0x4235E7: ifree (jemalloc.c:1233)
==21238==    by 0x4235E7: je_free (jemalloc.c:1308)
==21238==    by 0x419646: mk_mem_free (mk_memory.h:96)
==21238==    by 0x419A2B: mk_iov_free_marked (mk_iov.c:143)
==21238==    by 0x4199D6: mk_iov_free (mk_iov.c:126)
==21238==    by 0x6402C19: fcgi_exit (fcgi_handler.c:404)
==21238==    by 0x640309A: cb_fastcgi_on_read (fcgi_handler.c:520)
==21238==    by 0x4126A7: mk_server_worker_loop (mk_server.c:455)
==21238==    by 0x40AB7A: mk_sched_launch_worker_loop (mk_scheduler.c:394)
==21238==    by 0x4E3F6A9: start_thread (pthread_create.c:333)
==21238==    by 0x5360EEC: clone (clone.S:109)
==21238==  Address 0x6000100 is 192 bytes inside a block of size 552 free'd
==21238==    at 0x4C2CE2B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C87C2: fclose@@GLIBC_2.2.5 (iofclose.c:85)
==21238==    by 0x41A9DF: mk_rconf_create (mk_rconf.c:233)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238==  Block was alloc'd at
==21238==    at 0x4C2BBCF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21238==    by 0x52C911C: __fopen_internal (iofopen.c:69)
==21238==    by 0x41A4F2: mk_rconf_create (mk_rconf.c:106)
==21238==    by 0x408BB6: mk_config_read_files (mk_config.c:289)
==21238==    by 0x40938A: mk_config_start_configure (mk_config.c:474)
==21238==    by 0x407F2D: mk_server_setup (monkey.c:93)
==21238==    by 0x407AEE: main (monkey.c:306)
==21238== 
==21238== Invalid read of size 4
==21238==    at 0x4E41C84: pthread_mutex_lock (pthread_mutex_lock.c:67)
==21238==    by 0x43016E: je_malloc_mutex_lock (mutex.h:77)
==21238==    by 0x43016E: je_arena_dalloc_large (arena.c:1978)
==21238==    by 0x4235E7: je_iqalloct (jemalloc_internal.h:915)
==21238==    by 0x4235E7: je_iqalloc (jemalloc_internal.h:924)
==21238==    by 0x4235E7: ifree (jemalloc.c:1233)
==21238==    by 0x4235E7: je_free (jemalloc.c:1308)
==21238==    by 0x419646: mk_mem_free (mk_memory.h:96)
==21238==    by 0x419A2B: mk_iov_free_marked (mk_iov.c:143)
==21238==    by 0x4199D6: mk_iov_free (mk_iov.c:126)
==21238==    by 0x6402C19: fcgi_exit (fcgi_handler.c:404)
==21238==    by 0x640309A: cb_fastcgi_on_read (fcgi_handler.c:520)
==21238==    by 0x4126A7: mk_server_worker_loop (mk_server.c:455)
==21238==    by 0x40AB7A: mk_sched_launch_worker_loop (mk_scheduler.c:394)
==21238==    by 0x4E3F6A9: start_thread (pthread_create.c:333)
==21238==    by 0x5360EEC: clone (clone.S:109)
==21238==  Address 0x3ffff8 is not stack'd, malloc'd or (recently) free'd
==21238== 
[2015/12/09 19:39:29] [  Error] Segmentation fault (11), code=1, addr=0x3ffff8

@edsiper

This comment has been minimized.

Show comment
Hide comment
@edsiper

edsiper Dec 10, 2015

Member

I tested jemalloc 3.x and the latest 4.0.4, same issue, I've requested assistance from the Jemalloc dev team:

http://www.canonware.com/pipermail/jemalloc-discuss/2015-December/001219.html

Member

edsiper commented Dec 10, 2015

I tested jemalloc 3.x and the latest 4.0.4, same issue, I've requested assistance from the Jemalloc dev team:

http://www.canonware.com/pipermail/jemalloc-discuss/2015-December/001219.html

@J-Siu

This comment has been minimized.

Show comment
Hide comment
@J-Siu

J-Siu Dec 10, 2015

Should we split the two issues ... hmm ... into two issues? LOL
Btw, how about the config bug?

J-Siu commented Dec 10, 2015

Should we split the two issues ... hmm ... into two issues? LOL
Btw, how about the config bug?

@edsiper

This comment has been minimized.

Show comment
Hide comment
@edsiper

edsiper Dec 10, 2015

Member

Not required, master branch (development) have been updated with the right API usage:

4c43976: Plugins: fastcgi: fix usage of config API - Eduardo Silva
5987bab: Plugins: tls: fix usage of config API - Eduardo Silva
a57b5f6: Plugins: cheetah: fix usage of config API - Eduardo Silva
25bf806: Plugins: logger: fix usage of config API - Eduardo Silva
c36e6e2: Plugins: dirlisting: fix usage of config API - Eduardo Silva
3f9f14c: Plugins: mandril: fix usage of config API - Eduardo Silva

Member

edsiper commented Dec 10, 2015

Not required, master branch (development) have been updated with the right API usage:

4c43976: Plugins: fastcgi: fix usage of config API - Eduardo Silva
5987bab: Plugins: tls: fix usage of config API - Eduardo Silva
a57b5f6: Plugins: cheetah: fix usage of config API - Eduardo Silva
25bf806: Plugins: logger: fix usage of config API - Eduardo Silva
c36e6e2: Plugins: dirlisting: fix usage of config API - Eduardo Silva
3f9f14c: Plugins: mandril: fix usage of config API - Eduardo Silva

@edsiper edsiper changed the title from Master branch(2015-12-09) not loading fastcgi/plugins config to FastCGI / Crash with Jemalloc: Failed assertion: "pageind >= map_bias" Dec 10, 2015

edsiper added a commit that referenced this issue Dec 10, 2015

Plugins: fastcgi: fix crash when Jemalloc is enabled (Fix #221)
When Monkey is built with Jemalloc, when the plugin release the resources
associated to a FastCGI request, it crash. Jemalloc reports the following
error:

  Failed assertion: "pageind >= map_bias"

This problem happens because when the plugin compose the FastCGI request
internally, it needs to duplicate some string buffers and it was using
a direct strdup(3) call, which is wrong as Monkey provides wrappers for
memory and string manipulation which use the system allocator (malloc)
or the jemalloc functions. The problem was:

 - Duplicate string buffer with strdup(3)
 - Release buffer with Jemalloc

This patch fix the single line that generated the problem.

Signed-off-by: Eduardo Silva <eduardo@monkey.io>

@edsiper edsiper closed this in 7fd3e43 Dec 10, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment