GDB utilities for Nginx, ngx_lua, LuaJIT, and etc.
- Name
- Status
- Synopsis
- Description
- Commands
- Prerequisites
- Installation
- Authors
- Copyright and License
- See Also
This is still under early development.
# add the path of nginx.py and ngxlua.py modules to the PYTHONPATH env
(gdb) source luajit21.py
(gdb) lvmst
current VM state: C code from intperpreted Lua
(gdb) lbt
builtin#166
builtin#195
builtin#187
@/home/agentzh/git/lua-resty-core/lib/resty/core/regex.lua:588
content_by_lua:10
(gdb) source ngx-lua.gdb
(gdb) source luajit20.gdb
(gdb) lreg L &ngx_http_lua_ctx_tables_key
<tab: 0x412a68c8>
(gdb) lfunc regex.lua 444
Found function (GCfunc*)0x4025e168 at @/home/agentzh/git/lua-resty-core/lib/resty/core/regex.lua:444
(gdb) lproto regex.lua 444
Found proto (GCproto*)0x4025f380 at @/home/agentzh/git/lua-resty-core/lib/resty/core/regex.lua:444
(gdb) luv (GCfunc*)0x4025e168
0x4025e168
Found 23 upvalues.
upvalue "parse_regex_opts": value=(TValue*)0x4025df60 value_type=func closed=1
upvalue "type": value=(TValue*)0x4025e1e8 value_type=func closed=1
upvalue "tostring": value=(TValue*)0x4025e208 value_type=func closed=1
upvalue "band": value=(TValue*)0x4025dd88 value_type=func closed=1
upvalue "FLAG_COMPILE_ONCE": value=(TValue*)0x41b8daf8 value_type=number closed=1
upvalue "regex_cache": value=(TValue*)0x4025df80 value_type=table closed=1
upvalue "get_string_buf": value=(TValue*)0x4025dfa0 value_type=func closed=1
upvalue "MAX_ERR_MSG_LEN": value=(TValue*)0x4025dfc0 value_type=number closed=1
upvalue "C": value=(TValue*)0x41b8da38 value_type=userdata closed=1
...
(gdb) lval (TValue*)0x41b8da38
udata type: ffi clib
payload len: 16
payload ptr: 0x41b81df8
CLibrary handle: (void*)0x0
CLibrary cache: (GCtab*)0x41b8d188
(gdb) lval (TValue*)0x41907840
type cdata
cdata object: (GCcdata*)0x41aae7f0
cdata value pointer: (void*)0x41aae7f8
ctype object: (CType*)0x40268e70
ctype size: 1 byte(s)
ctype type: func
ctype element name: ngx_http_lua_ffi_destroy_regex
This toolkit provides various gdb extension commands for analyzing core dump files for nginx and/or luajit.
The following gdb commands are supported:
syntax: lbt [L]
syntax: lbt full [L]
file luajit21.py
Fetch the current backtrace from the current running Lua thread (when no L argument is given) or the Lua thread specified by the lua_State pointer.
The backtrace format is the same as the one used by the lj-lua-bt tool.
When analyzing ngx_lua's processes, this tool requires the Python module files nginx.py and ngxlua.py to obtain the global Lua state. You need to add the path of these .py files to the PYTHONPATH environment variable before starting gdb.
Below is an example:
(gdb) source luajit21.py
(gdb) lbt
builtin#166
builtin#195
builtin#187
@/home/agentzh/git/lua-resty-core/lib/resty/core/regex.lua:588
content_by_lua:10
You can also explicitly specify the Lua thread state you want to analyze, for instance,
(gdb) lbt 0x169e0e0
The lbt full command works like bt full, which dumps out the names and values of all the local variables (including function parameters) in every Lua function frame. For example,
(gdb) lbt full
C:ngx_http_lua_socket_tcp_receive
@/home/agentzh/git/lua-resty-mysql/lib/resty/mysql.lua:191
local "self":
table (0x40f181a8)
local "sock":
table (0x40f181b0)
@/home/agentzh/git/lua-resty-mysql/lib/resty/mysql.lua:530
local "self":
table (0x40f18148)
local "opts":
table (0x40f18150)
local "sock":
table (0x40f18158)
local "max_packet_size":
int 1048576
local "ok":
int 1
local "err":
nil
local "database":
string: "world" (len 5)
local "user":
string: "ngx_test" (len 8)
local "pool":
string: "ngx_test:world:127.0.0.1:3306" (len 29)
local "host":
string: "127.0.0.1" (len 9)
Only LuaJIT 2.1 is supported.
syntax: lvmst [L]
file luajit21.py
Prints out the current state of the LuaJIT 2.1 VM.
Below is an example,
(gdb) source luajit21.py
(gdb) lvmst
current VM state: C code from intperpreted Lua
You can also explicitly specify the lua VM state you want to analyze, for instance,
(gdb) lvmst 0x169e0e0
current VM state: C code from intperpreted Lua
You can specify any Lua thread's state in the VM you want to analyze.
The following VM states are supported:
- Compiled Lua code (trace #N)
- Interpreted
- C code (from interpreted Lua code)
- Garbage collector (from interpreter)
- Garbage collector (from compiled Lua code)
- Trace exit handler
- Trace recorder
- Optimizer
- Assembler
syntax: lval tvalue
syntax: lval gcobj
file luajit21.py
Prints out the content in a TValue or the dereferenced value (like GCtab and GCproto) from its pointer. By default the argument is assumed to be a TValue* pointer value.
Below are some examples:
(gdb) lval (TValue*)0x41f1f450
table (GCtab*)0x41f1f688 (narr=5, nrec=1):
[1] =
int 32
[2] =
true
[4] =
string: "hello" (len 5)
key:
string: "dog" (len 3)
value:
number 21.5
(gdb) lval (GCtab*)0x41f1f688
table (GCtab*)0x41f1f688 (narr=5, nrec=1):
[1] =
int 32
[2] =
true
[4] =
string: "hello" (len 5)
key:
string: "dog" (len 3)
value:
number 21.5
(gdb) lval 0x41f1f440
nil
(gdb) lval 0x41f1f458
Lua function (GCfunc*)0x4188e018 at @.../regex.lua:418
(gdb) lval 0x41f1f460
int 1
syntax: ltrace
syntax: ltrace traceno
file luajit21.py
Dump the contents in a LuaJIT trace object specified by the trace number (starting from 1).
For example,
(gdb) ltrace 658
(GCtrace*)0x40800268
machine code size: 335
machine code start addr: 0x7f2435c85870
machine code end addr: 0x7f2435c859bf
@.../lua/waf-core.lua:1202
The starting address and end address of the machine code region in the output can be used to obtain the machine code dump for the trace:
(gdb) set disassembly-flavor intel
(gdb) disas 0x7f2435c85870, 0x7f2435c859bf
Dump of assembler code from 0x7f2435c85870 to 0x7f2435c859bf:
0x00007f2435c85870: mov DWORD PTR ds:0x40ff1410,0x292
0x00007f2435c8587b: cmp DWORD PTR [rdx-0x8],0x419746c8
0x00007f2435c85882: jne 0x7f2435cd0010
0x00007f2435c85888: cmp DWORD PTR [rdx+0x4],0xfffffffb
0x00007f2435c8588c: jne 0x7f2435cd0010
0x00007f2435c85892: mov ebp,DWORD PTR [rdx]
...
0x00007f2435c859b4: mov r14d,0x40ff1f90
0x00007f2435c859ba: jmp 0x7f2444a899fa <lj_vm_exit_interp>
End of assembler dump.
When being invoked without any arguments, this command just prints out the total number of traces, for instance,
(gdb) ltrace
Found 253 traces.
syntax: lir traceno
file luajit21.py
Dumps out the IR code (with CPU register and snapshot details) for the LuaJIT trace specified by its trace number. The output format is the same as LuaJIT's own -jdump=+rs output.
For instance,
(gdb) lir 20
(GCtrace*)0x419ff678
IR count: 16
---- TRACE 20 start 19/? meteor.lua-3.lua:64
---- TRACE 20 IR
0001 rbp int SLOAD #13 PI
.... SNAP #0 [ ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 0001 ---- ---- 0001 ]
0002 rbx > tab SLOAD #11 T
0003 int FLOAD 0002 tab.asize
0004 > int ABC 0003 0001
0005 rbx p32 FLOAD 0002 tab.array
0006 p32 AREF 0005 0001
0007 > int ALOAD 0006
0008 rsi > str SLOAD #12 T
0009 str TOSTR 0001 INT
0010 rdi p32 BUFHDR [0x41fe1414] RESET
0011 rdi p32 BUFPUT 0010 0008
0012 rdi p32 BUFPUT 0011 "\,b"
0013 rdi p32 BUFPUT 0012 0009
0014 rax str BUFSTR 0013 0010
0015 rbp int ADD 0001 +1
.... SNAP #1 [ ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 0014 ]
0016 > int LE 0015 +99
.... SNAP #2 [ ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- 0014 0015 ---- ---- 0015 ]
syntax: lmainL
file luajit21.py
Prints out the lua_State pointer value for the main LuaJIT VM state. For example,
(gdb) lmainL
(lua_State*)0x41fe1378
syntax: lcurL
file luajit21.py
Prints out the lua_State pointer value for current running Lua thread. For example,
(gdb) lcurL
(lua_State*)0x41fe1378
syntax: lg
syntax: lg [L]
file luajit21.py
Prints out the global_State pointer value from the lua_State pointer value specified or from the current main VM state automatically discovered.
Below are some examples:
(gdb) lg
(global_State*)0x41fe13b8
(gdb) lg (lua_State*)0x41fe1378
(global_State*)0x41fe13b8
syntax: lglobtab
syntax: lglobtab L
file luajit21.py
Prints out the global environment table for the specified Lua thread (or the current running Lua thread if the argument is omitted).
For instance,
(gdb) lglobtab
(GCtab*)0x41fe29b0
(gdb) lglobtab (lua_State*)0x41fe1378
(GCtab*)0x41fe29b0
syntax: ltabgets tab field
file luajit21.py
Prints out the value of the specified string field in the Lua table specified by its TValue or GCtab pointer.
(gdb) ltabgets (GCtab*)0x41fe29b0 dog
Key "dog" not found.
(gdb) ltabgets (GCtab*)0x41fe29b0 assert
(TValue*)0x41fe2a20
function assert: (GCfunc*)0x41fe3d38
(gdb) ltabgets (TValue*)0x41f1f450 dog
(TValue*)0x41f1f6d8
number 21.5
syntax: lpc pc
file luajit21.py
Prints out the Lua prototype (GCproto object) whose bytecode contains the PC value specified as the BCIns pointer value. The Lua source line's location (file name and line number) will also be printed out.
For example,
(gdb) lpc 0x419eeb4c
proto: (GCproto*)0x419ee930
source line: @.../lua/waf-core.lua:1330
proto first line: 1282
syntax: lproto file lineno
file luajit21.py
Prints out all the Lua prototype objects (in the form of GCproto pointer values) filtered by the Lua file name and file line number where the corresponding Lua function is defined.
The file name can be specified as the last part of its path.
Below is an example,
(gdb) lproto regex.lua 273
Found Lua proto (GCproto*)0x41221740 at @.../lua-resty-core/lib/resty/core/regex.lua:273
This command works by walking through all the GC objects in the LuaJIT VM.
syntax: lfunc file lineno
file luajit21.py
Similar to the lproto command, but return all the Lua function objects (in GCfunc pointer values) instead of the Lua prototype objects.
Below is an example,
(gdb) lfunc base.lua 137
Found Lua function (GCfunc*)0x41b8efd0 at
@/home/agentzh/git/lua-resty-core/lib/resty/core/base.lua:137
syntax: luv func
file luajit21.py
Prints out names and values for all the upvalues associated with the GCfunc pointer value specified.
Below are some examples:
(gdb) luv (GCfunc*)0x41b8efd0
Found 3 upvalues.
upvalue "str_buf_size": value=(TValue*)0x41b82258 value_type=number closed=1
upvalue "ffi_new": value=(TValue*)0x41b8cc38 value_type=func closed=1
upvalue "str_buf": value=(TValue*)0x41b8cc80 value_type=cdata closed=1
(gdb) luv (GCfunc*)0x4188de10
Found 4 upvalues.
upvalue "C": value=(TValue*)0x41211128 value_type=userdata closed=1
upvalue "ngx_log": value=(TValue*)0x4188de48 value_type=function closed=1
upvalue "ngx_ERR": value=(TValue*)0x4188de68 value_type=number closed=1
upvalue "ffi_gc": value=(TValue*)0x4188de88 value_type=function closed=1
You can get the GCfunc pointer value via the lfunc command.
syntax: lgc
syntax: lgc L
file luajit21.py
Prints out the current size of the total memory that is allocated by the LuaJIT GC.
This is very useful for checking if the LuaJIT VM takes up too much memory on the Lua land.
Below is an example:
(gdb) lgc
The current memory size (allocated by GC): 898960 bytes
syntax: lgcstat
file luajit21.py
This command prints out a statistics summary for all the GC objects (both live ones and dead ones that are not yet collected).
The output is very similar to the systemtap tool, lj-gc-objs.
Below is an example:
(gdb) lgcstat
15172 str objects: max=2956, avg = 51, min=18, sum=779126
987 upval objects: max=24, avg = 24, min=24, sum=23688
104 thread objects: max=1648, avg = 1622, min=528, sum=168784
431 proto objects: max=226274, avg = 2234, min=78, sum=963196
952 func objects: max=144, avg = 30, min=20, sum=28900
446 trace objects: max=23400, avg = 1857, min=160, sum=828604
2965 cdata objects: max=4112, avg = 17, min=12, sum=51576
18961 tab objects: max=24608, avg = 207, min=32, sum=3943256
9 udata objects: max=176095, avg = 39313, min=32, sum=353822
sizeof strhash 65536
sizeof g->tmpbuf 512
sizeof ctype_state 8664
sizeof jit_state 53792
total sz 7274672
g->strnum 15172, g->gc.total 7274672
syntax: lgcpath size [type]
file luajit21.py
Finds large live LuaJIT GC objects with the size threshold (in bytes) and a type name ("tab", "str", "tab", "thr", "upval", "func", "tr"). The type name argument is optional. Also prints out the full referencing path from the GC roots to the object being matched.
For example, finds all the live Lua tables whose size has exceeded 100KB:
(gdb) lgcpath 100000 tab
path 000:[registry] ->Tab["_LOADED"] ->Tab["ffi"] ->Tab["gc"] ->cfunc ->env ->Tab sz:196640 (GCobj*)0x40784f58 ->END
path 001:[registry] ->Tab[tv=0x4132e470] ->Tab sz:524328 (GCobj*)0x40783108 ->END
syntax: lthreadpc
file luajit21.py
Prints out the next PC to be executed for a yielded Lua thread.
(gdb) lthreadpc (lua_State*)0x4169ece0
next PC: (BCIns*)0x40c5d8f0
proto: (GCproto*)0x40c5d898
BC pos: 5
source line: @/opt/app/dummy/lua/exit.lua:131
proto first line: 127
You need to enable the debuginfo in your LuaJIT build (and Nginx build if Nginx is involved).
To enable debuginfo in your LuaJIT build, pass the CCDEBUG=-g command-line argument to the make command, as in
make CCDEBUG=-g
Also, you are required to use gdb 7.6+ with python support enabled.
See Prerequisites first.
And then
- check out this project locally.
- add the following lines to your
~/.gdbinit(you must change the/path/topart to the real path):
directory /path/to/nginx-gdb-utils
py import sys
py sys.path.append("/path/to/nginx-gdb-utils")
source luajit20.gdb
source ngx-lua.gdb
source luajit21.py
source ngx-raw-req.py
set python print-stack full-
Guanlan Dai.
-
Yichun Zhang (agentzh) agentzh@gmail.com, CloudFlare Inc.
This module is licensed under the BSD license.
Copyright (C) 2013-2014, by Guanlan Dai.
Copyright (C) 2013-2014, by Yichun "agentzh" Zhang (章亦春) agentzh@gmail.com, CloudFlare Inc.
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
-
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
-
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- Nginx Systemtap Toolkit
- Sample tools in the stap++ project: https://github.com/agentzh/stapxx#samples