pgbouncer 1.5.4 segmentation fault

[wwodecki]

Hello,
we're suffering from a segmentation fault every now and than on CentOS6 x86_64 with pgbouncer 1.5.4. The instance is dying with the following message:

pgbouncer[19702]: segfault at 78 ip 0000000000405d81 sp 00007fffa858ea00 error 4 in pgbouncer[400000+26000]
pgbouncer[8044]: segfault at 78 ip 0000000000405d81 sp 00007fff73fbe7b0 error 4 in pgbouncer[400000+26000]
pgbouncer[19139]: segfault at 78 ip 0000000000405d81 sp 00007fffaf478a60 error 4 in pgbouncer[400000+26000]
pgbouncer[1601]: segfault at 78 ip 0000000000405d81 sp 00007fffbd201260 error 4 in pgbouncer[400000+26000]
pgbouncer[14367]: segfault at 78 ip 0000000000405d81 sp 00007ffffe0dc2c0 error 4 in pgbouncer[400000+26000]

We've attached gdb to a core file and traced the issue down to an uninitialized struct:

Program terminated with signal 11, Segmentation fault.
#0  0x0000000000405d81 in check_client_passwd (client=0x116f470, pkt=0x7ffffe0dc380) at src/client.c:41
41      src/client.c: No such file or directory.
    in src/client.c
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6_5.1.x86_64 libevent-1.4.13-4.el6.x86_64 nss-softokn-freebl-3.14.3-10.el6_5.x86_64
(gdb) bt
#0  0x0000000000405d81 in check_client_passwd (client=0x116f470, pkt=0x7ffffe0dc380) at src/client.c:41
#1  handle_client_startup (client=0x116f470, pkt=0x7ffffe0dc380) at src/client.c:272
#2  0x0000000000406573 in client_proto (sbuf=0x116f528, evtype=<value optimized out>, data=<value optimized out>) at src/client.c:415
#3  0x000000000041008e in sbuf_call_proto (sbuf=<value optimized out>, event=<value optimized out>) at src/sbuf.c:327
#4  0x00000000004102bb in sbuf_process_pending (sbuf=0x116f528) at src/sbuf.c:477
#5  0x00000000004104a8 in sbuf_main_loop (sbuf=0x116f528, skip_recv=<value optimized out>) at src/sbuf.c:647
#6  0x0000000000410757 in sbuf_accept (sbuf=0x116f528, sock=<value optimized out>, is_unix=false) at src/sbuf.c:99
#7  0x000000000040b811 in accept_client (sock=10, is_unix=false) at src/objects.c:1017
#8  0x000000000040eafc in pool_accept (sock=7, flags=<value optimized out>, arg=<value optimized out>) at src/pooler.c:313
#9  0x00007f0718096b44 in event_base_loop () from /usr/lib64/libevent-1.4.so.2
#10 0x00000000004098a3 in main_loop_once () at src/main.c:629
#11 0x0000000000409e25 in main (argc=<value optimized out>, argv=<value optimized out>) at src/main.c:782

after looking at client.c:41 we see that the passwd var is valid however the PgUser struct (or PgSocket struct) must be invalid:

(gdb) printf "%s", passwd
md5XXXXXXXXXXXXXXXXXXXXXXXXX
(gdb) printf "%s", user->passwd
Cannot access memory at address 0x78

The application log does not show any conclusive info. Please tell me if you need more information.

Thanks for help,

Wiktor

[PJMODOS]

Hi,
there are few things that would be useful to know:
client->state
is client->auth_user null?
client->wait_for_welcome
is client->pool null, if not then client->pool->user

[wwodecki]

Hey,
Here you go:

(gdb) print client->state
$1 = CL_LOGIN
(gdb) print client->auth_user 
$2 = (PgUser *) 0x0
(gdb) print client->wait_for_welcome
$3 = false
(gdb) print client->pool
$4 = (PgPool *) 0x0
(gdb) print  client->pool->user
Cannot access memory at address 0x28
(gdb) 

[PJMODOS]

Hmm, what PostgreSQL connection driver is your client app using? It looks to me like the driver is sending password without sending the startup packet first.

[wwodecki]

We're using postgresql-9.1-901.jdbc4.jar. Seems quite outdated when looking at it, but no change here for aeons

[markokr]

Fixed. Crashing is bad...

The client behaviour is rather abnormal, I guess there is multi-threading bug in client app and several threads are trying to use same connection or something like that. As it's unlikely to be protocol bug in pgjdbc driver.

[PJMODOS]

Closing as the bug is fixed.