Server crashes on bulk load

[monetdb-team]

Date: 2015-12-07 16:30:59 +0100
From: Daniel Sali <<daniel.sali>>
To: SQL devs <>
Version: 11.21.11 (Jul2015-SP1)
CC: @njnes

Last updated: 2016-03-25 09:59:13 +0100

Comment 21642

Date: 2015-12-07 16:30:59 +0100
From: Daniel Sali <<daniel.sali>>

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36
Build Identifier:

We are trying to load a large amount of data using Pentaho Kettle a Java ETL suite, which creates a COPY INTO table FROM fifo file statement and uses the driver. The Java process makes a COPY INTO call every 100,000 records, but after about 32 million records loaded, the server process dies with a segmentation fault.

Reproducible: Always

Steps to Reproduce:

1. Load 100000 records 33-34 times with COPY INTO table

Actual Results:

The server process died with a segmentation fault.

Expected Results:

Records should load successfully.

We have attached gdb to the mserver process after repeatedly experiencing the issue, and the following stack trace was captured:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f7f54bef700 (LWP 21904)]
0x00007f7f5a9460b2 in ebat_copy () from /usr/lib64/monetdb5/lib_sql.so
(gdb) where
0 0x00007f7f5a9460b2 in ebat_copy () from /usr/lib64/monetdb5/lib_sql.so
1 0x00007f7f5a9437bc in dup_bat () from /usr/lib64/monetdb5/lib_sql.so
2 0x00007f7f5a9447eb in append_col () from /usr/lib64/monetdb5/lib_sql.so
3 0x00007f7f5a833ecf in mvc_append_wrap () from /usr/lib64/monetdb5/lib_sql.so
4 0x00007f7f679c41c8 in runMALsequence () from /lib64/libmonetdb5.so.19
5 0x00007f7f679c514a in callMAL () from /lib64/libmonetdb5.so.19
6 0x00007f7f5a845acf in SQLexecutePrepared () from /usr/lib64/monetdb5/lib_sql.so
7 0x00007f7f5a845fca in SQLengineIntern () from /usr/lib64/monetdb5/lib_sql.so
8 0x00007f7f679df1c0 in runScenarioBody () from /lib64/libmonetdb5.so.19
9 0x00007f7f679dfcad in runScenario () from /lib64/libmonetdb5.so.19
10 0x00007f7f679e00f0 in MSserveClient () from /lib64/libmonetdb5.so.19
11 0x00007f7f679e06fb in MSscheduleClient () from /lib64/libmonetdb5.so.19
12 0x00007f7f67a8564c in doChallenge () from /lib64/libmonetdb5.so.19
13 0x00007f7f674b5d1f in thread_starter () from /lib64/libbat.so.12
14 0x00007f7f64ff7df5 in start_thread (arg=0x7f7f54bef700) at pthread_create.c:308
15 0x00007f7f64d1e1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

The output of the 'list' gdb command:
(gdb) list
76 else
77
78 /* This is a "normal" system call stub: if there is an error,
79 it returns -1 and sets errno. */
80
81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
82 ret
83 T_PSEUDO_END (SYSCALL_SYMBOL)
84
85 endif

Thread dump will be added as an attachment.

Thanks,
Daniel Sali

Comment 21643

Date: 2015-12-07 16:32:21 +0100
From: Daniel Sali <<daniel.sali>>

Created attachment 376
Output of the 'thr app all bt' gdb command

Attached file: monetdb_thread_dump.txt (text/plain, 8686 bytes)
Description: Output of the 'thr app all bt' gdb command

Comment 21649

Date: 2015-12-08 11:25:29 +0100
From: Daniel Sali <<daniel.sali>>

Please note that though the symptoms are similar to Bug #3848, the stack trace is different.

Comment 21714

Date: 2016-01-06 13:30:21 +0100
From: @njnes

I'm tried to reproduce this problem (and failed). Could you give some more details about the data types your loading. A schema /create table statement could help a lot. On what system are you running, do you have enough diskspace available for the db?

Comment 21894

Date: 2016-03-14 08:30:36 +0100
From: @njnes

the ebat_copy code has now more protection against out of memory situations.

Comment 21957

Date: 2016-03-25 09:59:13 +0100
From: @sjoerdmullender

Jul2015-SP3 has been released.