bulk loaded data corrupted when restart monetdb

[monetdb-team]

Date: 2013-05-14 10:02:34 +0200
From: Inho Kim <<georgios.kim>>
To: SQL devs <>
Version: 11.15.7 (Feb2013-SP2)
CC: @drstmane

Last updated: 2013-07-03 08:48:01 +0200

Comment 18720

Date: 2013-05-14 10:02:34 +0200
From: Inho Kim <<georgios.kim>>

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31
Build Identifier:

Alter load 65k rows with bulk load (copy), query will succeed and data was correct. When restart mserver5, same query returns below warning messages.

"1 field truncated!\n note: to disable dropping columns and/or truncating fields use \w-1\n write error"

Reproducible: Always

Steps to Reproduce:

[inho@ChakraMonet tmp]$ mclient -u monetdb -d test
password:
Welcome to mclient, the MonetDB/SQL interactive terminal (Feb2013-SP2)
Database: MonetDB v11.15.7 (Feb2013-SP2), 'mapi:monetdb://ChakraMonet:50002/test'
Type \q to quit, ? for a list of available commands
auto commit mode: on
sql></tmp/test.sql
operation successful
655360 affected rows
sql>select * from test limit 20 offset 655350;
+------+--------+-------+
| id | name | seq |
+======+========+=======+
| id0 | Test 0 | 1 |
| id2 | Test 2 | 2 |
| id3 | Test 3 | 3 |
| id4 | Test 4 | 4 |
| id5 | Test 5 | 5 |
| id6 | Test 6 | 6 |
| id7 | Test 7 | 7 |
| id9 | Test 9 | 9 |
| id8 | Test 8 | 99999 |
| id9 | Test 9 | 9 |
+------+--------+-------+
10 tuples (50.898ms)
sql>\q
[inho@ChakraMonet tmp]$ monetdb stop test
stopping database 'test'... done
[inho@ChakraMonet tmp]$ monetdb start test
starting database 'test'... done
[inho@ChakraMonet tmp]$ !mclient
mclient -u monetdb -d test
password:
Welcome to mclient, the MonetDB/SQL interactive terminal (Feb2013-SP2)
Database: MonetDB v11.15.7 (Feb2013-SP2), 'mapi:monetdb://ChakraMonet:50002/test'
Type \q to quit, ? for a list of available commands
auto commit mode: on
sql>select * from test limit 20 offset 655350;
+------+--------+-----------+
| id | name | seq |
+======+========+===========+
| id0 | Test 0 | 1 |
| id2 | Test 2 | 2 |
| id3 | Test 3 | 3 |
| id4 | Test 4 | 4 |
| id5 | Test 5 | 5 |
| id6 | Test 6 | 6 |
| id7 | Test 7 | 7 |
| id9 | Test 9 | 9 |
| id8 | Test 8 | 99999 |
| 녋랁u |10 tuples (42.354ms) !1 field truncated!
note: to disable dropping columns and/or truncating fields use \w-1
write error
sql>

Comment 18721

Date: 2013-05-14 10:04:22 +0200
From: Inho Kim <<georgios.kim>>

Created attachment 194
bulk load data

Attached file: test.sql.gz (application/x-gzip, 74157 bytes)
Description: bulk load data

Comment 18722

Date: 2013-05-14 10:15:31 +0200
From: @drstmane

Is your original output indeed correct, in particular the double "| id9 | Test 9 | 9 |" record (the second occurrence of which seem to be corrupted after the restart)?

Can you share the content of your monetdbd / merovingian log file that is related to stopping and restarting the server after the bulk load?

Comment 18723

Date: 2013-05-14 10:27:25 +0200
From: Inho Kim <<georgios.kim>>

Is your original output indeed correct, in particular the double "| id9 | Test 9 >| 9 |" record (the second occurrence of which seem to be corrupted after the >restart)?

yes..

Can you share the content of your monetdbd / merovingian log file that is related >to stopping and restarting the server after the bulk load?

I already uploaded test.sql.gz file ( for bulk load )

Comment 18724

Date: 2013-05-14 10:28:21 +0200
From: Inho Kim <<georgios.kim>>

Created attachment 195
merovingian.log

Attached file: merovingian.log (text/plain, 5290 bytes)
Description: merovingian.log

Comment 18725

Date: 2013-05-14 12:50:01 +0200
From: @sjoerdmullender

After loading the data and restarting the server, I get an assertion error:
mserver5: monetdb5/modules/atoms/str.c:1357: strLength: Assertion `l <2147483647' failed.
I'll investigate a bit more.

Comment 18726

Date: 2013-05-14 15:11:48 +0200
From: MonetDB Mercurial Repository <>

Changeset 83ada82a21d8 made by Sjoerd Mullender sjoerd@acm.org in the MonetDB repo, refers to this bug.

For complete details, see http//devmonetdborg/hg/MonetDB?cmd=changeset;node=83ada82a21d8

Changeset description:

Off-by-one error: don't extend if file already correct size.
This fixes bug #3282.
Extending a file is done by seeking to the desired size and writing a
byte.  In the case encountered here, the file was already the correct
size, but a byte was written anyway, overwriting the last byte in the
file and thereby corrupting the data.

Comment 18727

Date: 2013-05-14 15:49:09 +0200
From: MonetDB Mercurial Repository <>

Changeset 934a44f61041 made by Sjoerd Mullender sjoerd@acm.org in the MonetDB repo, refers to this bug.

For complete details, see http//devmonetdborg/hg/MonetDB?cmd=changeset;node=934a44f61041

Changeset description:

Test for bug #3282.

Comment 18728

Date: 2013-05-14 15:57:35 +0200
From: @sjoerdmullender

Bug is fixed, test is added.