Segmentation fault in mserver5 on Python2 UDF with TIMESTAMP column input that has NULL values (conversion.c:438, PyNullMask_FromBAT)

[monetdb-team]

Date: 2018-01-08 00:11:21 +0100
From: Niklas B <<niklas.bivald>>
To: SQL devs <>
Version: 11.27.11 (Jul2017-SP3)
CC: @njnes, niklas.bivald

Last updated: 2018-02-12 16:12:21 +0100

Comment 26058

Date: 2018-01-08 00:11:21 +0100
From: Niklas B <<niklas.bivald>>

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36
Build Identifier:

I haven't found this bug in the archive, so I'm reporting it. Hopefully it's not a dupe :)

Using a Python UDF which takes a TIMESTAMP column as input crashes mserver5 on Debian Stretch if the table column has NULL values in it.

Running (linux~debian stretch) on commit a7f8b1f865b7 (https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a7f8b1f865b7)

Thread 6 "mserver5" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe97c1700 (LWP 22566)]
0x00007ffff1c53d5c in PyNullMask_FromBAT (b=0x7fffd817d200, t_start=0, t_end=2) at conversion.c:438
438 CreateNullMask(lng);
(gdb) l
433 break;
434 case TYPE_int:
435 CreateNullMask(int);
436 break;
437 case TYPE_lng:
438 CreateNullMask(lng);
439 break;
440 case TYPE_flt:
441 CreateNullMask(flt);
442 break;

Only tried on Debian Stretch build from source at commit a7f8b1f865b7, with and without debugging enabled. GDB produced the output above.

Reproducible: Always

Steps to Reproduce:

drop all function test_timestamp_with_nulls;
CREATE FUNCTION test_timestamp_with_nulls(actual_takeoff_datetime timestamp) RETURNS STRING LANGUAGE PYTHON {
return "it works";
};

drop table if exists example;

CREATE TABLE example (
"column1" timestamp
);

insert into example ("column1") values ('2017-01-01 00:00:01');
insert into example ("column1") values (NULL);

select test_timestamp_with_nulls("column1") from example;

Actual Results:

Segmentation fault

Expected Results:

Query should return data

If I cast the column to a STRING it works:

select test_timestamp_with_nulls(cast("column1" as string)) from example;

Comment 26059

Date: 2018-01-08 00:17:28 +0100
From: Niklas B <<niklas.bivald>>

I realise I perhaps wasn't as clear in my communication as I hoped, I apologise, it's after midnight. To summarise:

If I have a Python UDF that takes timestamp input, like so:

CREATE FUNCTION test_timestamp_with_nulls(actual_takeoff_datetime timestamp) RETURNS STRING LANGUAGE PYTHON {
return "it works";
};

And I pass a column to it, like so:

select test_timestamp_with_nulls("column1") from example;

Then if "column1" in table "example" is a timestamp and any value in the column is NULL, mserver5 crashes. If there are no NULl values, it works.

I've only tested this on Linux Debian Stretch, but I can reproduce it every time. I haven't made any changes to the source.

Comment 26061

Date: 2018-01-08 10:16:12 +0100
From: MonetDB Mercurial Repository <>

Changeset 7d2a93f65310 made by Sjoerd Mullender sjoerd@acm.org in the MonetDB repo, refers to this bug.

For complete details, see https//devmonetdborg/hg/MonetDB?cmd=changeset;node=7d2a93f65310

Changeset description:

Use correct bat.
This fixes bug #6508.

Comment 26062

Date: 2018-01-08 10:16:17 +0100
From: MonetDB Mercurial Repository <>

Changeset cc06e0c02cb1 made by Sjoerd Mullender sjoerd@acm.org in the MonetDB repo, refers to this bug.

For complete details, see https//devmonetdborg/hg/MonetDB?cmd=changeset;node=cc06e0c02cb1

Changeset description:

Added test for bug #6508.

Comment 26063

Date: 2018-01-08 11:40:58 +0100
From: Niklas B <<niklas.bivald>>

Works like a charm, thank you so much :)

Comment 26064

Date: 2018-01-08 22:19:06 +0100
From: Niklas B <<niklas.bivald>>

Sorry for the re-open, but it would appears as if even though it doesn't crash, the NULL:s are actually converted into string "nil":

drop all function test_timestamp_with_nulls;

CREATE FUNCTION test_timestamp_with_nulls(actual_takeoff_datetime timestamp) RETURNS STRING LANGUAGE PYTHON {
import json
return json.dumps(actual_takeoff_datetime.tolist());
};

drop table if exists example;
CREATE TABLE example (
"column1" timestamp
);

insert into example ("column1") values ('2017-01-01 00:00:01');
insert into example ("column1") values (NULL);

select test_timestamp_with_nulls("column1") from example;

Gives:

+------------------------------------+
| L2 |
+====================================+
| ["2017-01-01 00:00:01.000", "nil"] |
+------------------------------------+

But should give:

+------------------------------------+
| L2 |
+====================================+
| ["2017-01-01 00:00:01.000", null] |
+------------------------------------+

I.e the null value shouldn't be converted to string "nil", (but probably a numpy or python NaN)

Running commit cc06e0c02cb1

Or is NIL actually expected behaviour? If so feel free to close.

Comment 26067

Date: 2018-01-10 16:43:46 +0100
From: MonetDB Mercurial Repository <>

Changeset 0dd61564584d made by Sjoerd Mullender sjoerd@acm.org in the MonetDB repo, refers to this bug.

For complete details, see https//devmonetdborg/hg/MonetDB?cmd=changeset;node=0dd61564584d

Changeset description:

Don't replace bat; updated test.
This fixes the reopened bug #6508.