Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opt_pushselect stuck with multi-table UDF #3817

Closed
monetdb-team opened this issue Nov 30, 2020 · 0 comments
Closed

opt_pushselect stuck with multi-table UDF #3817

monetdb-team opened this issue Nov 30, 2020 · 0 comments

Comments

@monetdb-team
Copy link

@monetdb-team monetdb-team commented Nov 30, 2020

Date: 2015-10-07 17:48:33 +0200
From: @swingbit
To: SQL devs <>
Version: 11.19.9 (Oct2014-SP2)
CC: @njnes

Last updated: 2016-01-15 11:38:03 +0100

Comment 21319

Date: 2015-10-07 17:48:33 +0200
From: @swingbit

User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36
Build Identifier:

*** BACKGRUND

I have an UDF that is used to produce a table for each row in input, and concatenate the output.

  • SQL signature:
    create function tokenize(id integer, s string)
    returns table (id integer, token string)
    external name str."UTF8tokenize";

  • MAL signature:
    command batstr.UTF8tokenize(id:bat[:oid,:int],s:bat[:oid,:str]) (:bat[:oid,:int],:bat[:oid,:str])
    address STRbat_utf8_tokenize_id;

Due to this UDF, it is not trivial to make this issue reproducible, but if this description is not enough to identify the problem, then I can share everything.

The UDF is used this way:

sql>CREATE TABLE ss (id INT, s CLOB);
operation successful (0.439ms)
sql>
sql>INSERT INTO ss VALUES (0,'one two');
1 affected row (0.377ms)
sql>INSERT INTO ss VALUES (1,'three four');
1 affected row (0.098ms)
sql>
sql>SELECT * FROM tokenize( (SELECT id, s FROM ss) );
+------+-------+
| id | token |
+======+=======+
| 0 | one |
| 0 | two |
| 1 | three |
| 1 | four |
+------+-------+
4 tuples (1.277ms)

This works.

*** ISSUE

Now, If I add a simple selection on token:

SELECT *
FROM tokenize( (SELECT id, s FROM ss) )
WHERE length(token) > 5;

then the server seems to hang indefinitely.

gdb suggests that it hangs in opt_pushselect. My wild guess is that it gets stuck while trying to push the select down the innermost selection (the input of the tokenize function). I tried with a standard table function, which takes no parameters and returns a table. No problem in that case.

The gdb session:

(gdb) thread apply all bt

Thread 12 (Thread 0x7f6b69ca9700 (LWP 20945)):
0 0x000000394ccf9063 in select () from /lib64/libc.so.6
1 0x00007f6b70ef5863 in MT_sleep_ms (ms=25) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/gdk/gdk_posix.c:1085
2 0x00007f6b714d43fa in profilerHeartbeat (dummy=0x0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_profiler.c:980
3 0x000000394d007555 in start_thread () from /lib64/libpthread.so.0
4 0x000000394cd02b9d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f6b681b1700 (LWP 20946)):
0 0x000000394ccf9063 in select () from /lib64/libc.so.6
1 0x00007f6b715d8918 in SERVERlistenThread (Sock=0x279f5f0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/modules/mal/mal_mapi.c:234
2 0x00007f6b70ef3854 in thread_starter (arg=0x279f680) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/gdk/gdk_system.c:458
3 0x000000394d007555 in start_thread () from /lib64/libpthread.so.0
4 0x000000394cd02b9d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f6b67fb0700 (LWP 20947)):
0 0x000000394ccf9063 in select () from /lib64/libc.so.6
1 0x00007f6b70ef5863 in MT_sleep_ms (ms=50) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/gdk/gdk_posix.c:1085
2 0x00007f6b68fcf64c in store_manager () at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/sql/storage/store.c:1598
3 0x00007f6b68f40d2d in mvc_logmanager () at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/sql/server/sql_mvc.c:133
4 0x00007f6b70ef3854 in thread_starter (arg=0x2980800) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/gdk/gdk_system.c:458
5 0x000000394d007555 in start_thread () from /lib64/libpthread.so.0
6 0x000000394cd02b9d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f6b67daf700 (LWP 20950)):
0 0x00007f6b716d1149 in OPTpushselectImplementation (cntxt=0x7f6b69cb0328, mb=0x7f6b581260a0, stk=0x0, pci=0x0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/optimizer/opt_pushselect.c:220
1 0x00007f6b716d3808 in OPTwrapper (cntxt=0x7f6b69cb0328, mb=0x7f6b581260a0, stk=0x0, p=0x7f6b5815c7a0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/optimizer/opt_wrapper.c:149
2 0x00007f6b716ce680 in optimizeMALBlock (cntxt=0x7f6b69cb0328, mb=0x7f6b581260a0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/optimizer/opt_support.c:262
3 0x00007f6b68ec4289 in addQueryToCache (c=0x7f6b69cb0328) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/sql/backends/monet5/sql_optimizer.c:294
4 0x00007f6b68ec1e18 in backend_dumpproc (be=0x7f6b580eafa0, c=0x7f6b69cb0328, cq=0x7f6b5811bb30, s=0x7f6b58107960) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/sql/backends/monet5/sql_gencode.c:2804
5 0x00007f6b68e8e1ae in SQLparser (c=0x7f6b69cb0328) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/sql/backends/monet5/sql_scenario.c:1206
6 0x00007f6b714e0bb0 in runPhase (c=0x7f6b69cb0328, phase=1) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_scenario.c:515
7 0x00007f6b714e0d57 in runScenarioBody (c=0x7f6b69cb0328) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_scenario.c:550
8 0x00007f6b714e0f02 in runScenario (c=0x7f6b69cb0328) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_scenario.c:579
9 0x00007f6b714e283e in MSserveClient (dummy=0x7f6b69cb0328) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_session.c:439
10 0x00007f6b714e2237 in MSscheduleClient (command=0x7f6b580008d0 "\001", challenge=0x7f6b67daedf0 "00lSGLgI", fin=0x7f6b58006b40, fout=0x7f6b580049c0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_session.c:319
11 0x00007f6b715d86e0 in doChallenge (data=0x7f6b600008d0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/modules/mal/mal_mapi.c:184
12 0x00007f6b70ef3854 in thread_starter (arg=0x7f6b60000a50) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/gdk/gdk_system.c:458
13 0x000000394d007555 in start_thread () from /lib64/libpthread.so.0
14 0x000000394cd02b9d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f6b67aae700 (LWP 20951)):
0 0x000000394d00e829 in do_futex_wait.constprop () from /lib64/libpthread.so.0
1 0x000000394d00e8c4 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
2 0x000000394d00e96a in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
3 0x00007f6b714b9c30 in q_dequeue (q=0x7f6b5811bbc0, cntxt=0x0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_dataflow.c:213
4 0x00007f6b714ba64b in DFLOWworker (T=0x7f6b71934880 ) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_dataflow.c:333
5 0x000000394d007555 in start_thread () from /lib64/libpthread.so.0
6 0x000000394cd02b9d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f6b678ad700 (LWP 20952)):
---Type to continue, or q to quit---
0 0x000000394d00e829 in do_futex_wait.constprop () from /lib64/libpthread.so.0
1 0x000000394d00e8c4 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
2 0x000000394d00e96a in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
3 0x00007f6b714b9c30 in q_dequeue (q=0x7f6b5811bbc0, cntxt=0x0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_dataflow.c:213
4 0x00007f6b714ba64b in DFLOWworker (T=0x7f6b719348c0 <workers+64>) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_dataflow.c:333
5 0x000000394d007555 in start_thread () from /lib64/libpthread.so.0
6 0x000000394cd02b9d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f6b676ac700 (LWP 20953)):
0 0x000000394d00e829 in do_futex_wait.constprop () from /lib64/libpthread.so.0
1 0x000000394d00e8c4 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
2 0x000000394d00e96a in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
3 0x00007f6b714b9c30 in q_dequeue (q=0x7f6b5811bbc0, cntxt=0x0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_dataflow.c:213
4 0x00007f6b714ba64b in DFLOWworker (T=0x7f6b71934900 <workers+128>) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_dataflow.c:333
5 0x000000394d007555 in start_thread () from /lib64/libpthread.so.0
6 0x000000394cd02b9d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f6b674ab700 (LWP 20954)):
0 0x000000394d00e829 in do_futex_wait.constprop () from /lib64/libpthread.so.0
1 0x000000394d00e8c4 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
2 0x000000394d00e96a in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
3 0x00007f6b714b9c30 in q_dequeue (q=0x7f6b5811bbc0, cntxt=0x0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_dataflow.c:213
4 0x00007f6b714ba64b in DFLOWworker (T=0x7f6b71934940 <workers+192>) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_dataflow.c:333
5 0x000000394d007555 in start_thread () from /lib64/libpthread.so.0
6 0x000000394cd02b9d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f6b672aa700 (LWP 20955)):
0 0x000000394d00e829 in do_futex_wait.constprop () from /lib64/libpthread.so.0
1 0x000000394d00e8c4 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
2 0x000000394d00e96a in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
3 0x00007f6b714b9c30 in q_dequeue (q=0x7f6b5811bbc0, cntxt=0x0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_dataflow.c:213
4 0x00007f6b714ba64b in DFLOWworker (T=0x7f6b71934980 <workers+256>) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_dataflow.c:333
5 0x000000394d007555 in start_thread () from /lib64/libpthread.so.0
6 0x000000394cd02b9d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f6b670a9700 (LWP 20956)):
0 0x000000394d00e829 in do_futex_wait.constprop () from /lib64/libpthread.so.0
1 0x000000394d00e8c4 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
2 0x000000394d00e96a in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
3 0x00007f6b714b9c30 in q_dequeue (q=0x7f6b5811bbc0, cntxt=0x0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_dataflow.c:213
4 0x00007f6b714ba64b in DFLOWworker (T=0x7f6b719349c0 <workers+320>) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_dataflow.c:333
5 0x000000394d007555 in start_thread () from /lib64/libpthread.so.0
6 0x000000394cd02b9d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f6b66ca7700 (LWP 20958)):
0 0x000000394d00e829 in do_futex_wait.constprop () from /lib64/libpthread.so.0
1 0x000000394d00e8c4 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
2 0x000000394d00e96a in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
3 0x00007f6b714b9c30 in q_dequeue (q=0x7f6b5811bbc0, cntxt=0x0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_dataflow.c:213
4 0x00007f6b714ba64b in DFLOWworker (T=0x7f6b71934a40 <workers+448>) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/mal/mal_dataflow.c:333
5 0x000000394d007555 in start_thread () from /lib64/libpthread.so.0
---Type to continue, or q to quit---
6 0x000000394cd02b9d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f6b70678800 (LWP 20944)):
0 0x000000394ccf9063 in select () from /lib64/libc.so.6
1 0x00007f6b70ef5863 in MT_sleep_ms (ms=5000) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/gdk/gdk_posix.c:1085
2 0x0000000000403ec4 in main (argc=20, av=0x7ffcbaed5dc8) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/tools/mserver/mserver5.c:668
(gdb) thread 9
[Switching to thread 9 (Thread 0x7f6b67daf700 (LWP 20950))]
0 0x00007f6b716d1149 in OPTpushselectImplementation (cntxt=0x7f6b69cb0328, mb=0x7f6b581260a0, stk=0x0, pci=0x0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/monetdb5/optimizer/opt_pushselect.c:220
220 } else if (isMapOp(q) && q->argc >= 2 && isaBatType(getArgType(mb, q, 1))) {
(gdb) l
215 InstrPtr s = old[vars[i1]];
216 if (getModuleId(s) == sqlRef && getFunctionId(s) == tidRef)
217 tid = getArg(q, 1);
218 }
219 break;
220 } else if (isMapOp(q) && q->argc >= 2 && isaBatType(getArgType(mb, q, 1))) {
221 int i1 = getArg(q, 1);
222 q = old[vars[i1]];
223 } else if (isMapOp(q) && q->argc >= 3 && isaBatType(getArgType(mb, q, 2))) {
224 int i2 = getArg(q, 2);

Reproducible: Always

$ mserver5 --version
MonetDB 5 server v11.21.6 (64-bit, 64-bit oids, 128-bit integers)
This is an unreleased version
Copyright (c) 1993-July 2008 CWI
Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved
Visit http://www.monetdb.org/ for further information
Found 15.6GiB available memory, 8 available cpu cores
Libraries:
libpcre: 8.37 2015-04-28 (compiled with 8.37)
openssl: OpenSSL 1.0.1k 8 Jan 2015 (compiled with OpenSSL 1.0.1k-fips 8 Jan 2015)
libxml2: 2.9.2 (compiled with 2.9.2)
Compiled by: roberto@photon.hq.spinque.com (x86_64-unknown-linux-gnu)
Compilation: gcc -g -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wformat=2 -Wno-format-nonliteral -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wpacked-bitfield-compat -Wsync-nand -Wjump-misses-init -Wmissing-include-dirs -Wlogical-op -Wunreachable-code
Linking : /usr/bin/ld -m elf_x86_64

Comment 21691

Date: 2015-12-24 22:36:44 +0100
From: MonetDB Mercurial Repository <>

Changeset 9633f26d33e6 made by Niels Nes niels@cwi.nl in the MonetDB repo, refers to this bug.

For complete details, see http//devmonetdborg/hg/MonetDB?cmd=changeset;node=9633f26d33e6

Changeset description:

protect against multi-output multiplex functions, solves bug #3817

Comment 21692

Date: 2015-12-24 22:37:08 +0100
From: @njnes

fixed in Jul 2015 branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant