Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HEAPextend: failed to extend to 2420077101056 #3791

Closed
monetdb-team opened this issue Nov 30, 2020 · 0 comments
Closed

HEAPextend: failed to extend to 2420077101056 #3791

monetdb-team opened this issue Nov 30, 2020 · 0 comments

Comments

@monetdb-team
Copy link

@monetdb-team monetdb-team commented Nov 30, 2020

Date: 2015-08-14 14:57:12 +0200
From: @swingbit
To: GDK devs <>
Version: 11.21.1 (Jul2015)
CC: @drstmane

Last updated: 2015-08-28 13:41:28 +0200

Comment 21148

Date: 2015-08-14 14:57:12 +0200
From: @swingbit

User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.155 Safari/537.36
Build Identifier:

This join fails trying to allocate 2.2TB

sql>START TRANSACTION;
auto commit mode: off
sql>CREATE TABLE all_termDict_unstemmed_tmp AS
more>SELECT termID, term, prob
more>FROM termDict
more>WHERE termID IN (SELECT termID FROM all_tc_unstemmed)
more>WITH DATA;
Object not found
GDKsave: failed name=25/2503, ext=tail, mode 0
GDKsave: failed name=25/2504, ext=tail, mode 0
GDKsave: failed name=25/2520, ext=tail, mode 0
HEAPextend: failed to extend to 2420077101056 for 03/322.tail: GDKmremap() failed

More info:

CREATE TABLE "spinque"."termdict" (
"termid" INTEGER NOT NULL,
"term" CHARACTER LARGE OBJECT NOT NULL,
"prob" DOUBLE NOT NULL DEFAULT 1.000000,
CONSTRAINT "termdict_termid_pkey" PRIMARY KEY ("termid"),
CONSTRAINT "termdict_term_unique" UNIQUE ("term")
);
sql>\d all_tc_unstemmed
CREATE TABLE "spinque"."all_tc_unstemmed" (
"termid" INTEGER NOT NULL,
"objid" INTEGER NOT NULL,
"prob" DOUBLE NOT NULL,
CONSTRAINT "all_tc_unstemmed_objid_fkey" FOREIGN KEY ("objid") REFERENCES "spinque"."dict" ("id"),
CONSTRAINT "all_tc_unstemmed_termid_fkey" FOREIGN KEY ("termid") REFERENCES "spinque"."all_termdict_unstemmed" ("termid")
);

sql>select count(*) from termDict;
+--------+
| L1 |
+========+
| 679338 |
+--------+

sql> select count(*) from all_tc_unstemmed;
auto commit mode: on
+----------+
| L1 |
+==========+
| 51956929 |
+----------+

Reproducible: Always

$ ./mserver5 --version
MonetDB 5 server v11.21.2 (64-bit, 64-bit oids, 128-bit integers)
This is an unreleased version
Copyright (c) 1993-July 2008 CWI
Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved
Visit http://www.monetdb.org/ for further information
Found 31.4GiB available memory, 8 available cpu cores
Libraries:
libpcre: 8.37 2015-04-28 (compiled with 8.37)
openssl: OpenSSL 1.0.1k 8 Jan 2015 (compiled with OpenSSL 1.0.1k-fips 8 Jan 2015)
libxml2: 2.9.2 (compiled with 2.9.2)
Compiled by: roberto@spinque02.spinque.com (x86_64-unknown-linux-gnu)
Compilation: gcc -O3 -pipe -Werror -Wall -Wextra -W -Werror-implicit-function-declaration -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wformat=2 -Wno-format-nonliteral -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-format-attribute -Wmissing-prototypes -Wold-style-definition -Wpacked -Wunknown-pragmas -Wvariadic-macros -fstack-protector-all -Wstack-protector -Wpacked-bitfield-compat -Wsync-nand -Wjump-misses-init -Wmissing-include-dirs -Wlogical-op -Wunreachable-code -D_FORTIFY_SOURCE=2
Linking : /usr/bin/ld -m elf_x86_64

Comment 21149

Date: 2015-08-14 15:29:28 +0200
From: @drstmane

What about the messages before the failing HEAPextend:

Object not found
GDKsave: failed name=25/2503, ext=tail, mode 0
GDKsave: failed name=25/2504, ext=tail, mode 0
GDKsave: failed name=25/2520, ext=tail, mode 0

?

Does one of the following work?

CREATE TABLE all_termDict_unstemmed_tmp AS
SELECT termID, term, prob
FROM termDict
WHERE termID IN (SELECT distinct termID FROM all_tc_unstemmed)
WITH DATA;

CREATE TABLE all_termDict_unstemmed_tmp AS
SELECT termDict.termID, termDict.term, termDict.prob
FROM termDict
JOIN (SELECT distinct termID FROM all_tc_unstemmed) as atu
ON termDict.termID = atu.termID
WITH DATA;

Could you provide an explain of your original query?

Comment 21150

Date: 2015-08-14 15:38:59 +0200
From: @swingbit

There is no message before those errors. I guess the repeated errors come from different threads.

Both the queries with distinct work, good catch!

Explain of the original query:

sql>explain CREATE TABLE all_termDict_unstemmed_tmp AS
more>SELECT termID, term, prob
more>FROM termDict
more>WHERE termID IN (SELECT termID FROM all_tc_unstemmed)
more>WITH DATA;
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mal |
+========================================================================================================================================================================================================================+
| function user.s2_33{autoCommit=true}():void; |
| X_52:void := querylog.define("explain create table all_termdict_unstemmed_tmp as\nselect termid, term, prob\nfrom termdict\nwhere termid in (select termid from all_tc_unstemmed)\nwith data;","default_pipe",75); |
| X_2 := sql.mvc(); |
| sql.catalog(24,"spinque",nil,0); |
| barrier X_130 := language.dataflow(); |
| X_8:bat[:oid,:oid] := sql.tid(X_2,"spinque","termdict"); |
| X_10:bat[:oid,:int] := sql.bind(X_2,"spinque","termdict","termid",0); |
| (X_12,r1_15) := sql.bind(X_2,"spinque","termdict","termid",2); |
| X_15:bat[:oid,:int] := sql.bind(X_2,"spinque","termdict","termid",1); |
| X_17 := sql.delta(X_10,X_12,r1_15,X_15); |
| X_18 := algebra.leftfetchjoin(X_8,X_17); |
| X_19 := bat.mirror(X_18); |
| X_73:bat[:oid,:oid] := sql.tid(X_2,"spinque","all_tc_unstemmed",0,8); |
| X_88:bat[:oid,:int] := sql.bind(X_2,"spinque","all_tc_unstemmed","termid",0,0,8); |
| X_96 := algebra.leftfetchjoin(X_73,X_88); |
| (X_104,X_105) := algebra.subjoin(X_18,X_96,nil:BAT,nil:BAT,false,nil:lng); |
| X_75:bat[:oid,:oid] := sql.tid(X_2,"spinque","all_tc_unstemmed",1,8); |
| X_89:bat[:oid,:int] := sql.bind(X_2,"spinque","all_tc_unstemmed","termid",0,1,8); |
| X_97 := algebra.leftfetchjoin(X_75,X_89); |
| (X_106,X_107) := algebra.subjoin(X_18,X_97,nil:BAT,nil:BAT,false,nil:lng); |
| X_77:bat[:oid,:oid] := sql.tid(X_2,"spinque","all_tc_unstemmed",2,8); |
| X_90:bat[:oid,:int] := sql.bind(X_2,"spinque","all_tc_unstemmed","termid",0,2,8); |
| X_98 := algebra.leftfetchjoin(X_77,X_90); |
| (X_108,X_109) := algebra.subjoin(X_18,X_98,nil:BAT,nil:BAT,false,nil:lng); |
| X_79:bat[:oid,:oid] := sql.tid(X_2,"spinque","all_tc_unstemmed",3,8); |
| X_91:bat[:oid,:int] := sql.bind(X_2,"spinque","all_tc_unstemmed","termid",0,3,8); |
| X_99 := algebra.leftfetchjoin(X_79,X_91); |
| (X_110,X_111) := algebra.subjoin(X_18,X_99,nil:BAT,nil:BAT,false,nil:lng); |
| X_81:bat[:oid,:oid] := sql.tid(X_2,"spinque","all_tc_unstemmed",4,8); |
| X_92:bat[:oid,:int] := sql.bind(X_2,"spinque","all_tc_unstemmed","termid",0,4,8); |
| X_100 := algebra.leftfetchjoin(X_81,X_92); |
| (X_112,X_113) := algebra.subjoin(X_18,X_100,nil:BAT,nil:BAT,false,nil:lng); |
| X_83:bat[:oid,:oid] := sql.tid(X_2,"spinque","all_tc_unstemmed",5,8); |
| X_93:bat[:oid,:int] := sql.bind(X_2,"spinque","all_tc_unstemmed","termid",0,5,8); |
| X_101 := algebra.leftfetchjoin(X_83,X_93); |
| (X_114,X_115) := algebra.subjoin(X_18,X_101,nil:BAT,nil:BAT,false,nil:lng); |
| X_85:bat[:oid,:oid] := sql.tid(X_2,"spinque","all_tc_unstemmed",6,8); |
| X_94:bat[:oid,:int] := sql.bind(X_2,"spinque","all_tc_unstemmed","termid",0,6,8); |
| X_102 := algebra.leftfetchjoin(X_85,X_94); |
| (X_116,X_117) := algebra.subjoin(X_18,X_102,nil:BAT,nil:BAT,false,nil:lng); |
| X_87:bat[:oid,:oid] := sql.tid(X_2,"spinque","all_tc_unstemmed",7,8); |
| X_95:bat[:oid,:int] := sql.bind(X_2,"spinque","all_tc_unstemmed","termid",0,7,8); |
| X_103 := algebra.leftfetchjoin(X_87,X_95); |
| (X_118,X_119) := algebra.subjoin(X_18,X_103,nil:BAT,nil:BAT,false,nil:lng); |
| X_121 := mat.packIncrement(X_104,8); |
| X_122 := mat.packIncrement(X_121,X_106); |
| X_123 := mat.packIncrement(X_122,X_108); |
| X_124 := mat.packIncrement(X_123,X_110); |
| X_125 := mat.packIncrement(X_124,X_112); |
| X_126 := mat.packIncrement(X_125,X_114); |
| X_127 := mat.packIncrement(X_126,X_116); |
| X_21 := mat.packIncrement(X_127,X_118); |
| X_25 := algebra.tinter(X_19,X_21); |
| X_26 := algebra.leftfetchjoin(X_25,X_18); |
| language.pass(X_18); |
| exit X_130; |
| X_27 := sql.append(X_2,"spinque","all_termdict_unstemmed_tmp","termid",X_26); |
| barrier X_133 := language.dataflow(); |
| X_29:bat[:oid,:str] := sql.bind(X_27,"spinque","termdict","term",0); |
| (X_32,r1_40) := sql.bind(X_27,"spinque","termdict","term",2); |
| X_34:bat[:oid,:str] := sql.bind(X_27,"spinque","termdict","term",1); |
| X_35 := sql.delta(X_29,X_32,r1_40,X_34); |
| X_36:bat[:oid,:str] := algebra.leftfetchjoinPath(X_25,X_8,X_35); |
| exit X_133; |
| X_37 := sql.append(X_27,"spinque","all_termdict_unstemmed_tmp","term",X_36); |
| barrier X_136 := language.dataflow(); |
| X_38:bat[:oid,:dbl] := sql.bind(X_37,"spinque","termdict","prob",0); |
| (X_42,r1_52) := sql.bind(X_37,"spinque","termdict","prob",2); |
| X_45:bat[:oid,:dbl] := sql.bind(X_37,"spinque","termdict","prob",1); |
| X_47 := sql.delta(X_38,X_42,r1_52,X_45); |
| X_48:bat[:oid,:dbl] := algebra.leftfetchjoinPath(X_25,X_8,X_47); |
| exit X_136; |
| sql.append(X_37,"spinque","all_termdict_unstemmed_tmp","prob",X_48); |
| sql.exportOperation(""); |
| end user.s2_33; |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
75 tuples (6.635ms)

Debug of the original query:

sql>debug CREATE TABLE all_termDict_unstemmed_tmp AS
more>SELECT termID, term, prob
more>FROM termDict
more>WHERE termID IN (SELECT termID FROM all_tc_unstemmed)
more>WITH DATA;
mdb>mdb.start();
mdb>
mdb>X_2=0@0:void := user.s3_33();
mdb>
mdb>X_51=0@0:void := querylog.define("debug create table all_termdict_unstemmed_tmp as\nselect termid, term, prob\nfrom termdict\nwhere termid in (select termid from all_tc_unstemmed)\nwith data;","default_pipe",75);
mdb>
mdb>X_1=0 := sql.mvc();
mdb>
mdb>sql.catalog(24,"spinque",nil,0);
mdb>
mdb>barrier X_153=false := language.dataflow();
mdb>
mdb>X_7=nil:bat[:oid,:oid] := sql.tid(X_1=0,"spinque","termdict");
mdb>
mdb>X_9=nil:bat[:oid,:int] := sql.bind(X_1=0,"spinque","termdict","termid",0);
mdb>
mdb>(X_11=nil:bat[:oid,:oid],r1_14=nil:bat[:oid,:int]) := sql.bind(X_1=0,"spinque","termdict","termid",2);
mdb>
mdb>X_14=nil:bat[:oid,:int] := sql.bind(X_1=0,"spinque","termdict","termid",1);
mdb>
mdb>X_16=nil:bat[:oid,:int] := sql.delta(X_9=<tmp_1604>[679338],X_11=<tmp_2540>[0],r1_14=<tmp_2334>[0],X_14=<tmp_2334>[0]);
mdb>
mdb>X_17=nil:bat[:oid,:int] := algebra.leftfetchjoin(X_7=<tmp_14260>[679338],X_16=<tmp_1604>[679338]);
mdb>
mdb>X_18=nil:bat[:oid,:oid] := bat.mirror(X_17=<tmp_12726>[679338]);
mdb>
mdb>X_80=nil:bat[:oid,:oid] := sql.tid(X_1=0,"spinque","all_tc_unstemmed",0,8);
mdb>
mdb>X_95=nil:bat[:oid,:int] := sql.bind(X_1=0,"spinque","all_tc_unstemmed","termid",0,0,8);
mdb>
mdb>X_104=nil:bat[:oid,:int] := algebra.leftfetchjoin(X_80=<tmp_14226>[6494616],X_95=<tmp_14334>[6494616]);
mdb>
mdb>(X_113=nil:bat[:oid,:oid],X_114=nil:bat[:oid,:oid]) := algebra.subjoin(X_17=<tmp_12726>[679338],X_104=<tmp_10232>[6494616],nil:BAT,nil:BAT,false,nil:lng);
mdb>
ERROR: MALException:algebra.subjoin:Object not found
ERROR: HEAPextend: failed to extend to 2420077101056 for 01/43/14334.tail: GDKmremap() failed
mdb>

Comment 21151

Date: 2015-08-14 17:06:25 +0200
From: @sjoerdmullender

Can you run the query with a C debugger and find out exactly which call to BATextend (which calls HEAPextend) it is that is failing? And moreover, can you also report the values of all relevant variables around that call?

Comment 21152

Date: 2015-08-17 11:29:49 +0200
From: @swingbit

Here are some more details.
It seems to be a BATextend on the right side of a mergejoin.

Note that the heap reallocation mentioned here is lower than the one reported initially (but it still fails). I guess it can be because 1) I am running the debug session on a different machine. 2) in the debug session a different thread with different chunk sizes could come first?

Breakpoint 1, HEAPextend (h=0x7ff0380039a8, size=10696196096, mayshare=0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/gdk/gdk_heap.c:204
204 failure = "GDKmremap() failed";

0 HEAPextend (h=0x7ff0380039a8, size=10696196096, mayshare=0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/gdk/gdk_heap.c:204
1 0x00007ff054617d7f in BATextend (b=0x7ff038003880, newcap=1337017706) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/gdk/gdk_bat.c:472
2 0x00007ff05480e3a6 in mergejoin (r1=0x7ff0380030f0, r2=0x7ff038003880, l=0x7ff020003360, r=0x7ff038003640, sl=0x0, sr=0x0, nil_matches=0, nil_on_miss=0, semi=0, must_match=0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/gdk/gdk_join.c:1545
3 0x00007ff05481e122 in BATsubjoin (r1p=0x7ff04ae01738, r2p=0x7ff04ae01740, l=0x7ff020003360, r=0x7ff038003640, sl=0x0, sr=0x0, nil_matches=0, estimate=9223372036854775807) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/gdk/gdk_join.c:2952

(gdb) up
1 0x00007ff054617d7f in BATextend (b=0x7ff038003880, newcap=1337017706) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/gdk/gdk_bat.c:472
472 HEAPextend(&b->T->heap, theap_size, b->batRestricted == BAT_READ) != GDK_SUCCEED)

(gdb) p theap_size
$1 = 10696141648

(gdb) up
2 0x00007ff05480e3a6 in mergejoin (r1=0x7ff0380030f0, r2=0x7ff038003880, l=0x7ff020003360, r=0x7ff038003640, sl=0x0, sr=0x0, nil_matches=0, nil_on_miss=0, semi=0, must_match=0) at /opt/spinque/MonetDBServer/MonetDB.Spinque_Jul2015/src/gdk/gdk_join.c:1545
1545 BATextend(r2, newcap) != GDK_SUCCEED) {

(gdb) p newcap
$2 = 1337017706

(gdb) p nl
$3 = 1
(gdb) p nr
$4 = 413554

(gdb) p lcand
$5 = (const oid *) 0x0
(gdb) p rcand
$6 = (const oid *) 0x0

(gdb) p *l->H
$11 = {id = 0x7ff054872488 "h", width = 0, type = 0 '\000', shift = 0 '\000', varsized = 1, key = 1, dense = 1, nonil = 1, nil = 0, sorted = 1, revsorted = 0, align = 1052347, nokey = {0, 0}, nosorted = 0, norevsorted = 0, nodense = 0, seq = 0, heap = {free = 0, size = 0,
base = 0x0, filename = 0x0, copied = 0, hashash = 0, forcemap = 0, storage = STORE_MEM, newstorage = STORE_MEM, dirty = 0 '\000', farmid = 0 '\000', parentid = 0}, vheap = 0x0, hash = 0x0, imprints = 0x0, props = 0x0}
(gdb) p *l->T
$12 = {id = 0x7ff05487248a "t", width = 4, type = 5 '\005', shift = 2 '\002', varsized = 0, key = 1, dense = 0, nonil = 1, nil = 0, sorted = 1, revsorted = 0, align = 1052054, nokey = {0, 0}, nosorted = 0, norevsorted = 0, nodense = 0, seq = 0, heap = {free = 2717352,
size = 2717352, base = 0x7ff04a15e000 "\022", filename = 0x7ff03c1005a0 "16/1604.tail", copied = 0, hashash = 0, forcemap = 0, storage = STORE_MMAP, newstorage = STORE_MMAP, dirty = 0 '\000', farmid = 0 '\000', parentid = -900}, vheap = 0x0, hash = 0x0, imprints = 0x0,
props = 0x0}
(gdb) p *l->S
$13 = {tid = 140670022981377, stamp = 923, copiedtodisk = 0, dirty = 1, dirtyflushed = 0, descdirty = 1, restricted = 1, persistence = 1, role = 1, unused = 0, sharecnt = 0, map_head = 0 '\000', map_tail = 0 '\000', map_hheap = 0 '\000', map_theap = 0 '\000', deleted = 0,
first = 0, inserted = 0, count = 679338, capacity = 679338}
(gdb) p *r->H
$14 = {id = 0x7ff054872488 "h", width = 0, type = 0 '\000', shift = 0 '\000', varsized = 1, key = 1, dense = 1, nonil = 1, nil = 0, sorted = 1, revsorted = 0, align = 0, nokey = {0, 0}, nosorted = 0, norevsorted = 0, nodense = 0, seq = 12989232, heap = {free = 0, size = 0,
base = 0x0, filename = 0x0, copied = 0, hashash = 0, forcemap = 0, storage = STORE_MEM, newstorage = STORE_MEM, dirty = 0 '\000', farmid = 0 '\000', parentid = 0}, vheap = 0x0, hash = 0x0, imprints = 0x0, props = 0x0}
(gdb) p *r->T
$15 = {id = 0x7ff05487248a "t", width = 4, type = 5 '\005', shift = 2 '\002', varsized = 0, key = 0, dense = 0, nonil = 1, nil = 0, sorted = 1, revsorted = 0, align = 1053124, nokey = {0, 0}, nosorted = 0, norevsorted = 0, nodense = 0, seq = 0, heap = {free = 25978464,
size = 25978464, base = 0x7ff012b4ccc0 "\346\002", filename = 0x7ff03c101230 "16/1617.tail", copied = 0, hashash = 0, forcemap = 0, storage = STORE_MMAP, newstorage = STORE_MMAP, dirty = 0 '\000', farmid = 0 '\000', parentid = -911}, vheap = 0x0, hash = 0x0, imprints = 0x0,
props = 0x0}
(gdb) p *r->S
$16 = {tid = 140670025082625, stamp = 944, copiedtodisk = 0, dirty = 1, dirtyflushed = 0, descdirty = 1, restricted = 1, persistence = 1, role = 1, unused = 0, sharecnt = 0, map_head = 0 '\000', map_tail = 0 '\000', map_hheap = 0 '\000', map_theap = 0 '\000', deleted = 0,
first = 0, inserted = 0, count = 6494616, capacity = 6494616}
(gdb) p *r1->H
$20 = {id = 0x7ff054872488 "h", width = 0, type = 0 '\000', shift = 0 '\000', varsized = 1, key = 1, dense = 0, nonil = 1, nil = 0, sorted = 1, revsorted = 1, align = 0, nokey = {0, 0}, nosorted = 0, norevsorted = 0, nodense = 0, seq = 0, heap = {free = 0, size = 0, base = 0x0,
filename = 0x0, copied = 0, hashash = 0, forcemap = 0, storage = STORE_MEM, newstorage = STORE_MEM, dirty = 0 '\000', farmid = 0 '\000', parentid = 0}, vheap = 0x0, hash = 0x0, imprints = 0x0, props = 0x0}
(gdb) p *r1->T
$21 = {id = 0x7ff05487248a "t", width = 8, type = 6 '\006', shift = 3 '\003', varsized = 0, key = 0, dense = 0, nonil = 1, nil = 0, sorted = 1, revsorted = 0, align = 1093780, nokey = {0, 0}, nosorted = 0, norevsorted = 0, nodense = 0, seq = 0, heap = {free = 5265408,
size = 10696196096, base = 0x7fed916b0000 "\b\002", filename = 0x7ff038003450 "02/273.tail", copied = 0, hashash = 0, forcemap = 0, storage = STORE_MMAP, newstorage = STORE_MMAP, dirty = 0 '\000', farmid = 0 '\000', parentid = 0}, vheap = 0x0, hash = 0x0, imprints = 0x0,
props = 0x0}
(gdb) p *r1->S
$22 = {tid = 140670025082625, stamp = 945, copiedtodisk = 0, dirty = 1, dirtyflushed = 0, descdirty = 1, restricted = 0, persistence = 1, role = 1, unused = 0, sharecnt = 0, map_head = 0 '\000', map_tail = 0 '\000', map_hheap = 0 '\000', map_theap = 0 '\000', deleted = 0,
first = 0, inserted = 0, count = 658176, capacity = 1337017706}

Comment 21153

Date: 2015-08-17 18:08:31 +0200
From: @sjoerdmullender

Ok. I understand now what's happening. I just have to come up with a solution.
The problem is that you have a pretty skewed situation: the first couple of hundred values in the left input match a large number of values in the right input. I use extrapolation over the left input only to estimate the total result size, but because of the skew, the calculation misfires. And in this case, since the left input is key, I can even know that the result is never going to be larger than the right input.

Comment 21154

Date: 2015-08-18 12:34:08 +0200
From: MonetDB Mercurial Repository <>

Changeset 1dad1e7691a0 made by Sjoerd Mullender sjoerd@acm.org in the MonetDB repo, refers to this bug.

For complete details, see http//devmonetdborg/hg/MonetDB?cmd=changeset;node=1dad1e7691a0

Changeset description:

Calculate maximum join result size and use that to cap growth of BATs.
This should fix bug #3791.

Comment 21155

Date: 2015-08-18 13:22:53 +0200
From: @swingbit

The query that used to fail now works.

Comment 21156

Date: 2015-08-18 13:27:35 +0200
From: @sjoerdmullender

Thanks for testing.

Comment 21178

Date: 2015-08-28 13:41:28 +0200
From: @sjoerdmullender

Jul2015 has been released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant