New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query fails for default resource group but not for admin resource group #6349

Closed
kaknikhil opened this Issue Nov 27, 2018 · 8 comments

Comments

Projects
None yet
4 participants
@kaknikhil
Copy link
Member

kaknikhil commented Nov 27, 2018

When resource groups are enabled and the query is run within a newly created postgres user with the default resource group, sometimes we encounter this exception ERROR: insufficient memory reserved for statement

Greenplum version or build

PostgreSQL 8.3.23 (Greenplum Database 5.13.0+dev.30.gc2f693f build dev) on x86_64-pc-linux-gnu, compiled by GCC gcc (Ubuntu 6.5.0-1ubuntu1~16.04) 6.5.0 20181026, 64-bit compiled on Nov 23 2018 11:10:51

Step to reproduce the behavior

  1. Enable resource groups https://gpdb.docs.pivotal.io/530/admin_guide/workload_mgmt_resgroups.html
  2. Run the following query
gpadmin:~$ createdb test
gpadmin:~$ psql test
psql (8.3.23)
DROP TABLE IF EXISTS vertex, edge;
drop user if exists test_user;
create user test_user;
NOTICE:  resource group required -- using default resource group "default_group"
GRANT ALL PRIVILEGES  on schema public TO test_user;
SET ROLE test_user;

CREATE TABLE vertex(
        id INTEGER,
        name TEXT
        );
CREATE TABLE edge(
        src INTEGER,
        dest INTEGER,
        weight FLOAT8
        );
INSERT INTO vertex VALUES
(0, 'A'),
(1, 'B');

INSERT INTO edge VALUES
(0, 1, 1.0);

DROP TABLE IF EXISTS out_degrees;
CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT

    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT

    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);
ERROR:  insufficient memory reserved for statement

Running the same query without setting the test_user role uses the admin_group resource group which goes through just fine. The default_group resource group has more resources than the admin_group resource group, so it doesn't make sense that the query fails when running in the default_group and not when running in admin_group.

Increasing the memory_limit to 70 for default_group makes it work but that's just a workaround.

@kaknikhil

This comment has been minimized.

Copy link
Member Author

kaknikhil commented Nov 27, 2018

Another way to repro this is to change the resource group of the default superuser from admin_group to default_group.

ALTER ROLE gpadmin  resource group default_group;
@gaos1

This comment has been minimized.

Copy link
Member

gaos1 commented Nov 29, 2018

could you share your complete settings for default and admin group by the following commands? single query memory quota can be calculated roughly by group memory limit divided by group concurrency.

\x
select * from gp_toolkit.gp_resgroup_config;

@gaos1 gaos1 self-assigned this Nov 29, 2018

@goutamtadi1

This comment has been minimized.

Copy link
Contributor

goutamtadi1 commented Nov 29, 2018

On a container, we see

gpadmin=# select * from gp_toolkit.gp_resgroup_config;
 groupid |   groupname   | concurrency | proposed_concurrency | cpu_rate_limit | memory_limit | proposed_memory_limit | memory_shared_quota | proposed_memor
y_shared_quota | memory_spill_ratio | proposed_memory_spill_ratio
---------+---------------+-------------+----------------------+----------------+--------------+-----------------------+---------------------+---------------
---------------+--------------------+-----------------------------
    6437 | default_group | 20          | 20                   | 30             | 30           | 30                    | 50                  | 50
               | 20                 | 20
    6438 | admin_group   | 10          | 10                   | 10             | 10           | 10                    | 50                  | 50
               | 20                 | 20
(2 rows)
@goutamtadi1

This comment has been minimized.

Copy link
Contributor

goutamtadi1 commented Nov 29, 2018

We are testing with the madlib "install check" suite, and NOT the isolated query above. The pg_log file includes the following entries:

2018-11-29 19:11:37.710303 UTC,"gpadmin","gpadmin",p2761,th1204098880,"[local]",,2018-11-29 19:11:37 UTC,0,con57,cmd51,seg-1,,dx80,,sx1,"LOG","00000","statement: SET gp_default_storage_options TO 'compresstype=none,blocksize=32768,appendonly=False,orientation=row,checksum=true'",,,,,,"SET gp_default_storage_options TO 'compresstype=none,blocksize=32768,appendonly=False,orientation=row,checksum=true'",0,,"postgres.c",1590,
2018-11-29 19:11:37.728350 UTC,"gpadmin","gpadmin",p2759,th1204098880,"127.0.0.1","39784",2018-11-29 19:11:36 UTC,0,con57,cmd52,seg-1,,dx81,,sx1,"LOG","00000","statement: DROP TABLE IF EXISTS s;",,,,,,"DROP TABLE IF EXISTS s;",0,,"postgres.c",1590,
2018-11-29 19:11:37.745142 UTC,"gpadmin","gpadmin",p2759,th1204098880,"127.0.0.1","39784",2018-11-29 19:11:36 UTC,0,con57,cmd53,seg-1,,dx82,,sx1,"LOG","00000","statement: SELECT matrix_sparsify('d', 'row=row_id', 's', 'row=row_id, col=col_id, val=val');",,,,,,"SELECT matrix_sparsify('d', 'row=row_id', 's', 'row=row_id, col=col_id, val=val');",0,,"postgres.c",1590,
2018-11-29 19:11:37.746057 UTC,,,p2138,th1204098880,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
Unexpected exception reached top of execution stack: major=200 minor=14 file=CTranslatorQueryToDXL.cpp line=349",,,,,,,,"SysLoggerMain","syslogger.c",633,
2018-11-29 19:11:37.746083 UTC,"gpadmin","gpadmin",p2759,th1204098880,"127.0.0.1","39784",2018-11-29 19:11:36 UTC,0,con57,cmd53,seg-1,,dx82,,sx1,"LOG","00000","2018-11-29 19:11:37:745962 UTC,THD000,NOTICE,""Feature not supported: SIRV functions"",",,,,,,"SELECT matrix_sparsify('d', 'row=row_id', 's', 'row=row_id, col=col_id, val=val');",0,,"COptTasks.cpp",591,
2018-11-29 19:11:37.746104 UTC,"gpadmin","gpadmin",p2759,th1204098880,"127.0.0.1","39784",2018-11-29 19:11:36 UTC,0,con57,cmd53,seg-1,,dx82,,sx1,"LOG","00000","Planner produced plan :0",,,,,,"SELECT matrix_sparsify('d', 'row=row_id', 's', 'row=row_id, col=col_id, val=val');",0,,"orca.c",60,
2018-11-29 19:11:37.748806 UTC,,,p2138,th1204098880,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
Unexpected exception reached top of execution stack: major=200 minor=14 file=CTranslatorUtils.cpp line=162",,,,,,,,"SysLoggerMain","syslogger.c",618,
2018-11-29 19:11:37.748884 UTC,"gpadmin","gpadmin",p2759,th1204098880,"127.0.0.1","39784",2018-11-29 19:11:36 UTC,0,con57,cmd54,seg-1,,dx82,,sx2,"LOG","00000","Planner produced plan :0",,,,,"SQL statement ""SELECT nspname, proname FROM pg_proc p JOIN pg_namespace n ON (p.pronamespace = n.oid) WHERE p.oid = 16584""
PL/Python function ""matrix_sparsify""","SELECT matrix_sparsify('d', 'row=row_id', 's', 'row=row_id, col=col_id, val=val');",0,,"orca.c",60,
2018-11-29 19:11:37.750908 UTC,"gpadmin","gpadmin",p2761,th1204098880,"[local]",,2018-11-29 19:11:37 UTC,0,con57,cmd55,seg-1,,dx82,,sx1,"LOG","00000","statement: SET gp_default_storage_options TO 'compresstype=none,blocksize=32768,appendonly=False,orientation=row,checksum=true'",,,,,,"SET gp_default_storage_options TO 'compresstype=none,blocksize=32768,appendonly=False,orientation=row,checksum=true'",0,,"postgres.c",1590,
2018-11-29 19:11:37.758682 UTC,,,p2138,th1204098880,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
Unexpected exception reached top of execution stack: major=200 minor=14 file=CTranslatorUtils.cpp line=162",,,,,,,,"SysLoggerMain","syslogger.c",633,
2018-11-29 19:11:37.758736 UTC,"gpadmin","gpadmin",p2759,th1204098880,"127.0.0.1","39784",2018-11-29 19:11:36 UTC,0,con57,cmd56,seg-1,,dx82,,sx6,"LOG","00000","Planner produced plan :0",,,,,"SQL statement ""
            SELECT EXISTS(
                SELECT 1
                FROM pg_class, pg_namespace
                WHERE relnamespace = pg_namespace.oid
                  AND nspname IN ('pg_catalog','madlib_installcheck_linalg','madlib')
                  AND (relname = 'd')
                  AND relkind IN ('r', 'v', 'm', 't', 'f')
            ) AS table_exists
            ""
PL/Python function ""matrix_sparsify""","SELECT matrix_sparsify('d', 'row=row_id', 's', 'row=row_id, col=col_id, val=val');",0,,"orca.c",60,
2018-11-29 19:11:37.768522 UTC,,,p2138,th1204098880,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
Unexpected exception reached top of execution stack: major=200 minor=14 file=CTranslatorUtils.cpp line=162",,,,,,,,"SysLoggerMain","syslogger.c",618,
2018-11-29 19:11:37.768571 UTC,"gpadmin","gpadmin",p2759,th1204098880,"127.0.0.1","39784",2018-11-29 19:11:36 UTC,0,con57,cmd58,seg-1,,dx82,,sx8,"LOG","00000","Planner produced plan :0",,,,,"SQL statement ""
            SELECT EXISTS(
                SELECT 1
                FROM pg_class, pg_namespace
                WHERE relnamespace = pg_namespace.oid
                  AND nspname IN ('pg_catalog','madlib_installcheck_linalg','madlib')
                  AND (relname = 's')
                  AND relkind IN ('r', 'v', 'm', 't', 'f')
            ) AS table_exists
            ""
PL/Python function ""matrix_sparsify""","SELECT matrix_sparsify('d', 'row=row_id', 's', 'row=row_id, col=col_id, val=val');",0,,"orca.c",60,
2018-11-29 19:11:37.770783 UTC,,,p2138,th1204098880,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
Unexpected exception reached top of execution stack: major=200 minor=14 file=CTranslatorUtils.cpp line=162",,,,,,,,"SysLoggerMain","syslogger.c",618,
2018-11-29 19:11:37.770991 UTC,"gpadmin","gpadmin",p2759,th1204098880,"127.0.0.1","39784",2018-11-29 19:11:36 UTC,0,con57,cmd59,seg-1,,dx82,,sx9,"LOG","00000","Planner produced plan :0",,,,,"SQL statement ""
        SELECT format_type(atttypid, atttypmod) AS data_type
        FROM pg_attribute
        WHERE attrelid = 'd'::regclass
          AND NOT attisdropped
          AND attnum > 0
          AND attname = 'val'
        ""
PL/Python function ""matrix_sparsify""","SELECT matrix_sparsify('d', 'row=row_id', 's', 'row=row_id, col=col_id, val=val');",0,,"orca.c",60,
2018-11-29 19:11:37.773551 UTC,,,p2138,th1204098880,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
Unexpected exception reached top of execution stack: major=200 minor=14 file=CTranslatorUtils.cpp line=162",,,,,,,,"SysLoggerMain","syslogger.c",618,
2018-11-29 19:11:37.773647 UTC,"gpadmin","gpadmin",p2759,th1204098880,"127.0.0.1","39784",2018-11-29 19:11:36 UTC,0,con57,cmd60,seg-1,,dx82,,sx10,"LOG","00000","Planner produced plan :0",,,,,"SQL statement ""SELECT array_agg(quote_ident(attname)::varchar
                                     ORDER BY attnum) AS cols
                    FROM pg_attribute
                    WHERE attrelid = 'd'::regclass
                      AND NOT attisdropped
                      AND attnum > 0""
PL/Python function ""matrix_sparsify""","SELECT matrix_sparsify('d', 'row=row_id', 's', 'row=row_id, col=col_id, val=val');",0,,"orca.c",60,
2018-11-29 19:11:37.853399 UTC,,,p2138,th1204098880,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
Unexpected exception reached top of execution stack: major=200 minor=14 file=CTranslatorUtils.cpp line=162",,,,,,,,"SysLoggerMain","syslogger.c",618,
2018-11-29 19:11:37.853454 UTC,"gpadmin","gpadmin",p2759,th1204098880,"127.0.0.1","39784",2018-11-29 19:11:36 UTC,0,con57,cmd64,seg-1,,dx82,,sx14,"LOG","00000","Planner produced plan :0",,,,,"SQL statement ""
          SELECT EXISTS(
                SELECT 1
                FROM pg_class, pg_namespace
                WHERE relnamespace = pg_namespace.oid
                  AND nspname IN ('pg_catalog','madlib_installcheck_linalg','madlib')
                  AND (relname = 'd')
                  AND relkind IN ('r', 'v', 'm', 't', 'f')
            ) AS table_exists
            ""
PL/Python function ""matrix_sparsify""","SELECT matrix_sparsify('d', 'row=row_id', 's', 'row=row_id, col=col_id, val=val');",0,,"orca.c",60,
2018-11-29 19:11:37.855548 UTC,,,p2138,th1204098880,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
Unexpected exception reached top of execution stack: major=200 minor=14 file=CTranslatorUtils.cpp line=162",,,,,,,,"SysLoggerMain","syslogger.c",633,
2018-11-29 19:11:37.855574 UTC,"gpadmin","gpadmin",p2759,th1204098880,"127.0.0.1","39784",2018-11-29 19:11:36 UTC,0,con57,cmd65,seg-1,,dx82,,sx15,"LOG","00000","Planner produced plan :0",,,,,"SQL statement ""SELECT array_agg(quote_ident(attname)::varchar
                                     ORDER BY attnum) AS cols
                    FROM pg_attribute
                    WHERE attrelid = 'd'::regclass
                      AND NOT attisdropped
                      AND attnum > 0""
PL/Python function ""matrix_sparsify""","SELECT matrix_sparsify('d', 'row=row_id', 's', 'row=row_id, col=col_id, val=val');",0,,"orca.c",60,
2018-11-29 19:11:37.857826 UTC,,,p2138,th1204098880,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
Unexpected exception reached top of execution stack: major=200 minor=14 file=CTranslatorUtils.cpp line=162",,,,,,,,"SysLoggerMain","syslogger.c",633,
2018-11-29 19:11:37.857850 UTC,"gpadmin","gpadmin",p2759,th1204098880,"127.0.0.1","39784",2018-11-29 19:11:36 UTC,0,con57,cmd66,seg-1,,dx82,,sx16,"LOG","00000","Planner produced plan :0",,,,,"SQL statement ""
        SELECT format_type(atttypid, atttypmod) AS data_type
        FROM pg_attribute
        WHERE attrelid = 'd'::regclass
          AND NOT attisdropped
          AND attnum > 0
          AND attname = 'val'
        ""
PL/Python function ""matrix_sparsify""","SELECT matrix_sparsify('d', 'row=row_id', 's', 'row=row_id, col=col_id, val=val');",0,,"orca.c",60,
2018-11-29 19:11:37.991459 UTC,"gpadmin","gpadmin",p2761,th1204098880,"[local]",,2018-11-29 19:11:37 UTC,0,con57,cmd68,seg-1,,dx82,,sx1,"LOG","00000","statement: SET gp_default_storage_options TO 'compresstype=none,blocksize=32768,appendonly=False,orientation=row,checksum=true'",,,,,,"SET gp_default_storage_options TO 'compresstype=none,blocksize=32768,appendonly=False,orientation=row,checksum=true'",0,,"postgres.c",1590,
2018-11-29 19:11:38.052363 UTC,"gpadmin","gpadmin",p2759,th1204098880,"127.0.0.1","39784",2018-11-29 19:11:36 UTC,0,con57,cmd68,seg-1,,dx82,,sx1,"ERROR","53000","plpy.SPIError: insufficient memory reserved for statement",,,,,"Traceback (most recent call last):
  PL/Python function ""matrix_sparsify"", line 22, in <module>
    matrix_out, out_args)
  PL/Python function ""matrix_sparsify"", line 480, in matrix_sparsify
  PL/Python function ""matrix_sparsify"", line 433, in validate_dense
PL/Python function ""matrix_sparsify""","SELECT matrix_sparsify('d', 'row=row_id', 's', 'row=row_id, col=col_id, val=val');",0,,"plpython.c",5038,
2018-11-29 19:11:38.052488 UTC,"gpadmin","gpadmin",p2759,th1204098880,"127.0.0.1","39784",2018-11-29 19:11:36 UTC,0,con57,cmd68,seg-1,,dx82,,sx1,"LOG","00000","An exception was encountered during the execution of statement: SELECT matrix_sparsify('d', 'row=row_id', 's', 'row=row_id, col=col_id, val=val');",,,,,,,0,,,,
@kaknikhil

This comment has been minimized.

Copy link
Member Author

kaknikhil commented Nov 29, 2018

@gaos1

Here are the settings on the container

foo=> select * from gp_toolkit.gp_resgroup_config;
-[ RECORD 1 ]----------------+--------------
groupid                      | 6437
groupname                    | default_group
concurrency                  | 20
proposed_concurrency         | 20
cpu_rate_limit               | 30
memory_limit                 | 30
proposed_memory_limit        | 30
memory_shared_quota          | 50
proposed_memory_shared_quota | 50
memory_spill_ratio           | 20
proposed_memory_spill_ratio  | 20
-[ RECORD 2 ]----------------+--------------
groupid                      | 6438
groupname                    | admin_group
concurrency                  | 10
proposed_concurrency         | 10
cpu_rate_limit               | 10
memory_limit                 | 10
proposed_memory_limit        | 10
memory_shared_quota          | 50
proposed_memory_shared_quota | 50
memory_spill_ratio           | 20
proposed_memory_spill_ratio  | 20

Here is the explain analyze from the admin_group

foo=# explain analyze CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);
NOTICE:  Table doesn't have 'DISTRIBUTED BY' clause. Creating a NULL policy entry.
                                                                   QUERY PLAN

-----------------------------------------------------------------------------------------------------------------
--------------------------------
 Result  (cost=0.00..0.00 rows=0 width=0)
   Rows out:  2 rows (seg0) with 1.317 ms to first row, 1.637 ms to end.
   ->  Redistribute Motion 2:2  (slice2; segments: 2)  (cost=0.00..2586.14 rows=2 width=20)
         Rows out:  2 rows at destination (seg0) with 1.314 ms to first row, 1.632 ms to end.
         ->  Result  (cost=0.00..2586.00 rows=2 width=24)
               Rows out:  Avg 1.0 rows x 2 workers.  Max 1 rows (seg0) with 2.458 ms to first row, 3.518 ms to en
d.
               ->  Sequence  (cost=0.00..2586.00 rows=2 width=24)
                     Rows out:  Avg 1.0 rows x 2 workers.  Max 1 rows (seg0) with 2.448 ms to first row, 3.508 ms
 to end.
                     ->  Shared Scan (share slice:id 2:0)  (cost=0.00..431.00 rows=1 width=1)
                           Rows out:  1 rows (seg0) with 1.453 ms to end.
                           ->  Materialize  (cost=0.00..431.00 rows=1 width=1)
                                 Rows out:  0 rows (seg0) with 1.452 ms to end.
                                 ->  GroupAggregate  (cost=0.00..431.00 rows=1 width=12)
                                       Group By: public.edge.dest
                                       Rows out:  1 rows (seg0) with 1.361 ms to end.
                                       ->  Sort  (cost=0.00..431.00 rows=1 width=12)
                                             Sort Key: public.edge.dest
                                             Sort Method:  quicksort  Max Memory: 33KB  Avg Memory: 33KB (2 segme
nts)
                                             Rows out:  1 rows (seg0) with 1.357 ms to end.
                                             Executor memory:  33K bytes avg, 33K bytes max (seg0).
                                             Work_mem used:  33K bytes avg, 33K bytes max (seg0). Workfile: (0 sp
illing)
                                             ->  Redistribute Motion 2:2  (slice1; segments: 2)  (cost=0.00..431.
00 rows=1 width=12)
                                                   Hash Key: public.edge.dest
                                                   Rows out:  1 rows at destination (seg0) with 1.339 ms to first
 row, 1.340 ms to end.
                                                   ->  Result  (cost=0.00..431.00 rows=1 width=12)
                                                         Rows out:  1 rows (seg1) with 0.034 ms to first row, 0.0
35 ms to end.
                                                         ->  GroupAggregate  (cost=0.00..431.00 rows=1 width=12)
                                                               Group By: public.edge.dest
                                                               Rows out:  1 rows (seg1) with 0.034 ms to end.
                                                               ->  Sort  (cost=0.00..431.00 rows=1 width=4)
                                                                     Sort Key: public.edge.dest
                                                                     Sort Method:  quicksort  Max Memory: 33KB  A
vg Memory: 33KB (2 segments)
                                                                     Rows out:  1 rows (seg1) with 0.030 ms to fi
rst row, 0.031 ms to end.
                                                                     Executor memory:  33K bytes avg, 33K bytes m
ax (seg0).
                                                                     Work_mem used:  33K bytes avg, 33K bytes max
 (seg0). Workfile: (0 spilling)
                                                                     ->  Table Scan on edge  (cost=0.00..431.00 r
ows=1 width=4)
                                                                           Filter: src <> dest AND NOT src IS NUL
L AND NOT dest IS NULL
                                                                           Rows out:  1 rows (seg1) with 0.021 ms
 to first row, 0.022 ms to end.
                     ->  Sequence  (cost=0.00..2155.00 rows=2 width=24)
                           Rows out:  Avg 1.0 rows x 2 workers.  Max 1 rows (seg0) with 0.993 ms to first row, 2.
052 ms to end.
                           ->  Shared Scan (share slice:id 2:1)  (cost=0.00..431.00 rows=1 width=1)
                                 Rows out:  1 rows (seg1) with 0.113 ms to end.
                                 ->  Materialize  (cost=0.00..431.00 rows=1 width=1)
                                       Rows out:  0 rows (seg0) with 0.112 ms to end.
                                       ->  GroupAggregate  (cost=0.00..431.00 rows=1 width=12)
                                             Group By: public.edge.src
                                             Rows out:  1 rows (seg1) with 0.033 ms to end.
                                             ->  Sort  (cost=0.00..431.00 rows=1 width=4)
                                                   Sort Key: public.edge.src
                                                   Sort Method:  quicksort  Max Memory: 33KB  Avg Memory: 33KB (2
 segments)
                                                   Rows out:  1 rows (seg1) with 0.031 ms to end.
                                                   Executor memory:  33K bytes avg, 33K bytes max (seg0).
                                                   Work_mem used:  33K bytes avg, 33K bytes max (seg0). Workfile:
 (0 spilling)
                                                   ->  Table Scan on edge  (cost=0.00..431.00 rows=1 width=4)
                                                         Filter: src <> dest AND NOT src IS NULL AND NOT dest IS
NULL
                                                         Rows out:  1 rows (seg1) with 0.020 ms to first row, 0.0
21 ms to end.
                           ->  Append  (cost=0.00..1724.00 rows=2 width=24)
                                 Rows out:  Avg 1.0 rows x 2 workers.  Max 1 rows (seg0) with 0.891 ms to first r
ow, 1.949 ms to end.
                                 ->  Hash Left Join  (cost=0.00..862.00 rows=1 width=24)
                                       Hash Cond: share0_ref2.dest = share1_ref2.src
                                       Rows out:  1 rows (seg0) with 0.890 ms to first row, 1.190 ms to end.
                                       Executor memory:  1K bytes.
                                       Work_mem used:  1K bytes. Workfile: (0 spilling)
                                       ->  Shared Scan (share slice:id 2:0)  (cost=0.00..431.00 rows=1 width=12)
                                             Rows out:  1 rows (seg0) with 0.001 ms to first row, 0.011 ms to end
.
                                       ->  Hash  (cost=431.00..431.00 rows=1 width=12)
                                             Rows in:  1 rows (seg1) with 0.004 ms to end, start offset by 7.783
ms.
                                             ->  Shared Scan (share slice:id 2:1)  (cost=0.00..431.00 rows=1 widt
h=12)
                                                   Rows out:  1 rows (seg1) with 0.003 ms to end.
                                 ->  Result  (cost=0.00..862.00 rows=1 width=24)
                                       Rows out:  1 rows (seg1) with 0.532 ms to first row, 0.682 ms to end.
                                       ->  Hash Left Anti Semi Join  (cost=0.00..862.00 rows=1 width=12)
                                             Hash Cond: share1_ref3.src = share0_ref3.dest
                                             Rows out:  1 rows (seg1) with 0.531 ms to first row, 0.681 ms to end
.
                                             Executor memory:  1K bytes.
                                             Work_mem used:  1K bytes. Workfile: (0 spilling)
                                             ->  Shared Scan (share slice:id 2:1)  (cost=0.00..431.00 rows=1 widt
h=12)
                                                   Rows out:  1 rows (seg1) with 0.001 ms to end.
                                             ->  Hash  (cost=431.00..431.00 rows=1 width=4)
                                                   Rows in:  1 rows (seg0) with 0.003 ms to end, start offset by
8.335 ms.
                                                   ->  Shared Scan (share slice:id 2:0)  (cost=0.00..431.00 rows=
1 width=4)
                                                         Rows out:  1 rows (seg0) with 0.001 ms to end.
 Slice statistics:
   (slice0)    Executor memory: 172K bytes avg x 2 workers, 172K bytes max (seg0).
   (slice1)    Executor memory: 207K bytes avg x 2 workers, 207K bytes max (seg0).  Work_mem: 33K bytes max.
   (slice2)    Executor memory: 2523K bytes avg x 2 workers, 2531K bytes max (seg0).  Work_mem: 33K bytes max.
 Statement statistics:
   Memory used: 0K bytes
 Optimizer status: PQO version 3.9.0
 Total runtime: 10.635 ms
(90 rows)

Here is the master log for the isolated non MADlib query failure with default_group. I set client_min_messages and log_min_messages to debug5.

2018-11-29 22:21:54.465997 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd3,seg-1,,,,,"DEBUG5","00000","First char: 'Q'; gp_role = 'dispatch'.",,,,,,,0,,"postgres.c",4940,
2018-11-29 22:21:54.466071 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd3,seg-1,,,,,"DEBUG1","00000","Message type Q received by from libpq, len = 523",,,,,,,0,,"postgres.c",4948,
2018-11-29 22:21:54.466337 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd3,seg-1,,,,,"DEBUG5","00000","Simple query stmt: CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);.",,,,,,,0,,"postgres.c",4955,
2018-11-29 22:21:54.466415 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd3,seg-1,,,,,"DEBUG5","00000","setupRegularDtxContext leaving with DistributedTransactionContext = 'Master Distributed-Capable'.",,,,,,,0,,"cdbtm.c",3452,
2018-11-29 22:21:54.466424 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,,,,"DEBUG3","00000","StartTransactionCommand",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"postgres.c",3202,
2018-11-29 22:21:54.466506 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","createDtx created new distributed transaction gid = 1543277087-0000020569, gxid = 20569.",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"cdbtm.c",2601,
2018-11-29 22:21:54.466556 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","setting SharedLocalSnapshotSlot->startTimestamp = 596845314466010[old=596845308900717])",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"xact.c",2331,
2018-11-29 22:21:54.466624 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","[Distributed Snapshot #0] *StartTransaction* (gxid = 20569, xid = 0, 'Master Distributed-Capable')",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"xact.c",2446,
2018-11-29 22:21:54.466685 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG3","00000","StartTransaction",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"xact.c",5656,
2018-11-29 22:21:54.466745 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG3","00000","name: unnamed; blockState:       DEFAULT; state: INPROGR, xid/subid/cid: 0/1/0, nestlvl: 1, children:",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"xact.c",5694,
2018-11-29 22:21:54.466837 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","StartTransaction in DTX Context = 'Master Distributed-Capable', distributed transaction {timestamp 0, xid 0} for local xid 0",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"xact.c",2531,
2018-11-29 22:21:54.466915 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"LOG","00000","statement: CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"postgres.c",1590,
2018-11-29 22:21:54.467039 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","setupRegularDtxContext leaving with DistributedTransactionContext = 'Master Distributed-Capable'.",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"cdbtm.c",3452,
2018-11-29 22:21:54.467101 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","GetSnapshotData maxCount 250, inProgressEntryArray 0x2f70c70",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"procarray.c",1066,
2018-11-29 22:21:54.467178 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","GetSnapshotData serializable true, xmin 0",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"procarray.c",1300,
2018-11-29 22:21:54.467248 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","GetSnapshotData setting globalxmin and xmin to 38984",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"procarray.c",1321,
2018-11-29 22:21:54.467315 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","FillInDistributedSnapshot DTX Context = 'Master Distributed-Capable'",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"procarray.c",884,
2018-11-29 22:21:54.467377 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","createDtxSnapshot distributed snapshot has xmin = 20569, count = 0, xmax = 20569.",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"cdbtm.c",2529,
2018-11-29 22:21:54.467458 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","[Distributed Snapshot #265320] *Create* (gxid = 20569, 'Master Distributed-Capable')",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"cdbtm.c",2534,
2018-11-29 22:21:54.467535 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","Got distributed snapshot from DistributedSnapshotWithLocalXids_Create = true",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"procarray.c",907,
2018-11-29 22:21:54.467623 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","GetSnapshotData(): WRITER currentcommandid 0 curcid 0 segmatesync 0",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"procarray.c",1490,
2018-11-29 22:21:54.468126 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG2","00000","Rolling over previous outstanding Optimizer allocated memory 0",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"memaccounting.c",700,
2018-11-29 22:21:54.472845 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG2","00000","
[OPT]: Using default search strategy",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"COptTasks.cpp",758,
2018-11-29 22:21:54.476000 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"NOTICE","00000","Table doesn't have 'DISTRIBUTED BY' clause. Creating a NULL policy entry.",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"CTranslatorQueryToDXL.cpp",844,
2018-11-29 22:21:54.529863 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG1","00000","GPORCA produced plan",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"orca.c",50,
2018-11-29 22:21:54.530004 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG3","00000","ProcessQuery",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"pquery.c",215,
2018-11-29 22:21:54.530063 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","[Distributed Snapshot #0] (gxid = 20569, 'Master Distributed-Capable')",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"tqual.c",1407,
2018-11-29 22:21:54.530130 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","GetSnapshotData maxCount 0, inProgressEntryArray (nil)",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"procarray.c",1066,
2018-11-29 22:21:54.530196 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","GetSnapshotData serializable false, xmin 38984",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"procarray.c",1300,
2018-11-29 22:21:54.530252 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","GetSnapshotData setting globalxmin and xmin to 38984",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"procarray.c",1321,
2018-11-29 22:21:54.530299 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","FillInDistributedSnapshot DTX Context = 'Master Distributed-Capable'",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"procarray.c",884,
2018-11-29 22:21:54.530362 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","createDtxSnapshot distributed snapshot has xmin = 20569, count = 0, xmax = 20569.",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"cdbtm.c",2529,
2018-11-29 22:21:54.530420 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","[Distributed Snapshot #265321] *Create* (gxid = 20569, 'Master Distributed-Capable')",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"cdbtm.c",2534,
2018-11-29 22:21:54.530488 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","Got distributed snapshot from DistributedSnapshotWithLocalXids_Create = true",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"procarray.c",907,
2018-11-29 22:21:54.530544 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd4,seg-1,,dx20569,,sx1,"DEBUG5","00000","GetSnapshotData(): WRITER currentcommandid 0 curcid 0 segmatesync 0",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"procarray.c",1490,
2018-11-29 22:21:54.530591 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd5,seg-1,,dx20569,,sx1,"ERROR","53000","insufficient memory reserved for statement",,,,,,"CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",0,,"memquota.c",1023,
2018-11-29 22:21:54.530640 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd5,seg-1,,dx20569,,sx1,"LOG","00000","An exception was encountered during the execution of statement: CREATE TABLE out_degrees AS
SELECT
coalesce(in_q.vertex, out_q.vertex) as id,
coalesce(indegree, 0) as indegree,
coalesce(outdegree, 0) as outdegree
FROM
(
SELECT
    dest as vertex,
    count(*) as indegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         dest
) as in_q
FULL OUTER JOIN
(
SELECT
    src as vertex,
    count(*) as outdegree
FROM edge
WHERE src != dest AND
      src IS NOT NULL AND
      dest IS NOT NULL
GROUP BY
         src
) as out_q
USING ( vertex);",,,,,,,0,,,,
2018-11-29 22:21:54.530686 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd5,seg-1,,dx20569,,sx1,"DEBUG5","00000","rollbackDtxTransaction called with state = Active Not Distributed, gid = 1543277087-0000020569",,,,,,,0,,"cdbtm.c",1155,
2018-11-29 22:21:54.530693 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd5,seg-1,,dx20569,,sx1,"DEBUG5","00000","releaseGxact called for gid = 1543277087-0000020569 (index = 0)",,,,,,,0,,"cdbtm.c",2622,
2018-11-29 22:21:54.530700 UTC,"gpadmin","foo",p41439,th-157141184,"[local]",,2018-11-29 22:21:34 UTC,0,con3858,cmd5,seg-1,,,,,"DEBUG5","00000","finishDistributedTransactionContext called to change DistributedTransactionContext from Master Distributed-Capable to Local Only (caller = CleanupTransaction, gxid = 0)",,,,,,,0,,"cdbtm.c",3747,

@gaos1 The logs that @goutamtadi1 posted in the above comment are for a madlib query . This query is different than the one in the description but fails for the same insufficient memory error. It will be much easier to debug the non MADlib query because then you don't need to install MADlib.

@pivotal-ning-yu

This comment has been minimized.

Copy link
Member

pivotal-ning-yu commented Nov 30, 2018

Thanks for the report, I can reproduce the issue with @kaknikhil 's reproducer by using the admin_group. It happens when ORCA is on only.

By converting this error report to an assert we got below information with gdb:

(gdb) bt
#0  0x00007f6359f04f97 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7ffc62f37560) at ../sysdeps/unix/sysv/linux/select.c:41
#1  0x0000563953e77b72 in pg_usleep (microsec=30000000) at pgsleep.c:43
#2  0x0000563953c72739 in elog_debug_linger (edata=0x56395425a000 <errordata>) at elog.c:4342
#3  0x0000563953c6994a in errfinish (dummy=0) at elog.c:616
#4  0x0000563953c67fad in ExceptionalCondition (conditionName=0x56395407f710 "!(!\"insufficient memory reserved for statement\")", errorType=0x56395407f38b "FailedAssertion",
    fileName=0x56395407f380 "memquota.c", lineNumber=1022) at assert.c:49
#5  0x0000563953cf7d3e in PolicyEagerFreeAssignOperatorMemoryKB (stmt=0x563956741240, memAvailableBytes=1048576) at memquota.c:1022
#6  0x00005639538fa74b in ExecutorStart (queryDesc=0x563955b73100, eflags=0) at execMain.c:312
#7  0x0000563953b22f22 in ProcessQuery (portal=0x563955991d20, stmt=0x563956741240, params=0x0, dest=0x5639567539f0, completionTag=0x7ffc62f37d40 "") at pquery.c:291
#8  0x0000563953b24f40 in PortalRunMulti (portal=0x563955991d20, isTopLevel=1 '\001', dest=0x5639567539f0, altdest=0x5639567539f0, completionTag=0x7ffc62f37d40 "") at pquery.c:1463
#9  0x0000563953b243ca in PortalRun (portal=0x563955991d20, count=9223372036854775807, isTopLevel=1 '\001', dest=0x5639567539f0, altdest=0x5639567539f0, completionTag=0x7ffc62f37d40 "") at pquery.c:1025
#10 0x0000563953b1bc1e in exec_simple_query (
    query_string=0x56395598dce0 "CREATE TABLE out_degrees AS\nSELECT\ncoalesce(in_q.vertex, out_q.vertex) as id,\ncoalesce(indegree, 0) as indegree,\ncoalesce(outdegree, 0) as outdegree\nFROM\n(\nSELECT\n    dest as vertex,\n    count(*) as i"..., seqServerHost=0x0, seqServerPort=-1) at postgres.c:1776
#11 0x0000563953b20eb1 in PostgresMain (argc=1, argv=0x563955808998, dbname=0x5639558087d8 "u", username=0x563955808798 "u") at postgres.c:4962
#12 0x0000563953aab51c in BackendRun (port=0x5639558107d0) at postmaster.c:6732
#13 0x0000563953aaaaa2 in BackendStartup (port=0x5639558107d0) at postmaster.c:6406
#14 0x0000563953aa2e02 in ServerLoop () at postmaster.c:2444
#15 0x0000563953aa13f5 in PostmasterMain (argc=15, argv=0x5639557df7d0) at postmaster.c:1528
#16 0x00005639539aac7b in main (argc=15, argv=0x5639557df7d0) at main.c:206

(gdb) f 5
#5  0x0000563953cf7d3e in PolicyEagerFreeAssignOperatorMemoryKB (stmt=0x563956741240, memAvailableBytes=1048576) at memquota.c:1022
1022    Assert(!"insufficient memory reserved for statement");

(gdb) l -
1017             * Check if memory exceeds the limit in the root group
1018             */
1019            const uint64 nonMemIntenseOpMemKB = (uint64)(*gp_resmanager_memory_policy_auto_fixed_mem);
1020            if (ctx.groupTree->groupMemKB < ctx.groupTree->numNonMemIntenseOps * nonMemIntenseOpMemKB)
1021            {
1022    Assert(!"insufficient memory reserved for statement");
1023                    ereport(ERROR,
1024                            (errcode(ERRCODE_INSUFFICIENT_RESOURCES),
1025                            errmsg("insufficient memory reserved for statement")));
1026            }

(gdb) p ctx.groupTree->groupMemKB
$9 = 1024

(gdb) p ctx.groupTree->numNonMemIntenseOps
$10 = 13

(gdb) p nonMemIntenseOpMemKB
$11 = 100

We can see that this plan contains 13 non-memory-intense operators, 1024 < 13 * 100. The value 1024 is decided by the group's spill ratio, in my environment the master is granted 879MB in all, so the admin_group has 879M * 10% = 87M memory, each concurrent has 87M * 50% / 10 = 4.35MB, the spill ratio is 20%, so 4.35MB * 20% = 0.87MB. To run the query successfully we can increase the spill ratio slightly, e.g.

-- set at session level
set memory_spill_ratio to 40;

-- or set at resgroup level
alter resource group admin_group set memory_spill_ratio 40;
@pivotal-ning-yu

This comment has been minimized.

Copy link
Member

pivotal-ning-yu commented Nov 30, 2018

Discussed with @gaos1 , we think one improvement is to relax the spill memory check, when the spill memory is too low instead of raising an error a better choice is to increase spill memory slightly. By doing so the query could run successfully if there is enough shared memory (this is usually the case), and will only fail at execution time when the shared memory is also too low. I'll work on a PR for this change.

@pivotal-ning-yu

This comment has been minimized.

Copy link
Member

pivotal-ning-yu commented Dec 17, 2018

This is fixed on 5X by #6500, it should be included in next 5.x release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment