No toast or table

[blaketan]

Problem:

Production warehouse instance has a large amount of tables, and makes heavy use of TOAST tables. Also every one of our extension tables are tables with 1000 columns of numeric arrays

Because Postgrex uses the binary wire protocol, it relies on maintaining an internal ETS Elixir.Postgrex.Types table to encode and decode types using OIDs. Which would be fine, except that the bootstrap query used in Postgrex pulled in way too much data, and blew up the memory usage of the table.

Solution

Modified fix as suggested in elixir-ecto#320, BUT we change a top level predicate from (t.typelem = 0 OR t.typelem NOT IN (SELECT oid FROM pg_catalog.pg_type WHERE typrelid!=0)) to (t.typelem = 0 OR t.typelem IN (SELECT oid FROM pg_catalog.pg_type WHERE typrelid = 0)), because the suggested version of the query times out after 5 minutes.

production_db=> SELECT COUNT(*) FROM (SELECT oid FROM pg_catalog.pg_type WHERE typrelid!=0) "eg";
 count
--------
 217964
(1 row)

production_db=> SELECT COUNT(*) FROM (SELECT oid FROM pg_catalog.pg_type WHERE typrelid=0) "eg";
 count
-------
 93486
(1 row)

This makes the bootstrap query as follows:

SELECT t.oid, t.typname, t.typsend, t.typreceive, t.typoutput, t.typinput,
       coalesce(d.typelem, t.typelem), coalesce(r.rngsubtype, 0), ARRAY (
  SELECT a.atttypid
  FROM pg_attribute AS a
  WHERE a.attrelid = t.typrelid AND a.attnum > 0 AND NOT a.attisdropped
  ORDER BY a.attnum
)
FROM pg_type AS t
LEFT JOIN pg_type AS d ON t.typbasetype = d.oid
LEFT JOIN pg_range AS r ON r.rngtypid = t.oid OR (t.typbasetype <> 0 AND r.rngtypid = t.typbasetype)
WHERE (t.typrelid = 0)
AND (t.typelem = 0 OR t.typelem IN (SELECT oid FROM pg_catalog.pg_type WHERE typrelid = 0))

Which returns 168 rows, instead of 311450 rows.