From a917034422858c1c64278cd450d2a63f5591a34c Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 3 Aug 2026 13:36:33 -0600 Subject: [PATCH] fix: hold what fits in the fetch cache instead of dropping it all (#359) The fetch cache dropped an entry whole when it exceeded COLUMNAR_FETCH_CACHE_MAX_BYTES, so an entry one byte over was not retained at all and every fetch re-read the group and re-decoded every column it touched. On the 100M fixture that is 2,833 ms at four aggregate columns and 134,147 ms at five, flat either side: a 47x step inside the space of ordinary queries. #357 shrank entries ~3x by moving decode scratch out, which moved the threshold from "any wide table" to "five or more aggregate columns" without changing the shape. Raising the cap would move it again. Decoding was already per column and lazy; only the eviction was per entry. Make the eviction as granular as the admission: each column decodes into its own child context, and a column that takes the entry over the cap is released after its value has been extracted and decodes into per-fetch scratch from then on. The columns admitted before it stay resident, and groupBuffer stays either way, so no fetch re-reads the group from disk. The resident set is first-fit and never rotates. That is load-bearing rather than incidental: the access pattern is cyclic, every fetch touching the same projected columns in attribute order, and LRU against a cyclic pattern evicts precisely the column about to be needed -- a 100% miss rate, which is the behaviour being removed. The position indexes from #143 stay in the entry context so a released column still reaches its row in constant time. A group whose raw bytes alone exceed the cap is still dropped whole, which keeps the cache bounded by 4 x the cap. columnar_index_fetch_penalty modelled this cliff by treating every group as re-decoded once the projection crossed the cap. It now scales by the overflow fraction, so the cost model and the cache agree (#355). Measured, PG18 assert, four to five projected columns: before 77 ms -> 1902 ms (24.7x) after 64 ms -> 368 ms (5.8x) test/native_fetch_cache.sh gains a projection-width case. The existing #353 case varies group size at a fixed two-column projection, so it never crosses the relocated cap and went green on the query family that still cliffed. The new check fails on unmodified main (24.9x against a 12x bound) while its two correctness checks pass there, so the timing check is what detects the fix. Gate: full suite matrix, PG18 and PG19 assert, in the pgcolumnar-dev container. Co-Authored-By: Claude Opus 5 (1M context) --- design/ISSUE_359_FETCH_CACHE_PARTIAL.md | 126 ++++++++++++++++++ src/columnar_customscan.c | 23 +++- src/columnar_reader.c | 164 +++++++++++++++++++----- src/columnar_vacuum.c | 21 +-- test/native_fetch_cache.sh | 67 ++++++++++ 5 files changed, 358 insertions(+), 43 deletions(-) create mode 100644 design/ISSUE_359_FETCH_CACHE_PARTIAL.md diff --git a/design/ISSUE_359_FETCH_CACHE_PARTIAL.md b/design/ISSUE_359_FETCH_CACHE_PARTIAL.md new file mode 100644 index 0000000..73cc1fd --- /dev/null +++ b/design/ISSUE_359_FETCH_CACHE_PARTIAL.md @@ -0,0 +1,126 @@ +# #359: make fetch-cache overflow proportional instead of a cliff + +Status: design, 2026-08-03. Successor to #353/#357. Coupled to #355 (merged as #360). + +## The defect + +`columnar_fetch_row` (`src/columnar_reader.c`) ends every fetch with: + +```c +if (MemoryContextMemAllocated(entry->cx, true) > COLUMNAR_FETCH_CACHE_MAX_BYTES) + columnar_fetch_entry_reset(entry); +``` + +That is all-or-nothing. An entry one byte over the 32 MB cap is not retained at +all, so the next fetch into the same group re-reads the group's bytes from disk +**and** re-decodes every column it touches. Measured on the 100M TSBS fixture, +holding rows and plan shape constant and varying only the number of aggregated +columns: 4 columns 2,833 ms, 5 columns 134,147 ms. A 47x step, flat either side. + +#357 shrank entries ~3x by moving decode scratch out. That moved the threshold +from "any wide table" to "five or more aggregate columns". It did not change the +shape of the failure, and raising the cap would not either. + +## What the entry already gets right + +Decoding is **already per column and lazy**: `entry->rawBuf[c]` is filled only +when column `c` is actually touched, and a column outside the projection is +neither read nor decoded. The cache is therefore already column-granular on the +way *in*. It is only the eviction that is whole-entry. + +That is the whole fix: make eviction as granular as admission already is. + +## Design: stable per-column admission + +Keep, always: + +- `groupBuffer` — the group's raw bytes. This is the on-disk (encoded, + compressed) form, and retaining it alone removes the per-fetch `ColumnarReadLogicalData` + disk read even when no column stays resident. +- `rankPrefix[c]` and `valOffset[c]` — the position indexes from #143. These are + small relative to the decoded stream (`rankPrefix` is 4 bytes per 64 rows; + `valOffset` is 4 bytes per value against a value that is typically wider), and + they remain **valid for a re-decode**, because decoding the same chunk bytes is + deterministic and produces the same layout. Keeping them is what stops #143's + quadratic from coming back through the overflow path. + +Evict, per column: + +- After a column is decoded *and its value has been extracted*, if the entry is + over the cap, delete just that column's decoded stream and mark the column + overflowed. From then on that column decodes into the per-fetch scratch + context (`tmp`) and is freed with it. + +Admission order is attribute order, which is deterministic, so the resident set +is **stable**: the same columns stay resident on every fetch into that group. + +### Why the resident set must be stable, not LRU + +This is the load-bearing decision. An LRU eviction *within* the entry is the +obvious design and it is wrong here. The access pattern is cyclic: each fetch +touches columns 0..n-1 in order. LRU against a cyclic access pattern whose +working set exceeds the cache evicts precisely the entry about to be needed — + +- fetch 1 decodes 0,1,2,3,4; over cap, LRU evicts 0 +- fetch 2 wants 0: miss, decode; over cap, LRU evicts 1 +- fetch 2 wants 1: miss, decode; over cap, LRU evicts 2 ... + +— giving a 100% miss rate, i.e. exactly today's behaviour with more bookkeeping. +A stable resident set of `k` of `n` columns re-decodes `n - k` per fetch. That is +the proportionality the issue asks for, and it is why "retain what fits" has to +mean *first-fit, then stop*, not *keep the hottest*. + +### Memory bound is preserved + +Today the entry transiently exceeds the cap (it is measured after the fetch that +filled it) and is then dropped. Under this design the entry transiently exceeds +the cap by one column's decoded stream and is then trimmed back under it, so +steady-state residency is still <= cap per entry, 4 entries. + +One new case: if `groupBuffer` *alone* exceeds the cap, every column overflows +and the entry would hold an unbounded raw buffer. Guard that explicitly — do not +cache the entry at all in that case, which is exactly today's behaviour and keeps +the bound at 4 x cap. + +### Two details that will bite + +1. **Baseline encoding aliases the group buffer.** When the encoding descriptor + is `COLUMNAR_NATIVE_ENCDESC_BASELINE`, `rawBuf[c] = base + validityBytes` — + a pointer *into* `groupBuffer`, not an allocation. It costs nothing to retain + and must never be "evicted" (that would be a free of an interior pointer). + Only a decoded stream is evictable. + +2. **The value must be extracted before the eviction.** `ColumnarDecodeValue` + copies into the caller's context for every case (by-value returns the datum; + fixed-length and varlena both `MemoryContextAlloc(targetContext)` + `memcpy`), + so freeing the column's stream after extraction is safe — but only after. + +## Consistency with #355 + +`columnar_index_fetch_penalty` (`src/columnar_customscan.c:638`) currently has: + +```c +/* #359 cliff: a group too wide to cache is re-decoded on every fetch */ +if ((double) rel->reltarget->width * R > (double) COLUMNAR_FETCH_CACHE_MAX_BYTES) + groups_decoded = groups_max; +``` + +That branch models the cliff this change removes. It must soften to the overflow +fraction: when the decoded group exceeds the cap, the share of the decode that +repeats per fetch is the share of the projection that does not fit, so the +penalty scales by that fraction rather than jumping to "every group re-decoded". + +## Test + +`test/native_fetch_cache.sh`'s #353 case measures `count(*), max(u)` — a +two-column projection — across two group sizes. That shape never crosses the +relocated cap, which is why it went green on the query family that still cliffs. +Both the issue author and the reviewer generalised from a single projection +width; the test has to vary the axis that was held constant. + +Add a case that **varies projection width** at fixed group size, and assert the +absence of a step rather than an absolute time: crossing the cap must cost +proportionally more, not multiples more. + +Per `prove-guards-by-removal`: the new check must be shown to fail with this +change reverted, on the same container, before it counts. diff --git a/src/columnar_customscan.c b/src/columnar_customscan.c index 689399b..938f599 100644 --- a/src/columnar_customscan.c +++ b/src/columnar_customscan.c @@ -609,6 +609,7 @@ columnar_index_fetch_penalty(RelOptInfo *rel, double rows, double rho, double groups_min, groups_max, groups_decoded, + decoded_width, csq; if (rows <= 0 || R < 1) @@ -635,9 +636,25 @@ columnar_index_fetch_penalty(RelOptInfo *rel, double rows, double rho, groups_decoded = groups_min + (groups_max - groups_min) * (1.0 - csq); } - /* #359 cliff: a group too wide to cache is re-decoded on every fetch */ - if ((double) rel->reltarget->width * R > (double) COLUMNAR_FETCH_CACHE_MAX_BYTES) - groups_decoded = groups_max; + /* + * A group too wide to hold entirely in the fetch cache re-decodes the part + * that did not fit, once per fetch rather than once per group. + * + * This was a cliff -- the whole entry was dropped, so exceeding the cap by + * any margin meant every group re-decoded, and the model said so. #359 made + * the cache admit columns until the cap and re-decode only the remainder, so + * the extra decoding is now the overflow *fraction* of the projection. Model + * it the same way: blend between decoding each group once and decoding one + * per fetch, by how much of the decoded group does not fit. + */ + decoded_width = (double) rel->reltarget->width * R; + if (decoded_width > (double) COLUMNAR_FETCH_CACHE_MAX_BYTES) + { + double resident = (double) COLUMNAR_FETCH_CACHE_MAX_BYTES / + decoded_width; + + groups_decoded += (groups_max - groups_decoded) * (1.0 - resident); + } if (groups_decoded < 0) groups_decoded = 0; diff --git a/src/columnar_reader.c b/src/columnar_reader.c index 7c69838..38257c7 100644 --- a/src/columnar_reader.c +++ b/src/columnar_reader.c @@ -1793,6 +1793,15 @@ ColumnarFreeLivenessCache(ColumnarLivenessCache *cache) * sizing from disk would overshoot worst on exactly the tables this helps most. * COLUMNAR_FETCH_CACHE_MAX_BYTES is defined in columnar.h so the index-fetch cost * model (#355) can name the same cap. + * + * The cap is enforced per column, not per entry (issue #359). It used to drop the + * whole entry, which made it a cliff: an entry one byte over was not retained at + * all, so every fetch re-read the group and re-decoded every column it touched. + * On the 100M fixture that was 2,833 ms at four aggregate columns and 134,147 at + * five, flat either side -- a 47x step inside the space of ordinary queries. + * Shrinking entries (#357 moved decode scratch out, ~3x) only moved the step. + * Now the columns that fit stay resident and only the remainder is re-decoded, so + * crossing the cap costs the overflow fraction rather than everything. */ typedef struct ColumnarFetchGroup @@ -1824,6 +1833,27 @@ typedef struct ColumnarFetchGroup */ uint32 **rankPrefix; /* [natts]; NULL until that column is decoded */ uint32 **valOffset; /* [natts]; NULL for fixed-length columns */ + + /* + * Per-column residency, so exceeding the cap costs proportionally rather + * than totally (issue #359). + * + * colCx[c] holds column c's decoded stream, one child context per column so + * a single column can be released without disturbing the rest of the entry. + * It is NULL when the column is undecoded, and also when the column decodes + * to a pointer into groupBuffer (baseline encoding allocates nothing). + * + * overflow[c] marks a column that was decoded, did not fit, and was + * released. Such a column decodes into per-fetch scratch from then on. The + * mark is never cleared: the resident set has to be *stable*, because the + * access pattern here is cyclic -- every fetch touches the same projected + * columns in attribute order -- and evicting the least recently used column + * against a cyclic pattern evicts precisely the column about to be needed, + * which is a 100% miss rate, i.e. the very behaviour this removes. First + * fit and then stop re-decodes only the columns that did not fit. + */ + MemoryContext *colCx; /* [natts] */ + bool *overflow; /* [natts] */ uint64 lastUsed; } ColumnarFetchGroup; @@ -2230,6 +2260,8 @@ columnar_fetch_row(Relation rel, Snapshot snapshot, uint64 rowNumber, entry->rawBuf = palloc0(sizeof(char *) * natts); entry->rankPrefix = palloc0(sizeof(uint32 *) * natts); entry->valOffset = palloc0(sizeof(uint32 *) * natts); + entry->colCx = palloc0(sizeof(MemoryContext) * natts); + entry->overflow = palloc0(sizeof(bool) * natts); MemoryContextSwitchTo(tmp); if (rg->byteLength > 0) @@ -2288,6 +2320,7 @@ columnar_fetch_row(Relation rel, Snapshot snapshot, uint64 rowNumber, char *rawBuf; char *cursor; uint64 present; + bool justDecoded; /* this fetch decoded it; may not fit */ /* * A column the caller did not ask for is neither decoded nor indexed, @@ -2317,43 +2350,87 @@ columnar_fetch_row(Relation rel, Snapshot snapshot, uint64 rowNumber, continue; } - if (entry->rawBuf[c] == NULL) + if (entry->rawBuf[c] != NULL) + { + rawBuf = entry->rawBuf[c]; + justDecoded = false; + } + else { - MemoryContext decOld = MemoryContextSwitchTo(entry->cx); + /* + * A baseline chunk is not encoded, so its "decoded" stream is a + * pointer into groupBuffer rather than an allocation. It costs the + * entry nothing beyond the group bytes it already holds, so it is + * always resident and is never a candidate for release below -- + * releasing it would mean freeing an interior pointer. + */ + bool baseline = (cc->encodingDescriptorLen == 1 && + (uint8) cc->encodingDescriptor[0] == + COLUMNAR_NATIVE_ENCDESC_BASELINE); + MemoryContext decCx; + MemoryContext decOld; + + if (baseline) + decCx = entry->cx; + else if (entry->overflow[c]) + decCx = tmp; /* known not to fit: decode per fetch */ + else + decCx = AllocSetContextCreate(entry->cx, + "columnar fetch column", + ALLOCSET_DEFAULT_SIZES); - if (cc->encodingDescriptorLen == 1 && - (uint8) cc->encodingDescriptor[0] == COLUMNAR_NATIVE_ENCDESC_BASELINE) - entry->rawBuf[c] = base + validityBytes; + decOld = MemoryContextSwitchTo(decCx); + if (baseline) + rawBuf = base + validityBytes; else - entry->rawBuf[c] = - columnar_native_decode_chunk(entry->cx, att, + rawBuf = + columnar_native_decode_chunk(decCx, att, base + validityBytes, (uint32) (cc->pageLength - validityBytes), cc->encodingDescriptor, cc->encodingDescriptorLen, cc->blockCodec, NULL, NULL); + MemoryContextSwitchTo(decOld); /* * Index the column while it is being decoded, so every fetch into - * this group afterwards reaches its row directly (issue #143). Both - * indexes live in the entry context and so are measured by the cap - * below and dropped with the rest of the entry. + * this group afterwards reaches its row directly (issue #143). + * + * The indexes live in the entry context, not in the column's, so + * they outlive a released column. That is deliberate: decoding the + * same chunk bytes is deterministic and yields the same layout, so + * the offsets stay valid across a re-decode. An overflowed column + * therefore pays its decode again but still reaches its row in + * constant time, which is what stops #143's quadratic returning + * through this path. They are small next to the stream: rankPrefix + * is four bytes per 64 rows, valOffset four per value. */ - entry->rankPrefix[c] = columnar_build_rank_prefix(vbits, - entry->rowCount); - if (att->attlen < 0) + if (entry->rankPrefix[c] == NULL) { - uint64 nblocks = (entry->rowCount + - COLUMNAR_RANK_BLOCK_ROWS - 1) / - COLUMNAR_RANK_BLOCK_ROWS; + MemoryContext idxOld = MemoryContextSwitchTo(entry->cx); + + entry->rankPrefix[c] = columnar_build_rank_prefix(vbits, + entry->rowCount); + if (att->attlen < 0) + { + uint64 nblocks = (entry->rowCount + + COLUMNAR_RANK_BLOCK_ROWS - 1) / + COLUMNAR_RANK_BLOCK_ROWS; - entry->valOffset[c] = - columnar_build_val_offsets(att, entry->rawBuf[c], - entry->rankPrefix[c][nblocks]); + entry->valOffset[c] = + columnar_build_val_offsets(att, rawBuf, + entry->rankPrefix[c][nblocks]); + } + MemoryContextSwitchTo(idxOld); } - MemoryContextSwitchTo(decOld); + + if (decCx != tmp) + { + entry->rawBuf[c] = rawBuf; + entry->colCx[c] = baseline ? NULL : decCx; + } + justDecoded = true; } - rawBuf = entry->rawBuf[c]; /* * The row's value sits at the rank-th position in the present-value @@ -2371,20 +2448,45 @@ columnar_fetch_row(Relation rel, Snapshot snapshot, uint64 rowNumber, values[c] = ColumnarDecodeValue(att, &cursor, target); nulls[c] = false; + + /* + * Release this column if it is the one that took the entry over the cap + * (issue #359). Measuring the context is what makes the cap mean decoded + * bytes rather than stored bytes; a column of wide text decodes to many + * times its size on disk, and sizing from disk would overshoot worst on + * exactly the tables this helps most. + * + * Releasing here rather than dropping the whole entry is what makes + * going over the cap cost proportionally: the columns admitted before + * this one stay resident and are decoded once, and only the remainder is + * decoded per fetch. groupBuffer stays either way, so no fetch re-reads + * the group from disk. + * + * It is safe because nothing handed back points into the column: the + * value returned was copied into the caller's context by the + * ColumnarDecodeValue call immediately above, and the position indexes + * live in entry->cx rather than in the column's own context. + */ + if (justDecoded && entry->colCx[c] != NULL && + MemoryContextMemAllocated(entry->cx, true) > + COLUMNAR_FETCH_CACHE_MAX_BYTES) + { + MemoryContextDelete(entry->colCx[c]); + entry->colCx[c] = NULL; + entry->rawBuf[c] = NULL; + entry->overflow[c] = true; + } } /* - * Hold the entry only while it is worth holding. Measuring the context is - * what makes the cap mean decoded bytes rather than stored bytes; a group - * over the cap is used for this fetch and then dropped, so an outsized group - * costs what it always did rather than pinning memory. - * - * Dropping it here is safe because nothing handed back points into it: the - * value that is returned is decoded into the caller's context by the - * ColumnarDecodeValue call above. Only the decoded stream and its two - * position indexes live in entry->cx. + * The per-column release above trims the entry back under the cap, so the + * entry is bounded by it -- except in one case it cannot help: a group whose + * *raw* bytes alone exceed the cap. Every column would overflow and the entry + * would still pin groupBuffer, so four such entries could hold arbitrarily + * much. Drop the entry whole there, which is what this path did for every + * oversized group before, and keeps the cache bounded by 4 x the cap. */ - if (MemoryContextMemAllocated(entry->cx, true) > COLUMNAR_FETCH_CACHE_MAX_BYTES) + if (rg->byteLength > COLUMNAR_FETCH_CACHE_MAX_BYTES) columnar_fetch_entry_reset(entry); MemoryContextSwitchTo(oldContext); diff --git a/src/columnar_vacuum.c b/src/columnar_vacuum.c index ee84d7c..50adb8e 100644 --- a/src/columnar_vacuum.c +++ b/src/columnar_vacuum.c @@ -147,16 +147,19 @@ rewrite_one_group(Relation rel, ColumnarIndexInsertState *ris, uint64 storageId, * Stream the group rather than fetching its rows one at a time. * * ColumnarReadRowByNumber decodes the whole group to return one value and - * relies on the fetch cache to make the next call cheap. That cache drops - * any group whose decoded form exceeds COLUMNAR_FETCH_CACHE_MAX_BYTES, and - * drops it after every fetch -- so a group over the cap by any margin made - * this loop decode the entire group once per row. + * relies on the fetch cache to make the next call cheap. That cache holds + * only what fits under COLUMNAR_FETCH_CACHE_MAX_BYTES, so a group whose + * decoded form exceeds the cap re-decodes the columns that did not fit, once + * per row rather than once per group. * - * It is a cliff rather than a slope. Three columns of 150,000 rows with one - * varlena among them decodes to 34,713,408 bytes against a 33,554,432 cap: - * 3.5% over. On an idle box, rewriting 200,000 rows of that shape did not - * finish inside 120 seconds, while the same table without the middle column - * -- under the cap, so cached -- took 1.9. + * That used to be a cliff rather than a slope: the whole entry was dropped, + * so a group over the cap by any margin decoded entirely, per row. Three + * columns of 150,000 rows with one varlena among them decodes to 34,713,408 + * bytes against a 33,554,432 cap: 3.5% over. On an idle box, rewriting + * 200,000 rows of that shape did not finish inside 120 seconds, while the + * same table without the middle column -- under the cap, so cached -- took + * 1.9. #359 made the overflow proportional, which shrinks that gap but does + * not close it; streaming remains strictly cheaper than fetching per row. * * A reader restricted to this group decodes it once and walks it, which is * what the loop wanted all along, and it does not care how large the group diff --git a/test/native_fetch_cache.sh b/test/native_fetch_cache.sh index 09ff834..e6b3c8d 100755 --- a/test/native_fetch_cache.sh +++ b/test/native_fetch_cache.sh @@ -135,4 +135,71 @@ check "the wide-group point query is correct (#353)" \ "$(q "SELECT count(*) || '|' || coalesce(max(u)::text,'z') FROM fc_wide WHERE h='h1'")" \ "$(q "SELECT count(*) || '|' || coalesce(max(u)::text,'z') FROM fc_wnarrow WHERE h='h1'")" +# --- #359: crossing the cap must cost proportionally, not totally ------------- +# The #353 case above measures a two-column projection across two group sizes. It +# therefore never crosses the relocated cap, and went green on exactly the query +# family that still cliffed. The axis it holds constant is the one that mattered: +# projection width at a fixed group size. This varies that instead. +# +# Each text column below decodes to ~154 bytes x 50,000 rows = ~7.7 MB, so the +# 32 MB cap falls between four and five projected columns. Going over it used to +# drop the entry whole, so the fifth column cost a re-read of the group and a +# re-decode of all five, on every fetch. The cache now keeps the columns that fit +# and re-decodes only the remainder. +# +# Measured here, PG18 assert, going from four to five columns. Both figures are +# from this suite rather than from a standalone probe, because the ~600 MB of +# fixtures built above leave the box in a different state than a cold one, and the +# ratio moves with it -- the same fixed build measures 2.2x standalone and 5.8x +# here. Comparing a suite number against a standalone number would be comparing +# two machines: +# before 77 ms -> 1902 ms (24.7x, flat either side -- the cliff) +# after 64 ms -> 368 ms (5.8x -- a ramp) +# Four runs of the fixed build in this suite gave 5.6x, 5.6x, 5.9x and 6.4x. The +# bound is 12x, roughly a factor of two clear of either build, so it discriminates +# the shape rather than encoding one box's timings. +psql_run "DROP TABLE IF EXISTS fc_w359; + CREATE TABLE fc_w359 (id int, h text, + t1 text,t2 text,t3 text,t4 text,t5 text) USING pgcolumnar; + SELECT pgcolumnar.set_options('fc_w359', stripe_row_limit => 50000); + INSERT INTO fc_w359 SELECT g, 'h' || (g % 1000), + repeat('a',150),repeat('b',150),repeat('c',150), + repeat('d',150),repeat('e',150) + FROM generate_series(1,100000) g; + CREATE INDEX fc_w359_h ON fc_w359 (h);" + +w359_ms() { # number of projected text columns + local n=$1 sel="count(*)" i start end + for i in $(seq 1 "$n"); do sel="$sel, max(t$i)"; done + # warm, so the comparison is decode cost rather than first-touch I/O + psql_run "SET max_parallel_workers_per_gather=0; SET enable_seqscan=off; + SET enable_bitmapscan=off; + SELECT $sel FROM fc_w359 WHERE h='h7';" >/dev/null 2>&1 + start=$(date +%s%N) + psql_run "SET max_parallel_workers_per_gather=0; SET enable_seqscan=off; + SET enable_bitmapscan=off; + SELECT $sel FROM fc_w359 WHERE h='h7';" >/dev/null 2>&1 + end=$(date +%s%N); echo $(( (end - start) / 1000000 )) +} + +under="$(w359_ms 4)" # ~30 MB decoded: fits +over="$(w359_ms 5)" # ~38 MB decoded: does not +echo "-- #359 projection width: four columns ${under} ms, five columns ${over} ms" +check_timing "crossing the fetch cache cap costs proportionally, not totally (#359)" \ + "$( [ "$under" -gt 0 ] && [ $(( over / (under > 0 ? under : 1) )) -lt 12 ] && echo yes || + echo "no (four=${under}ms five=${over}ms)")" \ + "yes" + +# The columns that overflow are re-decoded rather than skipped, so they must still +# read correctly -- a cache that quietly returned nulls for them would be fast and +# wrong, and the timing check alone would not notice. +check "the over-cap projection returns the same values as the under-cap one (#359)" \ + "$(q "SELECT max(t1) || '|' || max(t4) FROM fc_w359 WHERE h='h7'")" \ + "$(q "SELECT max(t1) || '|' || max(t4) FROM fc_w359 WHERE h='h7' AND t5 IS NOT NULL")" +check "every over-cap column reads back its written value (#359)" \ + "$(q "SELECT count(*) FROM fc_w359 WHERE h='h7' + AND t1 = repeat('a',150) AND t2 = repeat('b',150) AND t3 = repeat('c',150) + AND t4 = repeat('d',150) AND t5 = repeat('e',150)")" \ + "$(q "SELECT count(*) FROM fc_w359 WHERE h='h7'")" + pgc_summary