Skip to content

Rebuild the roadmap against what the issue threads actually say - #322

Merged
ChronicallyJD merged 4 commits into
mainfrom
docs/roadmap-155-correction
Aug 2, 2026
Merged

Rebuild the roadmap against what the issue threads actually say#322
ChronicallyJD merged 4 commits into
mainfrom
docs/roadmap-155-correction

Conversation

@ChronicallyJD

@ChronicallyJD ChronicallyJD commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Started as a one-entry fix for #155. An audit of the whole open list found that
all four entries were wrong, including the replacement entry this branch had
just written. Rewritten accordingly.

The failure mode

#320 rebuilt the list from issue state. An issue being open does not mean its
text is still true. Checking state and not content produced three distinct errors:

entry what was wrong
#310 said the 100M measurement "has not been re-run"; it had been, 11 hours earlier, 273,212 buffers to 8,917
#300 framed the remaining work as core COPY's parse; #300's own profile had already refuted that
#289 a copy of the issue body, no sign that #307 merged or that #321 is open
reltuples fixed on 2026-07-28 by #189, with a cause that was explicitly disproven

What each entry says now

#289, first. The only item with work in flight. Decompression half merged
(#307). Aggregation half open as #321, measured at 1.20x/1.38x, not merged: review
reproduced two wrong-answer defects and a vacuous test section. The entry now says
plainly that 1.20x is not the ~4x prize, that the larger lever is dictionary-coded
grouping, and that q6, the widest gap at 5.3x behind TimescaleDB and the only shape
where columnar loses to heap, is untouched by anything in flight.

#300, second. Bulk load is done (#155). The remaining lever is parallel ingest,
prototyped at 7.39x on eight writers with core COPY unchanged, not a parse
bypass: parse is ~21% of the load, encode ~53%. IMPORT_THROUGHPUT_PLAN.md is
flagged as the wrong reference, since it predates #283 to #286 and puts COPY under
"Not in scope".

#291, third, newly added. Its documentation half landed in #298; the code
comments remain, and no gate covers them.

#310 is moved out of the list entirely, under "not work, but still open", with
the measured before/after and the one thing actually outstanding: a confirmation
reading on the real dataset.

reltuples is removed. Fixed and now exact.

Structural fixes, all verified

Docs-only, no code changes, no matrix run. test/docs_style.sh passes.

Every claim above was checked against the issue thread, the PR state, or main.

@ChronicallyJD ChronicallyJD changed the title Correct the roadmap's #155 entry, which was stale in both directions Rebuild the roadmap against what the issue threads actually say Aug 1, 2026
@ChronicallyJD

Copy link
Copy Markdown
Collaborator Author

Review: accurate, and the right failure-mode fix

I fact-checked every entry I have primary knowledge of against my own measurements and the merged state; they are correct:

The core point is right and worth stating: #320 rebuilt the list from issue state, and open ≠ still-true. Checking content caught three stale entries plus the one this branch had just written — a good catch on itself. No inaccuracies found on my read.

One small thing (non-blocking): the #90 commit for reclaim split/coalesce — I couldn't confirm the term by grep in columnar_storage.c; worth a second's check that the cited commit is the right hash, since the whole point here is precision.

@ChronicallyJD

Copy link
Copy Markdown
Collaborator Author

Your approval predates the current content, so please do not treat it as standing.

  • Your review: 2026-08-01T17:57:43Z
  • My last commit: 2026-08-01T18:09:05Z

What you fact-checked was the revision that an audit then found two blockers in,
and both were in text I had written to correct earlier errors:

  1. The paragraph correcting two "listed as not built, actually built" items called
    the F1 delete-vector rename the one still outstanding. It is built too: the
    catalog is named pgcolumnar.delete_vector today. The correction committed the
    error it was correcting.
  2. "q6 is the only shape where columnar loses to heap" is false. Vectorized decompression + aggregation for full-scan aggregates (~4× behind TimescaleDB) #289's own table
    shows q8 loses as well, 21,608 ms against heap's 15,315 ms. That sentence was
    load-bearing, since it was the justification for singling q6 out.

Also wrong in what you approved: "34x" did not follow from the two figures in its
own sentence (1252/39 is 32.1); #317 was credited with closing #314, which was
closed an hour before #317 existed; and the file claimed a PostgreSQL 13-19 matrix
in three places when it is 15-19.

I am not raising this to relitigate the review. I am raising it because it is
evidence for the change I then made. You wrote that you fact-checked every entry you
had priors on, and I believe you did; two blockers still survived. Four revisions of
this file have now been wrong, three of them in the correction. That is not four
careless people, it is a file whose entries restate measurements that live somewhere
else, where every restated number is a number that goes stale the moment its issue
moves.

So the current revision carries no measurements in the open list at all. Each
entry says what the work is, what is in flight, and which issue holds the figures.
Claims that cannot drift stay; claims that can are now pointers. That is the actual
proposal, and it is worth a fresh look precisely because it is a different shape
from what you approved.

Concretely, worth re-checking: the three entries no longer assert any number, the
"formerly deferred" paragraph now says all three are built with the schema name as
evidence, and 13-19 is corrected to 15-19 in all three places.

@ChronicallyJD

Copy link
Copy Markdown
Collaborator Author

Re-reviewed the current revision (fresh — not standing on the 17:57 approval)

You were right to pull the earlier approval; I re-checked this revision against the merged state, and the corrections hold:

One status has already drifted since the 18:09 commit — the exact failure mode this rewrite exists to prevent:

Fix that one line and it's good to land — everything else checks out.

jdatcmd and others added 4 commits August 1, 2026 20:08
The refresh in #320 rebuilt the open list from issue STATE and not from issue
CONTENT. #155 was still open, so its original text was carried forward verbatim,
including the "about 4.9x slower than heap" figure. That number was the problem
statement from before the work, not the result: four encoder levers (#283 to #286,
merged via #290) had already taken the 100M-row load from 783 s to 383 s and the
gap to heap from about 4.9x to about 2.2x, with a byte-identical on-disk image.

So the roadmap named finished work as the next thing to take, and quoted a measured
figure that had been superseded. #155 itself was left open only because the
resolution comment said "closing as resolved" without closing it; it is closed now,
after verifying the four commits are on main.

The entry becomes #300, which is the real remaining work: core COPY's per-field
parse, paid identically by heap and TimescaleDB, and not reachable by another
encoder change.

Refs #155, #300. No issue is closed by this commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8
The refresh in #320 rebuilt the open list from issue STATE. The follow-up commit on
this branch fixed the #155 entry but repeated the same mistake on the entry it
wrote to replace it. An audit of every entry against its issue thread, its pull
requests and main found that all four were wrong, in three different ways.

#289 was a copy of the issue body and gave no sign that work is in flight. The
decompression half already merged (#307, 3.8 percent on q4 and 3.2 on q5) and the
aggregation half is open as #321. The "about 4x behind TimescaleDB" line reads as
the size of the prize for that work, but #321 measures 1.20x and 1.38x, and by its
own account the larger lever is dictionary-coded grouping. The widest gap, q6 at
5.3x behind and 3.1x slower than heap, is the only shape where columnar loses to
heap and nothing in flight touches it.

#300 was framed as core COPY's per-field parse. #300's own profile refuted that
before the entry was written: parse is about 21 percent, encode about 53 percent,
so bypassing the parser cannot make columnar beat heap. The measured top lever is
parallelism over the existing encoder with COPY unchanged, prototyped at 7.39x.
IMPORT_THROUGHPUT_PLAN.md was cited as the reference and is the wrong pointer: it
predates the #283 to #286 work and puts COPY under "Not in scope".

reltuples is removed. It was fixed on 2026-07-28 by #189 and is now exact on every
measured shape, and the cause the entry gave was explicitly disproven: it was a
block-offset mismatch, not blocks holding no row-group data. The line was written
about nine hours before the fix and survived two refreshes.

#310 is no longer listed as work. Both causes are merged and it was re-measured at
100M, 273,212 buffers to 8,917. It stays open for a confirmation reading on the
real dataset.

#291 was open and absent from the list; added, with the note that its
documentation half landed in #298.

Also fixed, all verified: the "Deferred, not yet built" paragraph listed two things
that have been on main since 2026-07-23; a cross-reference to "item 0" that #320's
renumbering left dangling; six Done rows naming the extension schema as columnar
rather than pgcolumnar, which a reader copying them would find does not exist; and
a closed-since line with the wrong date and three omissions.

Refs #289, #300, #291, #310. No issue is closed by this commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8
…d why

This file has now been wrong four times, and the last three times the error was in
the correction. An audit of the previous revision found two blockers in it:

The paragraph correcting two "listed as not built, actually built" errors called
the F1 delete-vector rename the one item still outstanding. It is built too, and
the schema shows it directly: the catalog is named pgcolumnar.delete_vector today.
The correcting paragraph committed the error it was correcting.

The #289 entry said q6 is the only shape where columnar loses to heap. #289's own
table shows q8 loses as well, 21,608 ms against heap's 15,315 ms. That sentence
was load-bearing: it was the justification for singling q6 out.

Also wrong and now fixed: "34x" did not follow from the two figures in its own
sentence (1252/39 is 32.1); #317 was credited with closing #314, which was closed
an hour before #317 was created; and the file claimed a PostgreSQL 13-19 matrix in
three places when the matrix is 15-19.

The common cause is structural, not clerical. Every revision restated
measurements that live in the issues, and a restated measurement drifts the moment
the issue moves. So the open list now carries no measurements at all. Each entry
says what the work is, what is in flight, and which issue holds the current
figures. Claims that cannot drift stay; claims that can are replaced by a pointer.

Refs #289, #300, #291, #310. No issue is closed by this commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8
The open list said the grouped aggregate (#321) "is not merged" with a review-state
detail, but #321 merged 2026-08-01T22:31:39Z -- exactly the restated status this
rewrite exists to stop carrying. State the durable fact (landed, behind the
default-off GUC) and let #321 hold its own state; keep the pointer to its body for
the numbers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UX1jrWiQsJJA1t4pkmkb4T
@ChronicallyJD
ChronicallyJD force-pushed the docs/roadmap-155-correction branch from b03a435 to 05c33e6 Compare August 2, 2026 02:08
@ChronicallyJD
ChronicallyJD merged commit 7ae8e6c into main Aug 2, 2026
11 checks passed
@ChronicallyJD
ChronicallyJD deleted the docs/roadmap-155-correction branch August 2, 2026 02:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants