From 1d2d8a99de500476e83741cbd704b05a775c0044 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Sun, 9 Aug 2026 20:52:18 -0600 Subject: [PATCH 1/2] test: the sorted-SUITES check pins its collation (#552) harness_selftest asserted the SUITES array is sorted without pinning a locale, and "sorted" is not machine-independent. Reproduced on unmodified main: $ LC_ALL=en_US.UTF-8 bash test/harness_selftest.sh .../pg17/bin/pg_config FAIL the suite list is sorted, so two new suites land in different places harness_selftest.sh: FAILED One pair decides it. C compares byte by byte so `_` (0x5F) precedes `e`; en_US.UTF-8 ignores punctuation at the first level and compares `sortstatus` against `sortedprojection`, where `e` precedes `s`: LC_ALL=C sort_status, sorted_projection LC_ALL=en_US.UTF-8 sorted_projection, sort_status The array holds the C order, so it is sorted in one locale and unsorted in the other. This has never been seen here because the container defaults to C.UTF-8, which collates like C. A contributor on en_US.UTF-8 gets a red on a clean checkout, unrelated to their change. The ambiguity matters more than the false red. Sorted order is what gives two agents' new suites different insertion points -- the measurement in that file shows one-per-line alone still conflicts when both append. If two contributors disagree about what sorted means, they insert in different places and the property stops delivering the merges it exists for. Pinned to LC_ALL=C, with the deciding pair asserted directly so that an edit "fixing" the order to UTF-8 collation fails with the reason rather than only failing the comparison. en_US.UTF-8 was FAILED, now 55 checks PASSED C.UTF-8 55 checks PASSED Proved by removal: unpinning the sort reddens the named check under en_US.UTF-8. CONTEXT.md:167 says "one name per line and sorted" and should say C order, since "sorted" alone is what created the ambiguity. Deliberately NOT changed here: #545 is open against CONTEXT.md, and two open PRs editing one file is the collision I raised on #546/#547 an hour ago. It goes in whichever of the two merges second. Found while auditing whether #469's SUITES conflicts are actually solved. They are -- three suites from two agents landed today with no conflict -- but the sortedness that depends on was locale-dependent. Closes #552 --- test/harness_selftest.sh | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/test/harness_selftest.sh b/test/harness_selftest.sh index 12ea1b51..59401f19 100755 --- a/test/harness_selftest.sh +++ b/test/harness_selftest.sh @@ -305,11 +305,31 @@ rm -f "$_fx" # # This check is what keeps the property true. Without it the order decays the # first time somebody appends by hand, and the reduction quietly goes away. -_sorted_expected="$(listed_suites | sort)" +# +# LC_ALL=C, and the collation is part of the property rather than a detail (#552). +# "Sorted" is not machine-independent: C compares byte by byte so `_` (0x5F) +# precedes `e`, while en_US.UTF-8 ignores punctuation at the first level and +# compares `sortstatus` against `sortedprojection`. The array holds sort_status +# then sorted_projection, so it is sorted in one and unsorted in the other, and +# this check pinned neither. On a clean main it FAILED under en_US.UTF-8 and +# passed here only because the container defaults to C.UTF-8, which collates +# like C. +# +# The ambiguity is worse than the false red. Sorted order is what gives two +# agents' new suites different insertion points; if two contributors disagree +# about what sorted means they insert in different places, and the property stops +# delivering the merges it exists for. +_sorted_expected="$(listed_suites | LC_ALL=C sort)" _sorted_actual="$(listed_suites)" -check "the suite list is sorted, so two new suites land in different places" \ +check "the suite list is sorted in C order, so two new suites land in different places" \ "$([ "$_sorted_actual" = "$_sorted_expected" ] && echo sorted || echo "not sorted")" "sorted" +# The pair that decides it, asserted directly so a future edit that "fixes" the +# order to UTF-8 collation fails here with the reason rather than only failing +# the comparison above. +check "premise: C collation puts sort_status before sorted_projection" \ + "$(printf 'sorted_projection\nsort_status\n' | LC_ALL=C sort | head -1)" "sort_status" + # A case over the cached list rather than `listed_suites | grep -qx`. The pipe # was the defect: grep -q returns on its match, printf takes EPIPE, and pipefail # turns that into a failed pipeline for a suite that IS registered. See the note From ae24140be818c9cb8a68ce22fe74b46640324d0c Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 10 Aug 2026 08:53:27 -0600 Subject: [PATCH 2/2] test: comm's two inputs are sorted the same way (#552 follow-up) Residual found by @ChronicallyJD reviewing #553. They counted the blast radius rather than trusting the single instance: 13 uses of sort/comm/join across test/*.sh, exactly one feeding an equality comparison against a stored order, which is the one this PR already pinned. This is the other one. test/rebuild.sh:130 runs `comm -23` over two `sort -u` outputs, neither pinned. comm requires both inputs in ONE collation and does not check -- fed inconsistently sorted input it does not error, it returns the wrong lines. They agree today because they share a locale. The hazard is the plausible next edit: somebody pins ONE of them because this PR taught them to, and the symbol check silently reports the wrong unresolved symbols. Either a false red, or the worse direction, a real unresolved symbol not reported at all. That function is what catches an ABI-incompatible .so before it reaches a cluster. All four pinned: both comm inputs, the ldd sort beside them for consistency, and comm itself. Inputs sorted one way and compared another is the same defect with an extra step. Guarded rather than only fixed, since the point is that the next edit is the dangerous one: a check asserts that any file using comm pins the collation of every sort feeding it. Over source text, which is the weaker kind, because reproducing this needs two locales and a built .so; premised on a comm still existing so the grep cannot approve a file that no longer has one. before FAIL a file that uses comm pins the collation of every sort feeding it: got [rebuild.sh] want [] after 81 checks, PASSED Proved by removal: unpinning either comm input reddens the check by name. rebuild.sh is what devloop and every suite build through, so harness_selftest and native_vecdecode were run against the change rather than assuming a pin is inert. Refs #552 --- test/harness_selftest.sh | 28 ++++++++++++++++++++++++++++ test/rebuild.sh | 11 +++++++---- 2 files changed, 35 insertions(+), 4 deletions(-) diff --git a/test/harness_selftest.sh b/test/harness_selftest.sh index 57185b6c..a4f38ed8 100755 --- a/test/harness_selftest.sh +++ b/test/harness_selftest.sh @@ -330,6 +330,34 @@ check "the suite list is sorted in C order, so two new suites land in different check "premise: C collation puts sort_status before sorted_projection" \ "$(printf 'sorted_projection\nsort_status\n' | LC_ALL=C sort | head -1)" "sort_status" +# ---- and comm's two inputs must be sorted the SAME way (#552 follow-up) ----- +# +# `comm` requires both inputs sorted in one collation and does not check. Fed +# inconsistently-sorted input it does not error; it returns the wrong lines. +# +# test/rebuild.sh:130 does `comm -23` over two `sort -u` outputs, neither pinned. +# They agree today because they share a locale. The plausible next edit is +# somebody pinning ONE of them because this PR taught them to, and the result is +# a symbol check that silently reports the wrong unresolved symbols -- either a +# false red, or the worse direction, a real unresolved symbol not reported. +# +# Asserted over source text, which is the weaker kind, because reproducing it +# needs two locales and a built .so. Premised on the comm still existing, or the +# grep approves a file that no longer has one. +_cm_files="$(grep -ln 'comm -' "$(dirname "${BASH_SOURCE[0]}")"/*.sh 2>/dev/null)" +check "premise: some suite still uses comm, or the check below is vacuous" \ + "$([ -n "$_cm_files" ] && echo yes || echo no)" "yes" + +_cm_unpinned="" +for _f in $_cm_files; do + # every `| sort` in a file that uses comm must carry LC_ALL=C + if grep -qE '\|[[:space:]]*sort' "$_f" && grep -E '\|[[:space:]]*sort' "$_f" | grep -qv 'LC_ALL=C'; then + _cm_unpinned="$_cm_unpinned $(basename "$_f")" + fi +done +check "a file that uses comm pins the collation of every sort feeding it" \ + "$(printf '%s' "$_cm_unpinned" | sed 's/^ //')" "" + # A case over the cached list rather than `listed_suites | grep -qx`. The pipe # was the defect: grep -q returns on its match, printf takes EPIPE, and pipefail # turns that into a failed pipeline for a suite that IS registered. See the note diff --git a/test/rebuild.sh b/test/rebuild.sh index e1f4afd9..60499a1f 100755 --- a/test/rebuild.sh +++ b/test/rebuild.sh @@ -116,18 +116,21 @@ if command -v nm >/dev/null 2>&1; then IGNORE='^(_ITM_|__gmon_start__$|__cxa_finalize$)' undef="$(nm -D --undefined-only "$SO" 2>/dev/null | awk '{print $NF}' | - strip_ver | grep -Ev "$IGNORE" | sort -u)" + strip_ver | grep -Ev "$IGNORE" | LC_ALL=C sort -u)" # The .so is dlopen'd into the running postgres, so its symbols resolve against # the server binary, everything the server itself links (libm, libssl, ...), and # the .so's own dependencies. All three belong in the reference set. defined="$(nm -D --defined-only "$BINDIR/postgres" 2>/dev/null | awk '{print $NF}')" for lib in $(ldd "$SO" "$BINDIR/postgres" 2>/dev/null | - awk '/=>/ {print $3}' | grep -v '^$' | sort -u); do + awk '/=>/ {print $3}' | grep -v '^$' | LC_ALL=C sort -u); do defined="$defined $(nm -D --defined-only "$lib" 2>/dev/null | awk '{print $NF}')" done - defined="$(echo "$defined" | strip_ver | sort -u)" - missing="$(comm -23 <(echo "$undef") <(echo "$defined"))" + defined="$(echo "$defined" | strip_ver | LC_ALL=C sort -u)" + # comm requires ONE collation across both inputs and does not check. Both + # sorts above are pinned to C, so comm is pinned to C too -- inputs sorted one + # way and compared another is the same defect with an extra step (#552). + missing="$(LC_ALL=C comm -23 <(echo "$undef") <(echo "$defined"))" if [ -n "$missing" ]; then echo "rebuild: UNRESOLVED SYMBOLS against $PGVER:" >&2 echo "$missing" | head -20 >&2