test(coverage): XA (0→57%) + on-disk upgrade (0→14%); document db_repsite gap

test/coverage/DB-REPSITE-TODO.md

@@ -0,0 +1,102 @@
+# db_repsite — reconstruction TODO (repmgr 100-series enabler)
+
+**Status: NOT reconstructed in this coverage PR (deliberately timeboxed).**
+XA and on-disk-upgrade were the surer wins and shipped; `db_repsite` is the
+hardest of the three and is documented here for a focused follow-up.
+
+## What it is
+
+`db_repsite` is a small **C++ command-line test utility** that the repmgr
+100-series Tcl tests (`repmgr100`–`repmgr112`) drive as one or more child
+processes over a stdin/stdout pipe. Each `db_repsite` process is one repmgr
+*site* (a separate OS process sharing an env home), which is exactly the
+multi-process repmgr topology the in-process `repmgrNN` (9–34) harness cannot
+exercise. Restoring it unlocks ~12 tests and the multi-process repmgr paths.
+
+- Referenced by build: `dist/Makefile.in` lines 1209, 1380–1383
+  (`db_repsite@o@: $(testdir)/repmgr/db_repsite.cpp`,
+  `DBREPSITE_OBJS`, `db_repsite: $(DBREPSITE_OBJS) $(DEF_LIB_CXX)`).
+- Expected source path: **`test/repmgr/db_repsite.cpp`** (the `$(testdir)` in
+  Makefile.in is `test`). **This file is absent from this fork** — only the
+  Windows project stub `build_windows/VS10/db_repsite.vcxproj` survives, and it
+  is *not* in git history as a `.cpp`.
+- Tcl entry points: `setup_site_prog` / `open_site_prog` in
+  `test/tcl/reputils.tcl` (~line 2585). `setup_site_prog` looks for
+  `$util_path/db_repsite` and `error "Skipping: ..."` if absent — which is why
+  the 100-series currently skip.
+
+## Command protocol (reverse-engineered from the tests)
+
+`db_repsite` reads one command per line from stdin, line-buffered
+(`fconfigure $s -buffering line`), and writes sentinels to stdout that the
+tests read with `gets`. Full verb set observed across
+`repmgr100.tcl … repmgr112.tcl`:
+
+| command | args | action | stdout expectation |
+|---------|------|--------|--------------------|
+| `home DIR` | env home path | remember home for `open_env` | (none) |
+| `local PORT` | port | set this site's local address (127.0.0.1:PORT) | (none) |
+| `remote HOST PORT` | | add a remote site (helper) | (none) |
+| `output FILE` | | redirect env `set_errfile`/verbose to FILE | (none) |
+| `open_env` | | `db_env_create` + `DB_ENV->open` with repmgr flags | (none) |
+| `start master` / `start client` | role | `DB_ENV->repmgr_start(nthreads, role)` | line matching `*Successful*` |
+| `open_db NAME` | db name | `db_create` + `DB->open` (btree, auto-commit) | (none) |
+| `put KEY VALUE` | | `DB->put` (auto-commit) | (none) |
+| `is_connected PORT` | | query repmgr site status; print connected/not | line the test matches (see repmgr101/112) |
+| `echo TOKEN` | any token | write `TOKEN\n` back — the test's sync sentinel | echoes TOKEN verbatim |
+| `exit` | | close db, close env, exit 0 | (none) |
+
+Notes:
+- `start` is the only command `open_site_prog` waits on (`gets $s` after
+  `start`), expecting a line containing `Successful`. Everything else is
+  fire-and-forget until an explicit `echo`.
+- Site config uses **DB_CONFIG**, written by `make_dbconfig` (repmgr_site /
+  rep_set_config lines), so `db_repsite` itself does NOT parse repmgr_site
+  from stdin for the local/bootstrap sites in most tests — it just opens the
+  env and the DB_CONFIG supplies the topology. `local`/`remote` stdin verbs
+  are used by the few tests that set the address programmatically
+  (`repmgr101`, `repmgr105`, `repmgr112`).
+
+## API surface it must call
+
+All public, all present in this fork's `db.h`:
+
+- `db_env_create`, `DB_ENV->set_errfile`, `DB_ENV->set_verbose`,
+  `DB_ENV->open` with `DB_CREATE|DB_INIT_REP|DB_INIT_LOCK|DB_INIT_LOG|
+  DB_INIT_MPOOL|DB_INIT_TXN|DB_THREAD|DB_RECOVER`.
+- `DB_ENV->repmgr_site` (add local/remote site, set DB_LOCAL_SITE /
+  DB_BOOTSTRAP_HELPER), `DB_REPMGR_SITE->set_config`, `->close`.
+- `DB_ENV->repmgr_start(dbenv, nthreads, DB_REP_MASTER|DB_REP_CLIENT)`.
+- `DB_ENV->repmgr_site_list` / `DB_ENV->repmgr_stat` for `is_connected`.
+- `db_create`, `DB->open`, `DB->put`, `DB->close`, `DB_ENV->close`.
+
+The canonical upstream implementation is Oracle BDB 5.3's
+`test/repmgr/db_repsite.cpp` (~250–300 lines). It is a straightforward
+stdin-command dispatcher; reconstruction is *tractable* but must be validated
+against the exact stdout sentinels the tests `gets`, and — critically — run
+under a **per-test timeout**, because a mis-handled connection/sync will hang
+the pipe (the tests block on `gets`).
+
+## Steps to finish (follow-up PR)
+
+1. Write `test/repmgr/db_repsite.cpp` implementing the verb table above.
+   Model the dispatch loop on the existing `test/xa/xa_direct.c` style
+   (assert-and-report), but C++ and long-lived (loop on `fgets`).
+2. It already has a Makefile.in target (`db_repsite`); confirm `--enable-test`
+   builds it (needs `--enable-cxx` / `DEF_LIB_CXX`). Add to the coverage
+   `make` step if missing.
+3. Run `repmgr100` first (simplest: 2-site master/client) under
+   `timeout 300 tclsh8.6 -c 'source ../test/tcl/test.tcl;
+   source ../test/tcl/reputils.tcl; repmgr100'`. Iterate on sentinels until it
+   passes without hanging. Then 101, 102, 105, 106.
+4. Register a `run_repmgr_100` set behind a `COV_REPMGR100=1` flag in
+   `test/coverage/run_coverage.sh` (per-test tclsh + timeout + TESTDIR clean,
+   mirroring the existing `COV_REP` block), NOT in the default subset (these
+   are multi-process and can hang — always timeout-guard).
+
+## Why deferred here
+
+Reconstructing a multi-process repmgr CLI and making 100-series tests pass
+*reliably and without hanging* is materially larger and riskier than the XA
+and upgrade wins, which were shippable and measurable in-budget. This doc
+captures the entire API + protocol so the follow-up is mechanical.

test/coverage/README.md

@@ -34,6 +34,7 @@ Knobs (all optional env vars):
 | `TCL_LIB`    | autodetected (nix store or `/usr/lib/tcl8.6`) | Tcl lib dir for `--with-tcl` |
 | `COV_TIMEOUT`| `2400`                                    | seconds ceiling for the test run |
 | `COV_REP`    | `0`                                       | set `1` to also run the replication (rep/repmgr) tests |
+| `COV_XA_UPG` | `1`                                       | run the XA + on-disk-upgrade drivers (fast, non-hanging) |
 
 Set `COV_REP=1` to add the replication suite (biggest cold surface: rep/ +
 repmgr/ ~= 12.4k lines at 0.8%). It moves them to ~56% line / ~42% branch. Those
@@ -42,6 +43,27 @@ few election/lease tests hang; see `REPLICATION-COVERAGE.md` for the exact set,
 the measured lift, and what still needs real multi-process orchestration
 (`rep*script.tcl` subprocess tests, repmgr 100-series needing `db_repsite`).
 
+`COV_XA_UPG` (on by default) runs two non-Tcl drivers that light up subsystems
+the Tcl suite never reaches. Both self-clean their home dir and run under a hard
+timeout, so they cannot hang:
+
+- **XA** — `test/xa/run_xa_direct.sh` builds `test/xa/xa_direct.c`, a
+  **Tuxedo-free** transaction manager that drives `db_xa_switch`
+  (`src/xa/xa.c` + `xa_map.c`) directly: `xa_open/start/end/prepare/commit/
+  rollback/recover/forget` plus the internal two-phase-commit path and the XA
+  recovery scan (`__txn_get_prepared`, DB_FIRST/DB_NEXT). The full `chk.xa`
+  harness needs an Oracle Tuxedo install (`atmi.h`, `tmboot`, …) which is not
+  available; this driver needs none. Lift: `xa.c` **0%→~57%**, `xa_map.c`
+  **0%→~78%**.
+- **On-disk upgrade** — `test/db/run_upgrade.sh` runs the `db_upgrade` utility
+  over the one committed old-format fixture (`test/csharp/bdb4.7.db`, a btree
+  meta version-9 db) and verifies the result. Lift: `db_upg.c` **0%→~14%**
+  (the utility open + version-check + no-op-when-current path). `db_upg_opd.c`
+  stays 0%: the off-page-duplicate conversion needs a genuinely old
+  (pre-version-9) db with off-page dups, and the full Tcl `upgrade` group's
+  per-version fixture tree (`test/tcl/upgrade/databases/`) is **not present in
+  this fork**. See `DB-REPSITE-TODO.md` for the analogous repmgr gap.
+
 Outputs land in `build_unix/` (gitignored):
 
 - `coverage-summary.txt` — the line/branch/function totals

test/coverage/run_coverage.sh

@@ -200,6 +200,35 @@ if [ "${COV_REP:-0}" = 1 ]; then
   echo "  .gcda files after rep: $(find . -name '*.gcda' | wc -l)"
 fi
 
+# --- optional XA + on-disk-upgrade drivers (COV_XA_UPG=1, default on) --------
+# These are NOT Tcl tests: they are standalone drivers that light up two cold
+# subsystems the Tcl suite never reaches.
+#   xa/xa.c + xa_map.c  -- the X/Open XA resource-manager switch (db_xa_switch).
+#     test/xa/run_xa_direct.sh compiles test/xa/xa_direct.c (a Tuxedo-free TM
+#     that drives the switch entry points + the internal 2PC / recovery path)
+#     and runs it under a timeout.  Lifts xa.c 0%->~57%, xa_map.c 0%->~78%.
+#   db/db_upg.c         -- the on-disk-format upgrade path.
+#     test/db/run_upgrade.sh runs the db_upgrade utility over the committed
+#     old-format fixture (test/csharp/bdb4.7.db).  Lifts db_upg.c 0%->~14%.
+#     (db_upg_opd.c stays cold: needs an old off-page-duplicate fixture that is
+#      not present in this fork -- see test/coverage/README.md.)
+# Both self-clean their home dir and run under a hard timeout, so they cannot
+# hang.  Set COV_XA_UPG=0 to skip.
+if [ "${COV_XA_UPG:-1}" = 1 ]; then
+  echo "== run XA + upgrade drivers (COV_XA_UPG=1) =="
+  if sh "$root/test/xa/run_xa_direct.sh" >/tmp/cov-xa.log 2>&1; then
+    echo "PASS xa_direct"
+  else
+    echo "FAIL xa_direct (rc=$?)"; tail -5 /tmp/cov-xa.log
+  fi
+  if sh "$root/test/db/run_upgrade.sh" >/tmp/cov-upg.log 2>&1; then
+    echo "PASS db_upgrade"
+  else
+    echo "FAIL db_upgrade (rc=$?)"; tail -5 /tmp/cov-upg.log
+  fi
+  echo "  .gcda files after xa/upg: $(find . -name '*.gcda' | wc -l)"
+fi
+
 # --- aggregate ---------------------------------------------------------------
 echo "== capture (lcov) =="
 # NOTE: capture from .libs (not .) -- libtool double-compiles, and only the

test/db/run_upgrade.sh

@@ -0,0 +1,46 @@
+#!/bin/sh -
+#
+# $Id$
+#
+# run_upgrade.sh --
+#	Exercise the on-disk-format upgrade path (src/db/db_upg.c,
+#	db_upg_opd.c) by running the db_upgrade utility + DB->upgrade over an
+#	old-format Berkeley DB file.
+#
+#	The full Tcl "upgrade" group (test/tcl/upgrade.tcl) needs a large
+#	per-version/per-method fixture tree under test/tcl/upgrade/databases/
+#	that is NOT present in this fork.  This script instead uses the one
+#	committed old-format fixture (test/csharp/bdb4.7.db, a btree meta
+#	version-9 database) to drive the utility's open/version-check path and
+#	DB->upgrade's no-op-when-current branch, then verifies the result.
+#
+# Usage (from build_unix):
+#	sh ../test/db/run_upgrade.sh
+#
+# Exits non-zero on failure.  Runs under a timeout so it cannot hang.
+
+set -e
+
+BUILD=${BUILD:-.}
+FIXTURE=${FIXTURE:-../test/csharp/bdb4.7.db}
+WORK=${WORK:-UPGTEST}
+TIMEOUT=${TIMEOUT:-60}
+
+[ -f "$FIXTURE" ] || { echo "FAIL: fixture $FIXTURE not found"; exit 1; }
+[ -x "$BUILD/db_upgrade" ] || { echo "FAIL: $BUILD/db_upgrade missing"; exit 1; }
+
+ABS_UPGRADE=$(cd "$BUILD" && pwd)/db_upgrade
+ABS_VERIFY=$(cd "$BUILD" && pwd)/db_verify
+ABS_FIXTURE=$(cd "$(dirname "$FIXTURE")" && pwd)/$(basename "$FIXTURE")
+
+rm -f "$WORK"/__db.* "$WORK"/log.* "$WORK"/*.db 2>/dev/null || true
+mkdir -p "$WORK"
+cp "$ABS_FIXTURE" "$WORK/t.db"
+
+echo "db_upgrade on $(basename "$FIXTURE")"
+( cd "$WORK" && timeout "$TIMEOUT" "$ABS_UPGRADE" -h . t.db )
+echo "db_verify on upgraded file"
+( cd "$WORK" && timeout "$TIMEOUT" "$ABS_VERIFY" t.db )
+
+echo "run_upgrade.sh: PASS"
+exit 0

test/xa/run_xa_direct.sh

@@ -0,0 +1,50 @@
+#!/bin/sh -
+#
+# $Id$
+#
+# run_xa_direct.sh --
+#	Build and run the Tuxedo-free XA driver (xa_direct.c), which exercises
+#	Berkeley DB's db_xa_switch (src/xa/xa.c + xa_map.c) and the internal
+#	two-phase-commit / recovery path that XA drives.  Unlike chk.xa this
+#	needs NO Tuxedo install.
+#
+# Usage (from build_unix):
+#	sh ../test/xa/run_xa_direct.sh
+#
+# It compiles xa_direct against the just-built libdb in ./.libs, runs it in a
+# guaranteed-clean home under a timeout, and reports PASS/FAIL.  Exits non-zero
+# on failure or hang.
+
+set -e
+
+BUILD=${BUILD:-.}			# build_unix dir (cwd by default)
+SRC=${SRC:-../test/xa/xa_direct.c}
+HOME_DIR=${HOME_DIR:-XA_TESTDIR}
+TIMEOUT=${TIMEOUT:-90}
+
+LIB="$BUILD/.libs/libdb-5.3.so"
+if [ ! -f "$LIB" ]; then
+	# Fall back to the versioned .so actually present.
+	LIB=$(ls "$BUILD"/.libs/libdb-*.so 2>/dev/null | head -1)
+fi
+[ -n "$LIB" ] || { echo "FAIL: libdb .so not found in $BUILD/.libs"; exit 1; }
+
+echo "Compiling xa_direct against $LIB"
+gcc -g -O1 ${CFLAGS:-} -I"$BUILD" -I../src/dbinc "$SRC" "$LIB" \
+    -luring -lpthread -Wl,-rpath,"$(cd "$BUILD/.libs" && pwd)" \
+    -o "$BUILD/xa_direct"
+
+# Guaranteed-clean home: remove env/log/db artifacts (not rm -rf).
+rm -f "$HOME_DIR"/__db.* "$HOME_DIR"/log.* "$HOME_DIR"/*.db \
+    "$HOME_DIR"/DB_CONFIG 2>/dev/null || true
+mkdir -p "$HOME_DIR"
+
+echo "Running xa_direct (timeout ${TIMEOUT}s)"
+if timeout "$TIMEOUT" "$BUILD/xa_direct"; then
+	echo "run_xa_direct.sh: PASS"
+	exit 0
+else
+	rc=$?
+	echo "run_xa_direct.sh: FAIL (rc=$rc)"
+	exit $rc
+fi