test: automate compactor error paths, advanced flags, and exotic table name/bound coverage - #74
Conversation
…089/092/096/098/099/105/107/108/109/110/113/114/115/116/138 in journey.sh Coverage: - TC-043: iceberg_metadata() Parquet file probe - TC-058: pg_dump excludes S3/GCS/Azure credentials - TC-059: Lakekeeper rejects bad-credential warehouse request - TC-062: cold rows survive SeaweedFS container restart - TC-063: PG extensions and DuckDB secret reload after pg restart - TC-066/067/070: partition column types (timestamp no-tz, date, text rejected) - TC-087/089/092/096/098/099/105: extended type round-trips (timestamp, time, char, json, interval, oid, full-column-set) - TC-107/108/109: multi-table isolation (failure, list config, disable-one) - TC-110: idempotent re-register updates config, no duplicate row - TC-113/114: unsupported table types (UNLOGGED, non-existent) rejected at register - TC-115/116: LIST and HASH partition strategies rejected at archive time - TC-138: same table name in two schemas yields distinct Iceberg namespaces
…gged register behavior, partition PK inheritance, and bound-parse grep patterns
…t cascade failures
# Conflicts: # ci/journey.sh
…dge/ColdFront into test/automate-functional-tcs
…e name/bound coverage - oid column type rejected at table provisioning - missing required flag exits with usage message; nonexistent table exits with error - dry-run reports compaction groups; actual compaction merges files; no-op when already compact - snapshot expiry retains minimum count; expire-keep-files leaves data for orphan pass; orphan files cleaned up - all three maintenance steps combined in one invocation; custom target file size accepted - unreachable catalog endpoint rejected with error - concurrent compactors serialised by bakery without conflict - leading-underscore and over-long table names rejected at registration - mixed-case table name collision rejected at registration - table names with dot, hyphen, and space archive and cold-read correctly - BC-era and open-lower-bound partitions tier correctly - near-max, 32-bit overflow, and six-digit year bounds accepted by PostgreSQL and archiver
📝 WalkthroughWalkthroughThe CI journey script adds compactor error, advanced-option, unreachable-endpoint, concurrent-compaction, and quoted-name stories. It expands assertions for compaction, maintenance, partition bounds, source names, and case collisions. Tiered orchestration runs the new coverage. ChangesCI journey coverage
Estimated code review effort: 4 (Complex) | ~45 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Up to standards ✅🟢 Issues
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
ci/journey.sh (2)
4090-4103: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winProve the quoted-name rows are cold before asserting the cold read.
Lines 4098-4103 read each row through the view. A row that never tiered returns the same value from the hot partition, so TC-140/TC-143/TC-144 can pass without a cold read. Other stories assert
archived <partition>in the log for this reason (see lines 3449-3452). Add the same evidence here.♻️ Proposed addition
+ # Only the full export pipeline logs "archived", so this distinguishes a real + # cold export from a row still served by the hot partition. + assert_eq "TC-140/TC-143/TC-144: three quoted-name tables reached the cold tier" "3" \ + "$(q "$HOST" "SELECT count(*) FROM coldfront.tiered_views WHERE schema_name='qtn';")" + # Each cold row is readable through the unified tiered view. assert_eq "TC-140: cold row readable from qtn.\"my.table\"" "my.table" \ "$(q "$HOST" "SELECT val FROM qtn.\"my.table\" LIMIT 1;")"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci/journey.sh` around lines 4090 - 4103, Add explicit evidence that each quoted-name table was archived before the cold-read assertions, using the existing /tmp/journey-qtn.log and the established “archived <partition>” log-validation pattern. Update the TC-140, TC-143, and TC-144 flow around archive_only and the three assert_eq calls so each test verifies its corresponding archive event before validating the readable value.
3946-3960: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winBind TC-136 to the endpoint failure, and count the log check as a pass.
Two points:
/tmp/journey-badlk.yamlomits thearchiver:section that/tmp/journey-archiver.yamlcarries. Any non-zero exit satisfies the test, including a config-validation error that never reaches the unreachable endpoint. Assert that the error output names the connection failure.- Line 3959 records a failure only. On success it adds nothing to the tally. Use
assert_containsso both outcomes are counted.♻️ Proposed change
if "$COMPACTOR" --config /tmp/journey-badlk.yaml --table events \ >/dev/null 2>/tmp/journey-tc136.log; then fail "TC-136: expected non-zero exit with bad Lakekeeper endpoint, got 0" else pass "TC-136: unreachable Lakekeeper endpoint rejected (exit $?)" - [ -s /tmp/journey-tc136.log ] || fail "TC-136: no error output emitted" + # The failure must come from the endpoint, not from config validation. + assert_contains "TC-136: error names the unreachable endpoint" "127.0.0.1:19999" \ + "$(cat /tmp/journey-tc136.log)" fi🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci/journey.sh` around lines 3946 - 3960, Update the TC-136 test around the bad Lakekeeper endpoint to include the same required archiver configuration as /tmp/journey-archiver.yaml, then assert that /tmp/journey-tc136.log contains the expected connection-failure text so configuration errors cannot satisfy the test. Replace the standalone file-size check with assert_contains, ensuring the log assertion contributes to the pass/fail tally in both outcomes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@ci/journey.sh`:
- Around line 4090-4103: Add explicit evidence that each quoted-name table was
archived before the cold-read assertions, using the existing
/tmp/journey-qtn.log and the established “archived <partition>” log-validation
pattern. Update the TC-140, TC-143, and TC-144 flow around archive_only and the
three assert_eq calls so each test verifies its corresponding archive event
before validating the readable value.
- Around line 3946-3960: Update the TC-136 test around the bad Lakekeeper
endpoint to include the same required archiver configuration as
/tmp/journey-archiver.yaml, then assert that /tmp/journey-tc136.log contains the
expected connection-failure text so configuration errors cannot satisfy the
test. Replace the standalone file-size check with assert_contains, ensuring the
log assertion contributes to the pass/fail tally in both outcomes.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: b91a5665-60ab-45f7-8570-9d12c899d20d
📒 Files selected for processing (1)
ci/journey.sh
|
@imtiazqa Retitled TC-146/148/149 to assert the archiver left these partitions hot ( |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
ci/journey.sh (1)
3341-3344: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winAdd a hot-attachment assertion for the
events_wide(six-digit-year) partition.The fixture creates four future-dated partitions:
events_y2038,events_y9999,events_wide(FROM'10000-06-01'TO'20000-01-01'), andevents_y294276. The new assertions at lines 3372-3377 verifypg_inheritsattachment forevents_y9999(TC-146),events_y2038(TC-148), andevents_y294276(TC-149), but no assertion coversevents_wide.The PR objectives specifically call out "six-digit years" as an automated exotic-bound test case, and
events_wideis the only fixtured partition matching that description distinct fromevents_y294276. Without a dedicatedpg_inheritscheck, this partition's hot/cold classification is only implicitly covered by the generic "archiver completed without aborting" check at line 3350, which does not prove it stayed attached to the hot parent.🧪 Proposed fix to add the missing assertion
assert_eq "TC-149: year-294276 partition left hot, not archived" "1" \ "$(q "$HOST" "SELECT count(*) FROM pg_inherits i JOIN pg_class c ON c.oid=i.inhrelid WHERE i.inhparent=$hotp AND c.relname='events_y294276';")" + assert_eq "TC-XXX: year-20000 (six-digit-year) partition left hot, not archived" "1" \ + "$(q "$HOST" "SELECT count(*) FROM pg_inherits i JOIN pg_class c ON c.oid=i.inhrelid WHERE i.inhparent=$hotp AND c.relname='events_wide';")"Replace
TC-XXXwith the correct test-case identifier used elsewhere in this suite.Also applies to: 3366-3377
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci/journey.sh` around lines 3341 - 3344, Add a dedicated pg_inherits hot-attachment assertion for the events_wide partition alongside the existing events_y9999, events_y2038, and events_y294276 checks, using the suite’s correct test-case identifier instead of TC-XXX. Verify that events_wide remains attached to xb.events.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@ci/journey.sh`:
- Around line 3341-3344: Add a dedicated pg_inherits hot-attachment assertion
for the events_wide partition alongside the existing events_y9999, events_y2038,
and events_y294276 checks, using the suite’s correct test-case identifier
instead of TC-XXX. Verify that events_wide remains attached to xb.events.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 85d03119-9c3a-47dc-9662-43c78f165ec2
📒 Files selected for processing (1)
ci/journey.sh
2 participants
Automates 24 previously manually validated test cases in
ci/journey.shacross three areas: compactor CLI behaviour, exotic partition bounds, and
special-character table names. All cases pass on PG 16, 17, and 18 across
the full backend/mode matrix (verified in two independent CI runs).
Coverage
Compactor CLI
small files; subsequent dry-run confirms nothing left to compact
without removing data files; orphan pass then cleans up freed files
no conflicts
Table name validation
Quoted table names round-trip
cold rows correctly through the unified tiered view
Exotic partition bounds
accepted as partition bounds without aborting the archiver
Type rejection