`base.mk` has:
```make
Don't have installcheck bomb on error
.IGNORE: installcheck
installcheck: $(TEST_RESULT_FILES) $(TEST_SQL_FILES) | ...
test: $(TEST_DEPS)
@if [ -r $(TESTOUT)/regression.diffs ]; then cat $(TESTOUT)/regression.diffs; fi
```
`.IGNORE: installcheck` makes `make` treat `installcheck` as always successful
regardless of `pg_regress`'s actual exit status, and `test`'s recipe only
`cat`s `regression.diffs` for visibility — it never checks whether that file
is non-empty or whether `installcheck` "failed." So `make test` always exits
0, even when every single test fails.
This isn't just a local annoyance — it's silently defeating CI. Confirmed on
two real PRs in extension_tools (Postgres-Extensions/extension_tools#6 and #7):
every `pg_regress` test failed (`not ok 1`, `not ok 2`, `not ok 3`) in every
PG-version job's logs, yet every job and the overall check both reported
`"conclusion":"success"`. A completely broken extension merges with a green
checkmark.
The `verify-results` safeguard added in 2.0.0 doesn't help here — it only
guards `make results` (`results: verify-results`), preventing you from
blessing failing output as new expected output. It's never invoked by
`make test`/`installcheck` itself, so CI remains blind.
Suggest: make `test`/`installcheck` actually fail (exit non-zero) when
`regression.diffs` is non-empty or `pg_regress`'s own exit status is
non-zero, while still printing the diff for visibility. If `.IGNORE` is there
specifically to let `test`'s recipe run and print diffs even after a
failure, that's fine — just make the recipe itself `exit 1` afterward when a
real failure occurred, similar to how `verify-results`'s non-pgtap branch
already detects this.
`base.mk` has:
```make
Don't have installcheck bomb on error
.IGNORE: installcheck$(TEST_RESULT_FILES) $ (TEST_SQL_FILES) | ...
installcheck:
test: $(TEST_DEPS)$(TESTOUT)/regression.diffs ]; then cat $ (TESTOUT)/regression.diffs; fi
@if [ -r
```
`.IGNORE: installcheck` makes `make` treat `installcheck` as always successful
regardless of `pg_regress`'s actual exit status, and `test`'s recipe only
`cat`s `regression.diffs` for visibility — it never checks whether that file
is non-empty or whether `installcheck` "failed." So `make test` always exits
0, even when every single test fails.
This isn't just a local annoyance — it's silently defeating CI. Confirmed on
two real PRs in extension_tools (Postgres-Extensions/extension_tools#6 and #7):
every `pg_regress` test failed (`not ok 1`, `not ok 2`, `not ok 3`) in every
PG-version job's logs, yet every job and the overall check both reported
`"conclusion":"success"`. A completely broken extension merges with a green
checkmark.
The `verify-results` safeguard added in 2.0.0 doesn't help here — it only
guards `make results` (`results: verify-results`), preventing you from
blessing failing output as new expected output. It's never invoked by
`make test`/`installcheck` itself, so CI remains blind.
Suggest: make `test`/`installcheck` actually fail (exit non-zero) when
`regression.diffs` is non-empty or `pg_regress`'s own exit status is
non-zero, while still printing the diff for visibility. If `.IGNORE` is there
specifically to let `test`'s recipe run and print diffs even after a
failure, that's fine — just make the recipe itself `exit 1` afterward when a
real failure occurred, similar to how `verify-results`'s non-pgtap branch
already detects this.