Skip to content

test: adopt upstream bats-assert and cover the apache hardening - #51

Merged
zebby76 merged 2 commits into
Smals-Webtech:mainfrom
zebby76:test/adopt-upstream-bats-assert
Sep 3, 2026
Merged

test: adopt upstream bats-assert and cover the apache hardening#51
zebby76 merged 2 commits into
Smals-Webtech:mainfrom
zebby76:test/adopt-upstream-bats-assert

Conversation

@zebby76

@zebby76 zebby76 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Two commits: the library swap, then the assertions it makes possible.

Why the vendored helpers had to go

test/lib held 857 lines copied from bats-assert at some point, with no version and no
provenance — and the copy had drifted. It parses the short options only, and an unrecognised
long option falls through to the argument branch. So this, written while covering the apache
hardening:

refute_output --partial "<address>"

asserted that the output does not contain the literal string --partial — true of everything. It
passed against an apache image that does emit the signature.

Nothing else in the suites used a long form, so that was the only assertion affected. The shape
of the failure is the point: a silently tautological assertion is worse than an absent one.

What replaces them

Upstream at pinned tags — bats-support 0.3.0, bats-assert 2.1.0 — resolved through
BATS_LIB_PATH:

  • locally, make -C test deps clones them into test/lib, now untracked;
  • in CI, bats-core/bats-action installs the same tags and the path is handed to the suites,
    since the action exposes it as a step output rather than exporting it.

The upstream API differs from the copy, so all 60 call sites move:

vendored upstream
assert_output -l "X" assert_line "X"
assert_output -l -r "X" assert_line --regexp "X"
assert_output -l 0 "X" assert_line -n 0 "X"
refute_output -l "X" refute_line "X"
refute_output -p "X" refute_output --partial "X"

test/helpers/containers.bash was the easy one to miss — it asserts too, and
assert_output -l "healthy" under the upstream helper compares the output to the string -l.
The image suites caught it on the next run, which is the argument for this change in one line.

The deps target compares the checked-out tag rather than the directory, so it is idempotent when
current and re-fetches after a version bump. It uses FORCE rather than .PHONY: make skips the
implicit rule search for a phony target, so a pattern rule declared phony never fires.

The assertions

The dotfile assertions were guarded to nginx because apache had not been hardened yet. #49 landed,
both images answer 404 for the same requests, so the guard goes and they cover the pair. Three
more cover what was specific to apache — a browsable document root, TRACE, and the server
signature in the error page footer.

Against the published 8.5.9 apache image, which carries none of the fixes:

12 dotfiles are not served                    not ok
13 file inside a dot directory is not served  not ok
14 /.well-known keeps its normal handling     ok
15 the document root is not browsable         not ok
16 TRACE is refused                           not ok
17 error pages carry no server signature      not ok

Verified

nginx  prd   17/17
apache prd   17/17   (three skipped: no separate monitoring port)
cli    prd    4/4
demo suites  parse: 19, 12, 6 and 1 tests
published 8.5.9-apache   5 failures — the suite stays honest

AGENTS.md gains the two test tiers, make -C test deps, and a correction: the suites no longer
assume floating :<variant> tags, and snapshot-<variant> is the same tag on both branches, so a
rebuild is needed after switching.

test/lib held 857 lines copied from bats-assert at some point, with no version
and no provenance, and the copy had drifted: it parses the short options only.
An unrecognised long option falls through to the argument branch, so

  refute_output --partial "<address>"

asserted that the output does not contain the literal string "--partial", which
is true of everything. Written against an apache image that does emit the
signature, it passed. Nothing else in the suites used a long form, so this was
the only assertion affected, but the shape of the failure is the point: a
silently tautological assertion is worse than an absent one.

The libraries now come from upstream at pinned tags -- bats-support 0.3.0,
bats-assert 2.1.0 -- resolved through BATS_LIB_PATH. `make -C test deps` clones
them into test/lib, which is untracked; the CI job installs the same tags
through bats-core/bats-action and hands the path to the suites, since the action
exposes it as a step output rather than exporting it.

The upstream API differs from the copy, so all 60 call sites move:

  assert_output -l "X"        assert_line "X"
  assert_output -l -r "X"     assert_line --regexp "X"
  assert_output -l 0 "X"      assert_line -n 0 "X"
  refute_output -l "X"        refute_line "X"
  refute_output -p "X"        refute_output --partial "X"

test/helpers/containers.bash was the easy one to miss -- it asserts too, and
`assert_output -l "healthy"` under the upstream helper compares the output to
the string "-l". The image suites caught it immediately, which is the argument
for the change in one line.

The deps target compares the checked-out tag rather than the directory, so it is
idempotent when current and re-fetches after a version bump here. It uses FORCE
rather than .PHONY: make skips the implicit rule search for a phony target, so a
pattern rule declared phony never fires.

Verified on images built from this branch: nginx 17/17, apache 17/17, cli 4/4,
and the demo suites still parse (19, 12, 6 and 1 tests).
… variants

The dotfile assertions were guarded to the nginx variant because apache had not
been hardened yet. It has now, and both images answer 404 for the same requests,
so the guard goes and the three assertions cover the pair.

Three more cover what was specific to apache: a browsable document root, TRACE,
and the server signature in the error page footer.

Pointed at the published 8.5.9 apache image, which carries none of the fixes:

  12 dotfiles are not served                    not ok
  13 file inside a dot directory is not served  not ok
  14 /.well-known keeps its normal handling     ok
  15 the document root is not browsable         not ok
  16 TRACE is refused                           not ok
  17 error pages carry no server signature      not ok

Five failures against the published image, seventeen passes against one built
from this branch, on both variants -- apache skipping the three that describe
the nginx monitoring port, which it does not have.
@zebby76
zebby76 merged commit 04109c3 into Smals-Webtech:main Sep 3, 2026
19 checks passed
zebby76 added a commit that referenced this pull request Sep 3, 2026
…#54)

* test: replace the vendored assertion helpers with upstream bats-assert

test/lib held 857 lines copied from bats-assert at some point, with no version
and no provenance, and the copy had drifted: it parses the short options only.
An unrecognised long option falls through to the argument branch, so

  refute_output --partial "<address>"

asserted that the output does not contain the literal string "--partial", which
is true of everything. Written against an apache image that does emit the
signature, it passed. Nothing else in the suites used a long form, so this was
the only assertion affected, but the shape of the failure is the point: a
silently tautological assertion is worse than an absent one.

The libraries now come from upstream at pinned tags -- bats-support 0.3.0,
bats-assert 2.1.0 -- resolved through BATS_LIB_PATH. `make -C test deps` clones
them into test/lib, which is untracked; the CI job installs the same tags
through bats-core/bats-action and hands the path to the suites, since the action
exposes it as a step output rather than exporting it.

The upstream API differs from the copy, so all 60 call sites move:

  assert_output -l "X"        assert_line "X"
  assert_output -l -r "X"     assert_line --regexp "X"
  assert_output -l 0 "X"      assert_line -n 0 "X"
  refute_output -l "X"        refute_line "X"
  refute_output -p "X"        refute_output --partial "X"

test/helpers/containers.bash was the easy one to miss -- it asserts too, and
`assert_output -l "healthy"` under the upstream helper compares the output to
the string "-l". The image suites caught it immediately, which is the argument
for the change in one line.

The deps target compares the checked-out tag rather than the directory, so it is
idempotent when current and re-fetches after a version bump here. It uses FORCE
rather than .PHONY: make skips the implicit rule search for a phony target, so a
pattern rule declared phony never fires.

Verified on images built from this branch: nginx 17/17, apache 17/17, cli 4/4,
and the demo suites still parse (19, 12, 6 and 1 tests).

* test: cover the apache hardening and assert the dotfile rules on both variants

The dotfile assertions were guarded to the nginx variant because apache had not
been hardened yet. It has now, and both images answer 404 for the same requests,
so the guard goes and the three assertions cover the pair.

Three more cover what was specific to apache: a browsable document root, TRACE,
and the server signature in the error page footer.

Pointed at the published 8.5.9 apache image, which carries none of the fixes:

  12 dotfiles are not served                    not ok
  13 file inside a dot directory is not served  not ok
  14 /.well-known keeps its normal handling     ok
  15 the document root is not browsable         not ok
  16 TRACE is refused                           not ok
  17 error pages carry no server signature      not ok

Five failures against the published image, seventeen passes against one built
from this branch, on both variants -- apache skipping the three that describe
the nginx monitoring port, which it does not have.
@zebby76
zebby76 deleted the test/adopt-upstream-bats-assert branch September 4, 2026 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant