testing: use Hurl in CI to test Caddy against spec

[mohammed90]

Since #5704 was posted, we've been on-and-off brainstorming how to approach testing of a web server. We sorta agreed that declarative approach is desired but weren't aware of any tools that'd facilitate the declarative approach nor had a concrete plan. We just knew we need solid tests.

I have recently come across Hurl (https://github.com/Orange-OpenSource/hurl) and was curious if it meets our needs. It is declarative. It makes HTTP calls. It stands on shoulders of The Giant®, namely curl. The PoC presented in this branch seems to work. In fact, the PR #6249 is a fix for a bug found while building this PoC.

This PR is to discuss the approach and to collaboratively add the tests. The core idea is simple:

HTTP handlers claim to conform to a particular behavior. The behavior can be specified through a collection of Hurl tests, i.e. a collection of HTTP requests and responses. The Hurl file defines the Spec the handler shall meet.

TODO:

• Agree on the structure/placement of the spec files
• Write the spec files for all the existing handlers by inspecting the docs and the code

For TODO number 2, code coverage is a helpful tool. There's a way to extract execution coverage of the hurl tests†, but I haven't found a neat way to present it on GitHub PRs/Actions.

Based on the work done to resolve #5849 and the existence of the project REDbot, we can translate those expectations and rules into Hurl files.

† Using this article as guide: Build Caddy with coverage instrumentation using go build -cover. Run Caddy using the command GOCOVERDIR=./coverdir caddy run, then run the Hurl tests. Stop Caddy with either caddy stop or Ctrl-C. Run go tool covdata textfmt -i=coverdir -o profile.txt. Run go tool cover -html profile.txt. An HTML page is opened in the browser with each file annotated by color for whether it was executed or not.

[mohammed90]

Example of how it looks in the Actions run summary:

Example of how failure is presented:

Example of how success is presented:

[github-actions]

Test Results

33 tests   32 ✅  11s ⏱️
33 suites   0 💤
 1 files     1 ❌

For more details on these failures, see this check.

Results for commit 6b2eb81.

♻️ This comment has been updated with latest results.

[mholt]

Ooo, I like where this is going! Will revisit this after 2.8.

[dkarlovi]

@mohammed90 this looks great! \o/

I'm only wondering one thing:

we can translate those expectations and rules into Hurl files

Since the expectations are basically "the HTTP protocol" (and related stuff), would Caddy actually be the right place to do this? RedBot's value here IMO is exactly that it already knows what requests need to be done and what assertions need to be done against them, converting that into Hurl-based specs seems like it would be a sizeable project with a sizeable deliverable.

Would creating this "Hurl-based HTTP spec test suite" be better as a standalone project which Caddy (and others, hopefully) can take advantage of and, hopefully, maintain?

[mohammed90]

Since the expectations are basically "the HTTP protocol" (and related stuff), would Caddy actually be the right place to do this? RedBot's value here IMO is exactly that it already knows what requests need to be done and what assertions need to be done against them, converting that into Hurl-based specs seems like it would be a sizeable project with a sizeable deliverable.

Caddy can be the right place :) we're aiming to test Caddy conformance to the spec. I skimmed the REDbot repo, and it isn't too complex. Integrating REDbot itself into the CI pipeline might be more of a hassle to maintain. Translating the behavior into Hurl files makes the expectations easier to understand and poke.

Would creating this "Hurl-based HTTP spec test suite" be better as a standalone project which Caddy (and others, hopefully) can take advantage of and, hopefully, maintain?

Perhaps, but maintaining such project is beyond my capacity. I can't initiate and commit to it (my personal backlog is too long already). I may help if it's maintained by a group.

[mohammed90]

The intercept and redir cases are unpushed yet. With them, here's the coverage growth:

== TOP COVERAGE GAINS (packages exercised more by new specs) ==
   delta   base%  full%   +stmt  total  package
  +77.42    4.30  81.72     +72     93  github.com/caddyserver/caddy/v2/modules/caddyhttp/map
  +71.84    3.88  75.73     +74    103  github.com/caddyserver/caddy/v2/modules/caddyhttp/tracing
  +61.67    3.33  65.00     +74    120  github.com/caddyserver/caddy/v2/modules/caddyhttp/intercept
  +60.40    3.96  64.36     +61    101  github.com/caddyserver/caddy/v2/modules/caddyhttp/push
  +58.19    3.45  61.64    +135    232  github.com/caddyserver/caddy/v2/modules/caddyhttp/encode
  +55.93    6.78  62.71     +33     59  github.com/caddyserver/caddy/v2/modules/caddyhttp/logging
  +45.95   35.14  81.08     +17     37  github.com/caddyserver/caddy/v2/modules/metrics
  +33.33   22.22  55.56      +9     27  github.com/caddyserver/caddy/v2/modules/caddyhttp/encode/gzip
  +29.09   10.91  40.00     +16     55  github.com/caddyserver/caddy/v2/modules/caddyhttp/encode/zstd
  +21.38   18.77  40.15    +205    959  github.com/caddyserver/caddy/v2/modules/caddyhttp/fileserver
  +16.91    1.15  18.05     +59    349  github.com/caddyserver/caddy/v2/modules/caddyhttp/templates
   +9.09   36.56  45.65    +300   3299  github.com/caddyserver/caddy/v2/modules/caddyhttp
   +7.34   35.47  42.81    +211   2873  github.com/caddyserver/caddy/v2/caddyconfig/httpcaddyfile
   +7.06   59.04  66.10     +25    354  github.com/caddyserver/caddy/v2/modules/caddyhttp/rewrite
   +6.25   42.29  48.54     +64   1024  github.com/caddyserver/caddy/v2/caddyconfig/caddyfile
   +5.47   41.80  47.27     +14    256  github.com/caddyserver/caddy/v2/modules/caddyhttp/headers
   +4.93   19.65  24.58    +154   3125  github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy
   +3.05    5.01   8.06     +31   1017  github.com/caddyserver/caddy/v2/modules/logging
   +2.78   48.52  51.30     +61   2195  github.com/caddyserver/caddy/v2

== NEW PACKAGES EXERCISED (0% -> >0%) ==

Total new packages exercised: 0

OVERALL: baseline 6196/22546 = 27.48% | full 7814/22546 = 34.66%
Delta:   +1618 statements covered, +7.18 percentage points

[mohammed90]

Summary by copilot:

Caddyfile HTTP directives — spec coverage

Directive	Registered in	Spec dir	Status
abort	builtins	abort/	Covered
acme_server	caddypki/acmeserver	—	Not covered (needs ACME client; integration test exists)
basic_auth / basicauth	caddyauth	basicauth/	Covered
bind	builtins	—	Not covered (listener-binding, not request-observable)
copy_response	reverseproxy	copy_response/	Covered
copy_response_headers	reverseproxy	copy_response/	Covered (in same spec)
encode	encode	encode/	Covered
error	builtins	error/	Covered
file_server	fileserver	file_server/	Covered
forward_auth	reverseproxy/forwardauth	forward_auth/	Covered
fs	builtins	fs/	Covered
handle	builtins	handle/	Covered
handle_errors	builtins	handle_errors/	Covered
handle_path	rewrite	handle_path/	Covered
header	headers	headers/	Covered
intercept	intercept	intercept/	Covered
invoke	builtins	invoke/	Covered
log	builtins	log/	Covered
log_append	logging	log/	Covered (in same spec)
log_name	builtins	log/	Covered (in same spec)
log_skip / skip_log	builtins	log/	Covered (in same spec)
map	map	map/	Covered
method	rewrite	method/	Covered
metrics	metrics	metrics/	Covered
php_fastcgi	reverseproxy/fastcgi	—	Not covered (requires PHP-FPM sidecar)
push	push	push/	Covered
redir	builtins	redir/	Covered
request_body	requestbody	requestbody/	Covered
request_header	headers	request_header/	Covered
respond	builtins	static_response/	Covered
reverse_proxy	reverseproxy	reverse_proxy/	Covered
rewrite	rewrite	rewrite/	Covered
root	builtins	root/	Covered
route	builtins	route/	Covered
templates	templates	templates/	Covered
tls	builtins	—	Not covered (exercised indirectly via every HTTPS request; deep coverage lives in acme_test.go)
try_files	fileserver	try_files/	Covered
uri	rewrite	uri/	Covered
vars	builtins	vars/	Covered

Summary: 36 of 40 registered directives have spec coverage. The 4 uncovered (bind, tls, acme_server, php_fastcgi) are either not request-observable or require external sidecars that don't fit the Hurl model — they belong in the Go integration suite under integration.