[Introducing a testing framework similar to test::nginx] burl: A simple but flexible HTTP/3 testing framework based on bash and curl

[kingluo]

As we all know, test-nginx is a great testing framework, but no offense, in my daily use, I think it is still a bit far from perfect.

The expressive capabilities of test::nginx are limited, even though it already supports sending requests using curl, because it is not a scripting system (Of course, you can extend it using Perl modules, but due to the limitations of the framework design, this is a difficult task for most people).

1. It assumes your input is a text string, you cannot use other formats such as protobuf, you cannot compress or encrypt text.
2. The only output post-processing supported is regular matching, so you cannot parse and verify other forms of content, such as decryption.
3. You cannot describe complex test procedures, such as sending three requests to trigger the requset quota and verify the rate limiting results.
4. You cannot make other types of requests such as GRPC, SOAP.

In my practice, when I have such needs that exceed test::nginx's capabilities, I have to delegate them to the lua part, which is very troublesome.

For example, when I need to parse and verify the correctness of the CAS Auth plugin, it involves a lot of client-side CAS protocol details, so you have to implement the entire logic in the lua part, which increases the difficulty of maintaining this mix. As we all know, neither Perl nor Lua are popular languages. ;-(

You can see the test case below involves additional lua lib development to test CAS. And, to accommodate test::nginx's lack of response parsing, you have to do the conversion in the lua environment. Quite annoying, at least to me, because I've written a lot of test cases like this over the years.

=== TEST 2: login and logout ok
--- config
    location /t {
        content_by_lua_block {
            local http = require "resty.http"
            local httpc = http.new()
            -- Test-specific CAS lua lib
            local kc = require "lib.keycloak_cas"

            local path = "/uri"
            local uri = "http://127.0.0.1:" .. ngx.var.server_port
            local username = "test"
            local password = "test"

            local res, err, cas_cookie, keycloak_cookie = kc.login_keycloak(uri .. path, username, password)
            if err or res.headers['Location'] ~= path then
                ngx.log(ngx.ERR, err)
                ngx.exit(500)
            end
            res, err = httpc:request_uri(uri .. res.headers['Location'], {
                method = "GET",
                headers = {
                    ["Cookie"] = cas_cookie
                }
            })
            assert(res.status == 200)
            ngx.say(res.body)

            res, err = kc.logout_keycloak(uri .. "/logout", cas_cookie, keycloak_cookie)
            assert(res.status == 200)
        }
    }
--- response_body_like
uri: /uri
cookie: .*
host: 127.0.0.1:1984
user-agent: .*
x-real-ip: 127.0.0.1

So I was thinking, is there a simple testing framework that can meet the following needs?

1. No DSL, just simple shell scripts
2. Easy to extend and programming language agnostic
3. Can be used to initialize and test any server, not limited to nginx, such as envoy

I've created a new HTTP/3 testing framework based on bash and curl:

https://github.com/kingluo/burl

Synopsis:

#!/usr/bin/env burl

# Optional initialization here...
# Before all test cases are executed.
# For example, render nginx.conf and start nginx.
SET NGX_CONF_HTTP <<EOF
upstream test_backend {
    server $(dig +short nghttp2.org):443;

    keepalive 320;
    keepalive_requests 1000;
    keepalive_timeout 60s;
}
EOF

SET NGX_CONF <<'EOF'
location / {
    add_header Alt-Svc 'h3=":443"; ma=86400';
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host "nghttp2.org";
    proxy_pass https://test_backend;
}
EOF

START_NGX

TEST 1: test case

# Send request
# REQ is a curl wrapper so you can apply any curl options to suit your needs.
# Check https://curl.se/docs/manpage.html for details.
REQ /httpbin/anything --http3 -d foo=bar -d hello=world

# Validate the response headers
# HEADER is a grep wrapper so you can apply any grep options and regular expressions to suit your needs.
HEADER -x "HTTP/3 200"

# Validate the response body, e.g. JSON body
# JQ is a jq wrapper so you can apply any jq options and jq expression to suit your needs.
JQ '.method=="POST"'
JQ '.form=={"foo":"bar","hello":"world"}'

TEST 2: another test case
# ...

# More test cases...

Please check out the blog post for background and design details:

http://luajit.io/posts/a-simple-http3-testing-framework-based-on-bash-and-curl/