Empty "ENCRYPTION SETTINGS" panel with --log-format=CADDY and Caddy v2.7.4

[steffenbusch]

I have a question/problem with GoAccess (latest Docker Image) and Caddy v2.7.4 regarding the "ENCRYPTION SETTINGS" panel, which does not show any data.

My setup is latest Caddy v2.7.4 and GoAccess using the Docker Image - following this useful guide https://twitter.com/ALEMRANCU/status/1642337601578471424 (retweeted on the goaccess twitter page)

/var/log/caddy/goaccess.json --log-format=CADDY -o /var/www/goaccess/index.html --real-time-html --ws-url=wss://example.arm.stbu.net:443/ws --port=7890

By default, the JSON access logs from Caddy as of v2.7.4 are like these:

{"level":"info","ts":1696067106.8245707,"logger":"http.log.access.arm.stbu.net","msg":"handled request","request":{"remote_ip":"134.209.243.63","remote_port":"57526","client_ip":"134.209.243.63","proto":"HTTP/1.1","method":"GET","host":"arm.stbu.net","uri":"/","headers":{"Accept-Encoding":["gzip, deflate"],"User-Agent":["OhDear.app (+https://ohdear.app/docs/checks/uptime)"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"arm.stbu.net"}},"bytes_read":0,"user_id":"","duration":0.00328789,"size":32,"status":302,"resp_headers":{"Server":["TheRock/2.0"],"Vary":["Accept"],"Content-Type":["text/plain; charset=utf-8"],"Content-Length":["32"],"Date":["Sat, 30 Sep 2023 09:45:06 GMT"],"X-Frame-Options":["SAMEORIGIN"],"Location":["/dashboard"]}}

So the %K should be 772 and %k should be 4865.

Or with the human readable values output using this Caddyfile (using github.com/caddyserver/transform-encoder)

       log access-json {
                include http.log.access
                output file log/access.json
                format filter {
                        wrap json
                        fields {
                                request>tls>version tls_version TLSv
                                request>tls>cipher_suite tls_cipher
                        }
                }
        }

which produces JSON access logs like this:

{"level":"info","ts":1696067106.8245707,"logger":"http.log.access.arm.stbu.net","msg":"handled request","request":{"remote_ip":"134.209.243.63","remote_port":"57526","client_ip":"134.209.243.63","proto":"HTTP/1.1","method":"GET","host":"arm.stbu.net","uri":"/","headers":{"Accept-Encoding":["gzip, deflate"],"User-Agent":["OhDear.app (+https://ohdear.app/docs/checks/uptime)"],"Accept":["*/*"]},"tls":{"resumed":false,"version":"TLSv1.3","cipher_suite":"TLS_AES_128_GCM_SHA256","proto":"http/1.1","server_name":"arm.stbu.net"}},"bytes_read":0,"user_id":"","duration":0.00328789,"size":32,"status":302,"resp_headers":{"Date":["Sat, 30 Sep 2023 09:45:06 GMT"],"X-Frame-Options":["SAMEORIGIN"],"Location":["/dashboard"],"Server":["TheRock/2.0"],"Vary":["Accept"],"Content-Type":["text/plain; charset=utf-8"],"Content-Length":["32"]}}

where the %K should be TLSv1.3 and %k should be TLS_AES_128_GCM_SHA256.

But the ENCRYPTION SETTINGS panel in the real-time HTML is still not showing any data while all the other panels show data.

Any idea how to fix this? Has the Caddy JSON format maybe changed since the last update of the --log-format=CADDY definition?

[allinurl]

Thank you for sharing this. Do you happen to know if this is the default field output or if it has been changed at some point? The current output you provided is:

"version":"TLSv1.3","cipher_suite": "TLS_AES_128_GCM_SHA256"

However, the default in GoAccess includes "cipher_suite" and "proto". In any event, the following should still function properly:

goaccess access.log --log-format='{ "ts": "%x.%^", "request": { "remote_ip": "%h", "proto": "%H", "method": "%m", "host": "%v", "uri": "%U", "headers": { "User-Agent": [ "%u" ], "Referer": [ "%R" ] }, "tls": { "cipher_suite": "%k", "version": "%K" } }, "duration": "%T", "size": "%b", "status": "%s", "resp_headers": { "Content-Type": [ "%M" ] } }' --date-format=%s --time-format=%s

[steffenbusch]

However, the default in GoAccess includes "cipher_suite" and "proto". In any event, the following should still function properly:

goaccess access.log --log-format='{ "ts": "%x.%^", "request": { "remote_ip": "%h", "proto": "%H", "method": "%m", "host": 

Thank your for your help @allinurl.
That solved the problem.

The first JSON posted above is the default field output of Caddy v2.7.4. with

"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"arm.stbu.net"}

Using the GoAccess v1.8.0 with `--caddy-format=CADDY" wasn't working.
You might want to reproduce that using:

$ /home/stbu/goaccess --version | grep GoAccess
GoAccess - 1.8.
$ echo '{"level":"info","ts":1696325286.9971106,"logger":"http.log.access.arm.stbu.net","msg":"handled request","request":{"remote_ip":"46.101.122.201","remote_port":"39062","client_ip":"46.101.122.201","proto":"HTTP/1.1","method":"GET","host":"arm.stbu.net","uri":"/","headers":{"Accept-Encoding":["gzip, deflate"],"User-Agent":["OhDear.app (+https://ohdear.app/docs/checks/uptime)"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","server_name":"arm.stbu.net"}},"bytes_read":0,"user_id":"","duration":0.004788534,"size":32,"status":302,"resp_headers":{"Content-Type":["text/plain; charset=utf-8"],"Content-Length":["32"],"Date":["Tue, 03 Oct 2023 09:28:06 GMT"],"X-Frame-Options":["SAMEORIGIN"],"Location":["/dashboard"],"Server":["TheRock/2.0"],"Vary":["Accept"]}}' | \
/home/stbu/goaccess --log-format=CADDY  -o /tmp/goaccess-caddy.html -

Since the values of 772 for the TLS version and 4865 for the cipher-suite are not very user-friendly in the default Caddy JSON as well as in the GoAccess HTML report (if they would show up), I have modified the Caddy JSON log configuration like this:

       log access-json {
                include http.log.access
                output file log/access.json
                format filter {
                        wrap json
                        fields {
                                request>tls>version tls_version TLSv
                                request>tls>cipher_suite tls_cipher
                        }
                }
        }

and that produces this human friendly output (but the field labels are the same, just now strings instead of integers)

"tls":{"resumed":false,"version":"TLSv1.3","cipher_suite":"TLS_AES_128_GCM_SHA256","proto":"http/1.1","server_name":"arm.stbu.net"}

If I use your suggested --log-format it's working for me:

echo '{"level":"info","ts":1696324268.7687688,"logger":"http.log.access.arm.stbu.net","msg":"handled request","request":{"remote_ip":"167.71.54.244","remote_port":"36770","client_ip":"167.71.54.244","proto":"HTTP/1.1","method":"GET","host":"arm.stbu.net","uri":"/","headers":{"Accept-Encoding":["gzip, deflate"],"User-Agent":["OhDear.app (+https://ohdear.app/docs/checks/uptime)"],"Accept":["*/*"]},"tls":{"resumed":false,"version":"TLSv1.3","cipher_suite":"TLS_AES_128_GCM_SHA256","proto":"http/1.1","server_name":"arm.stbu.net"}},"bytes_read":0,"user_id":"","duration":0.003010369,"size":32,"status":302,"resp_headers":{"Content-Length":["32"],"Date":["Tue, 03 Oct 2023 09:11:08 GMT"],"Server":["TheRock/2.0"],"X-Frame-Options":["SAMEORIGIN"],"Location":["/dashboard"],"Vary":["Accept"],"Content-Type":["text/plain; charset=utf-8"]}}' | \
/home/stbu/goaccess --log-format='{ "ts": "%x.%^", "request": { "remote_ip": "%h", "proto": "%H", "method": "%m", "host": "%v", "uri": "%U", "headers": { "User-Agent": [ "%u" ], "Referer": [ "%R" ] }, "tls": { "cipher_suite": "%k", "version": "%K" } }, "duration": "%T", "size": "%b", "status": "%s", "resp_headers": { "Content-Type": [ "%M" ] } }' --date-format=%s --time-format=%s -o /tmp/goaccess-caddy.html -

So basically, the --log-format=CADDY is either not capturing the %K and %k properly (maybe because the JSON fields have changed since its inception), or the integer values of e.g. 772 for the TLS version and e.g. 4865 for cipher-suite are not recognized and shown by GoAccess.
However, I am very happy with the modified Caddy JSON and your suggested solution of --log-format='{ "ts": "%x.%^", "request": { "remote_ip": "%h", "proto": "%H", "method": "%m", "host": "%v", "uri": "%U", "headers": { "User-Agent": [ "%u" ], "Referer": [ "%R" ] }, "tls": { "cipher_suite": "%k", "version": "%K" } }, "duration": "%T", "size": "%b", "status": "%s", "resp_headers": { "Content-Type": [ "%M" ] } }' --date-format=%s --time-format=%s