Support per-host Prometheus metrics

[muety]

Initiated through this forum post – involving @hairyhenderson –, I'd like to open the discussion about extending the metrics directive to support metrics (request count, etc.) on a per-host basis.

I'd like to have metrics exposed for every single site block (speaking in Caddyfile terminology), i.e. metrics about different sites / vhosts result in different Prometheus metrics labels.

Since determining the labels automatically based on the incoming requests' host headers is problematic (see discussion above), a solution would be to manually assign a tag to the metrics directive. For instance:

example.org {
    root /var/www/html
    file_server
    metrics /metrics {
        host "example.org"
    }
}

I'm not really involved with Caddy's code base or development process, so this is rather supposed to be a starter for further discussion.

[hairyhenderson]

Thanks for logging this @muety!

Just to bring the goal into view, the v1 prometheus module had the ability to set a value to be used by the host label, which effectively solved the cardinality problem (i.e. the fact that there isn't a bounded set of values that can be used if it's determined automatically by looking at the Host header).

Caddy v2 is quite different from v1, and one of those differences is that request matchers are used extensively, and now the hostname in the site block is actually just a different kind of matcher.

This is maybe best explained by using caddy adapt to convert the following Caddyfile into the canonical JSON format:

example.com {
  respond /foo "hello world"
}
example2.com {
  respond /bar "hello world"
}
example2.com:8443 {
  respond /baz "hello world"
}

Here's the result of running caddy adapt:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [ ":443" ],
          "routes": [ {
              "match": [ { "host": [ "example2.com" ] } ],
              "handle": [ {
                  "handler": "subroute",
                  "routes": [ {
                      "handle": [ { "body": "hello world", "handler": "static_response" } ],
                      "match": [ { "path": [ "/bar" ] } ]
                    } ]
                } ],
              "terminal": true
            },
            {
              "match": [ { "host": [ "example.com" ] } ],
              "handle": [ {
                  "handler": "subroute",
                  "routes": [ {
                      "handle": [ { "body": "hello world", "handler": "static_response" } ],
                      "match": [ { "path": [ "/bar" ] } ]
                    } ]
                } ],
              "terminal": true
            } ]
        },
        "srv1": {
          "listen": [ ":8443" ],
          "routes": [
            {
              "match": [ { "host": [ "example2.com" ] } ],
              "handle": [ {
                  "handler": "subroute",
                  "routes": [ {
                      "handle": [ { "body": "hello world", "handler": "static_response" } ],
                      "match": [ { "path": [ "/baz" ] } ]
                    } ]
                } ],
              "terminal": true
            } ]
        }
      }
    }
  }
}

So right now, the metrics support knows about the server (by name, i.e. srv0/srv1) and the handler (by module name, i.e. subroute, static_response, etc...). But there is not yet any support for matchers.

Since matchers are evaluated late (i.e. only at request time), and the only thing that matters is whether a request was matched (rather than how it was matched), it's impossible with the current code to be able to determine which host or path was matched in the metrics.

This is all a long-winded way of saying that I think what we need here is some sort of integration into the different matchers so that we can track the host label for the given host matcher.

The advantage of this approach is that we could use a similar approach to track a path label too.

Anyway, this is just a vague brain-dump of possibilities 😉

[muety]

In the meanwhile, I set up grok_exporter as a workaround to read Caddy's log files and expose metrics from those to Prometheus. However, having them provided directly by Caddy and in greater detail is still desirable :)

[mqus]

I just wanted to set up Prometheus (haven't looked at it once before today, so I am quite a newbie) and ran into this issue. I wanted to add my own perspective.

For me, a single, static "host" option I can set manually would be enough.

But if cardinality is no issue, then I would really like to go all the way and be able to use a string with placeholders (e.g. https://caddyserver.com/docs/caddyfile/concepts#placeholders) to modify the label (or possibly multiple labels). Of course, this could incur significant runtime costs and probably not all placeholders can be applicable for each metric. I haven't checked the last part though.

PS: Of course, this is also just a braindump, I don't know if this is feasible.

[jasonmccallister]

Would love support for this, but I see the problem pointed out by @hairyhenderson.

However, I wonder if it makes sense to add a generic tags to pass onto metrics?

This would allow you to set something like the following:

example.com {
  respond /foo "hello world"
  tags {
    service foo
    host example2
    environment testing  
  }
}
example2.com {
  respond /bar "hello world"
  tags {
    service bar
    host example2
    environment hello  
  }
}
example2.com:8443 {
  respond /baz "hello world"
  tags {
    service baz
    host example2
    environment testing  
  }
}

This would allow fine grain queries on the metrics or during an ETL process like @muety mentioned.

[maroziza]

Maybe as a stop-gap solution there can be just boolean option to include per-host metrics. yes, i understand the consequences, just give me the hosts. i spent a week for porting old v1 plugin and it's kind of works stand-alone, but crashing constantly with caddy-docker-proxy, so it would be great to have official solution.
i'm using caddy for TLS termination for old legacy microservice application build around nginx. i want to migrate these nginx configs to caddy, but i want to gather metrics to guide the transition. For now i'm using grok_exporter, but it's cumbersome, because i need to store logs and parse them to get this logs.

[hairyhenderson]

Thanks for the input @maroziza - understood! The last few months have been very busy for me so I haven't been able to spend much time working on Caddy. I'll have some time off in the next few weeks and hopefully I'll be able to get around to this!

[rgdev]

any update on this ? The addition of per-host metrics would be a game changer for the caddy monitoring stack!

[hairyhenderson]

Apologies @rgdev, and everyone else! I (obviously) wasn't able to get to this over the holidays 😅... I've recently started a new job and so am still swamped.

But! I now have a post-it-note on my screen telling me to work on this. So, that'll probably help 😉

[hairyhenderson]

FYI - I haven't forgotten about this issue, but there's a more general solution that'll work for the host and many other custom labels, described in #4016. I'm thinking it may be a good approach, so eyes on that issue would be appreciated!

[JXGA]

Hello!

Has there been much progress on this? I'd still really like this feature!

Thanks!

[francislavoie]

@JXGA nothing to report at this time. The prometheus library doesn't let us reconfigure at runtime, which is fundamentally incompatible with how Caddy is configured (i.e. Caddy can gracefully reload config in-process). So until a solution for that is figured out, we don't have a good way to add new features that might involve configuration.

[AdrianFletcher]

Just throwing in another vote for this feature. We're running multiple upstreams from the same front-end port on a single Caddy instance, so being able to track/monitor per match/host etc.. would be most welcome.

Is it potentially throwing the baby out with the bathwater a bit to say this proposal would be incompatible with how Caddy is configured?

One could cache these configuration options on first launch and ignore/write to the error log on any subsequent changes on config reloads, and document this clearly on caddy docs?

[ungive]

I would also love to see this feature!

Since I am using docker compose, my workaround was to simply define multiple Caddy services. One "main" one, which did all the HTTPS certificate handling and which had reverse proxy definitions to my other HTTP-only caddy instances. I created one caddy instance per server that needed per-host metrics and then just added each of those hosts to my prometheus configuration file. Works well so far. Maybe this helps someone as a workaround!