Discussion around using variables in Caddyfile

[brucejo75]

I was reading the variable assignment thread in the forum essentially outlining that you could assign a placeholder using map or import. The author of the post specifically asked about being able to assign non-environmental variables (e.g. placeholders).

It occurred to me that "Environment Variables" could be more generalized to "Variables" if you require a declaration before usage...

I would like to do something like this:

:55555 {
{$MY_BASE_SITE=env.MYBASESITE | mysite.com}
  map {host} {port} {
    "a.{$MY_BASE_SITE}" 1111
    "b.{$MY_BASE_SITE}" 2222
  }

  reverse_proxy 0.0.0.0:{port}
}

I can make this work by assigning an environment variable before running caddy, but oftentimes it would be useful to declare in the Caddyfile (e.g. use of more than a few variables).

Feature Request

• Create a new variable type, modelled after Environment Variables with the requirement that they need to be explicitly declared.

# Always need to declare variables before using them
# set to DEFAULTVALUE unless env.VARIABLE exists
{$VARIABLE=env.VARIABLE | DEFAULTVALUE}

# subsequent references will be equal to whatever was assigned e.g. will be DEFAULTVALUE
# if env.VARIABLE is not defined
respond {$VARIABLE}

# and a default override in place could still be used in the reference
respond {$VARIABLE:foobar}

# if env.VARIABLE2 does not exist VARIABLE2 will be empty
{$VARIABLE2= env.VARIABLE2}

#invalid because there is no declaration e.g. {$VARIABLE3=...}
{$VARIABLE3}

[mholt]

(Your original post in the website repo was the start of a great wiki entry you could submit!)

Variables can already be declared with the vars HTTP handler module.

(This is how the root directive works, it just sets a var called "root".)

Other than the map directive, I'm not sure we directly expose the vars module to the Caddyfile (with map being a fancy way to assign variables based on conditions, and vars would just be a direct assignment).

[brucejo75]

Thanks @mholt! This makes me laugh, I will drag you through my process... (I am sharing it because, most of my attempts to learn something new about caddy go like this)

I read vars HTTP handler module.

---

Thought bubble:

Hmm, what is the mystical block that has 2 empty strings in it? a typo? Which part is a typo? Maybe an omission? Maybe there is some context somewhere... Why isn't there an example?
Maybe there is more info in the vars documentation? No help there. That is just a matcher.
Let me try guessing...

Try 1

Is it a global options block?

{
    "hello":"foobar"
}

:2015 {
  respond "{https.vars.hello}, world! {https.vars.hello}"
}

nope: run: adapting config using caddyfile: CaddyTest:4: unrecognized global option: Hello

Try 2

Maybe it should be a vars block?

vars {
    "hello":"foobar"
}

:2015 {
  respond "{https.vars.hello}, world! {https.vars.hello}"
}

nope: run: adapting config using caddyfile: CaddyTest:4: unrecognized directive: Hello

Try 3

Maybe put it in the Site block?

:2015 {
{
  "hello":"foobar"
}
  respond "{https.vars.hello}, world! {https.vars.hello}"
}

nope: run: adapting config using caddyfile: CaddyTest:7 - Error during parsing: Site addresses cannot contain a comma ',': '{https.vars.hello}, world! {https.vars.hello}' - put a space after the comma to separate site addresses

Try 4

Maybe I need to put it in a vars block inside the global block?

{
  vars {
    "hello":"foobar"
  }
}

:2015 {
  respond "{https.vars.hello}, world! {https.vars.hello}"
}

nope: run: adapting config using caddyfile: CaddyTest:3: unrecognized global option: vars

Try 5

I know maybe it is a vars block in the Site block!

:2015 {
{
  "hello":"foobar"
}
  respond "{https.vars.hello}, world! {https.vars.hello}"
}

grrr: run: adapting config using caddyfile: CaddyTest:3: unrecognized global option: vars

Oh yeah, I get most questions answered by going through the forum.

errr, search for "vars" turns up some references but I cannot find vars on most of the pages.

Does the module need to be installed?

The module claims it is standard so it should be in the executable... No luck there.
Look at the go code? I see the SetVar but that is no help.

How do I reference a module? I cannot find anything on it in the docs. I included the tls godaddy module and that extends the tls directive but the documentation for the godaddy module is just as opaque. Luckily, it was all explained in the github.

Wait look for http.vars on the forum!

Found this, a clue!

Wait a minute, those are JSON files. What did @mholt say above?

I'm not sure we directly expose the vars module to the Caddyfile (with map being a fancy way to assign variables based on conditions, and vars would just be a direct assignment).

D'oh! This is a JSON only feature? let me try it

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":2015"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "vars",
                  "hello": "foobar"
                },
                {
                  "body": "{http.vars.hello}, world! {http.vars.hello}",
                  "handler": "static_response"
                }
              ]
            }
          ]
        }
      }
    }
  }
}

Works like a charm! I guess it is not available in a Caddyfile, that's a shame 😞

Observations

• The module documentation is woefully lacking.

{
 "":""
}

What the heck is that?

• Maybe good to add a JSON example and a Caddyfile example.
• If it does not work in Caddyfile, say so.
• Maybe some more info on how modules integrate with caddy. Apparently, the handler keyword is important. I do not see that described in the docs? At least when I search for the word "handler" the only real hit is this.

erg: 2 hours I will never get back

[mholt]

@brucejo75 Well, thank you for writing all that out. 😅 A very insightful read.

A few thoughts:

As you discovered, these are the JSON config docs. Caddy's native configuration is JSON, and we emphasize this from the Getting Started page. In other words, this is not the area of the site describing the Caddyfile config. I could have been clearer in my reply above, where I tried to contrast that with the Caddyfile (when I said "I'm not sure we directly expose the vars module to the Caddyfile"), but upon re-reading it, I realize why that wasn't clear and I'm sorry for the confusion. I was just saying that there is a module that lets you define variables. It just has to be done with JSON. Also sorry for the lost time.

I think the module documentation is very clear and concise:

The key is the variable name, and the value is the value of the variable. Both the name and value may use or contain placeholders.

Hence, the two empty strings: { "": "" }. It's a key and value of a JSON object. And the docs say exactly what they are.

We could probably use examples; it's always hard to draw the line between reference documentation and examples though. I like keeping them separate to maintain good organization.

For examples, we have our wiki but not nearly enough contributors. ☹️ Actually, your discoveries about the vars module would be a handy addition to the wiki. You should post there!

I guess it is not available in a Caddyfile, that's a shame 😞

No one has asked for it before now. 🤷‍♂️ I guess we could go ahead and add it. Any thoughts @francislavoie ?

Again, I really appreciate your time here, and apologize for the confusion. This is just a feature that not many people have used or asked about.

[francislavoie]

Yeah we should add the Caddyfile directive, it was an oversight. No harm in it existing.

It's really not that useful on its own though, IMO. map is much more flexible.

[mholt]

Real quick, before we do add a vars directive, is it already possible to simply "set a variable" using the map directive?

[francislavoie]

Yes but it's always a 3-liner and it's kinda awkward I guess:

map 1 {output} {
    1 value
}

Something like that might work. But obviously it will still cause a string comparison on every request.

[mholt]

Ah, clever. That is kinda awkward, you're right.

What about a special-case syntax for the map directive that basically accepts map <var> <value> (no block)? As a shorthand. That way, you just use map to assign variables, period.

[francislavoie]

I'm not a big fan of overloading directives that much. It's cheap to just add vars as a directive IMO

[mholt]

It's cheap, but IMO it makes more sense to have map be the directive that sets variables. 🤷‍♂️

[brucejo75]

Am I correct in assuming that the vars directive only sets placeholders? If so, the title: "Expose 'vars' handler as Caddyfile directive" is not what I was requesting or need. I am requesting an extension of how environment variables work.

I am requesting variables that get substituted before parsing not during runtime.

1. setting a variable via map works and I am OK with it if that is the way to go for placeholders. No need to change anything for setting placeholders, there is a reasonable workaround via map.
2. I could not accomplish what I want to do via placeholders in my original example. I tried setting map parameters with placeholders and it does not work. And it does not make sense for it to work. The map parameters are defining a fixed table at parse time I believe. That is why I really want more out of the environment variable paradigm.

But a real objection is the documentation. Guys, this stuff is hard to connect the dots on... (And it is a shame, because the product is EXCELLENT!)
As an example, here are some small tweaks to the vars module documentation, that would have saved me hours:

Change the top portion to lay out the entire JSON object:

JSON Directive:
{
      "handler" "vars",
      "<keyword>": "<value>
}

Caddyfile Directive:
-NOT AVAILABLE-
Note: placeholders can be set using the `map` directive in Caddyfiles.

These simple changes to be more explicit would have saved me 2 hours. I am happy to take a shot at fixing these things as I run across them and submitting PRs.

In my limited experience with caddy you have made very reasonable engineering tradeoffs (you are good at what you do). But discovery of these tradeoffs is an exercise in scanning everywhere docs (in multiple places) & the forum). I just think it would be easy to spell the tradeoffs out explicitly in the docs.

[francislavoie]

Am I correct in assuming that the vars directive only sets placeholders?

Doesn't exactly set placeholders, it sets "vars" in the request context, which can be used with a placeholder (slight distinction).

The map directive also does that but it's basically a fancy switch statement to set one or more "vars" at the same time.

I am requesting variables that get substituted before parsing not during runtime.

That's exactly how they work right now, in the Caddyfile {$ENV} style, they get replaced at Caddyfile adapt time.

Defining new variables though, would be a lot of added complexity I think, and I don't think that's warranted here. Just set your variables ahead of time.

Or use snippets to reuse some common bits of config https://caddyserver.com/docs/caddyfile/concepts#snippets

As an example, here are some small tweaks to the vars module documentation, that would have saved me hours

The JSON docs are auto-generated from the code's comments and Go code structure. With the way it's done right now, there's currently no way to control what those string values end up containing in the generated output.

But noted, that could be a future addition to the JSON doc generator, but it would also require adding something to the code to fill that in.

Caddyfile Directive:
-NOT AVAILABLE-
Note: placeholders can be set using the map directive in Caddyfiles.

I don't think this makes sense. If you're in the JSON docs, you shouldn't assume at all that any of it is available in the Caddyfile.

The Caddyfile adapter outputs a JSON config, but JSON can't go back to Caddyfile. It's a one-way relationship.

[brucejo75]

This thread now has 3 separate concepts:

vars does not accomplish what I am looking for lets stop talking about it

Example:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":2015"
          ],
          "routes": [
            {
              "handle": [
                {
                  "handler": "vars",
                  "base": "mysite.com"
                },
                {
                  "defaults": [
                    "a.{base}"
                  ],
                  "destinations": [
                    "{myvar}"
                  ],
                  "handler": "map",
                  "source": "1"
                },
                {
                  "body": "myvar: {myvar} base: {http.vars.base}",
                  "handler": "static_response"
                }
              ]
            }
          ]
        }
      }
    }
  }
}

curl localhost:2015: myvar: a.{base} base: mysite.com
lets take vars out of the conversation

Environment variables almost do it (for Caddyfiles only)

All that I am saying is "Environment variables are great" and can be used to accomplish what I need. For cases where there are only 1, 2 or 3 variables Environment variables do the job as is.

For cases where there might be a lot more parameters in a file it would be useful to:

• Be able to initialize environment variables within the Caddyfile rather than their value always coming in from the environment.

Your users are programmers we want to parameterize our config files.

This seems very intuitive to me that a developer would want to make their config file parameterized. This is a requested feature of nginx, but nginx is resisting the feature. One user even developed his own pre-processor that he has shared: https://stackoverflow.com/a/15432888/7669642 (this was 9 years ago, but people are still responding to it as recently as 2020)

Possible reactions to this request:

• This is a great idea and would differentiate us from nginx, we should think about how we could support macro expansion within the config files (and within json files!). Stay tuned it may take a while.
• We are better than nginx that's good enough.
• Ugh, too much work. We have other more important things to do. Maybe a later date.
• This an unreasonable idea, nobody would want this. Don't think anyone would want more than 2-3 variables in a config file.
• Hey have you tried vars? (yes, they do not accomplish the task)
• Have you tried snippets? (yes, I have used snippets to great effect but I cannot use them to parameterize the map parameters)

Documentation has issues

I will open separate issues in https://github.com/caddyserver/website