Support Go plugin mechanism for Caddy plugins/modules

[FranklinYu]

This is a feature request.

Right now, the DNS-provider module seems to be compiled with the main program to form a single binary. This is easier to download, but more difficult for distribution to include those providers in packages. Go language actually provide a mechanism called plugin, where a program can load other Go code during runtime. It would be nice if individual DNS-providers can be packaged as plugins, and caddy simply load them on-demand (e.g. after parsing the Caddyfile).

[emilylange]

Just to provide a bit more context, @septatrix (https://caddy.community/u/septatrix) requested the same on the forums in August and was denied:

https://caddy.community/t/support-for-go-plugins/16724

[FranklinYu]

In that post:

You said before that one philosophies of the Caddy project is a simple deployment workflows.
Having to compile caddy yourself is definitely a lot more involved than using packages from a distributions official repositories.

I strongly agree with this. I’m using NGINX + Certbot right now, and it all comes from Debian official repository, so I don’t need to compile anything. I understand that big corporations want to compile everything from scratch for less dependency on the operating system itself, but compiling everything from scratch sounds overkill for small companies or personal server (homelab).

Without the Go Plugin mechanism, even Caddy itself cannot provide a deb repository with DNS support, not to mention the repository from the Debian team. The current deb repository only has the “bare” Caddy. I hope that the Caddy team can take another look at the proposal.

[ChrTall]

I strongly agree with this request. Providing seperate packages for each module/plugin would be exactly what is needed for packagers. Official distro repositories only include base caddy.

[mholt]

I sympathize with the desire for traditional package managers. But I want to clarify a few things that the linked forum thread didn't go into enough detail about.

so I don’t need to compile anything.

You already don't have to compile anything -- we can do that for you with our build server (as a courtesy) or you can use xcaddy locally. (It wraps all the compilation for you.)

Note the hugely significant drawbacks documented by the Go plugin package:

However, the plugin mechanism has many significant drawbacks that should be considered carefully during the design. For example:

• Plugins are currently supported only on Linux, FreeBSD, and macOS, making them unsuitable for applications intended to be portable.

• Applications that use plugins may require careful configuration to ensure that the various parts of the program be made available in the correct location in the file system (or container image). By contrast, deploying an application consisting of a single static executable is straightforward.

• Reasoning about program initialization is more difficult when some packages may not be initialized until long after the application has started running.

• Bugs in applications that load plugins could be exploited by an attacker to load dangerous or untrusted libraries.

• Runtime crashes are likely to occur unless all parts of the program (the application and all its plugins) are compiled using exactly the same version of the toolchain, the same build tags, and the same values of certain flags and environment variables.

• Similar crashing problems are likely to arise unless all common dependencies of the application and its plugins are built from exactly the same source code.

• Together, these restrictions mean that, in practice, the application and its plugins must all be built together by a single person or component of a system. In that case, it may be simpler for that person or component to generate Go source files that blank-import the desired set of plugins and then compile a static executable in the usual way.

In the end, even the package docs recommend just going back to what we're doing.

It would be useful if package managers could execute trusted/verified installer scripts (for example, something like xcaddy or even a signed bash script) that could take the (signed) vanilla Caddy source and add a couple import lines then run go build. Unfortunately they do not seem to have this capability.

Until those issues documented upstream are resolved, I don't think Go plugin package is a good solution for Caddy.

I'll close this, but feel free to continue discussion if there's anything actionable we can or should do about this; but I think it's a combination of deficiencies in package managers and technical hurdles in Go (which was always designed to be a statically linked).

[francislavoie]

FWIW @mholt I largely disagree:

• Plugins are currently supported only on Linux, FreeBSD, and macOS, making them unsuitable for applications intended to be portable.

  I think this is fine, 95%+ of users are on those platforms. We can just panic if users try to make use of plugins on other platforms (realistically, Windows).

• Applications that use plugins may require careful configuration to ensure that the various parts of the program be made available in the correct location in the file system

  Not a relevant concern for us. The program can function without plugins (vanilla Caddy) so plugins only add new functionality.

• Reasoning about program initialization is more difficult when some packages may not be initialized until long after the application has started running.

  Not a relevant concern for us. We'd simply initialize plugins right near program startup (before any config loading) to ensure all the init() in plugins are invoked properly.

• Bugs in applications that load plugins could be exploited by an attacker to load dangerous or untrusted libraries.

  Not a relevant concern either IMO because we would only load configs at program startup.

• Runtime crashes are likely to occur unless all parts of the program (the application and all its plugins) are compiled using exactly the same version of the toolchain, the same build tags, and the same values of certain flags and environment variables.

  This is annoying, but not a dealbreaker, IMO. This is more of a tooling and end-user-diligence thing. Crashes also aren't a guarantee according to that wording, so it might work in some cases without matching Go/Caddy versions? Something we need to experiment with, we shouldn't assume it's a critical issue.

• Similar crashing problems are likely to arise unless all common dependencies of the application and its plugins are built from exactly the same source code.

  I'm not exactly sure what this is implying. But same as above, though, more of a tooling and diligence thing.

• Together, these restrictions mean that, in practice, the application and its plugins must all be built together by a single person or component of a system.

  IMO this just means that xcaddy tooling needs to be designed in such a way that it's easy to use for users, and ideally that the Caddy website download page could produce the plugins the same way. Doesn't sound like a dealbreaker to me either.

I think there's a lot of value in trying to explore this possibility further, and of course we would include an "EXPERIMENTAL" disclaimer for the forseeable future.

[septatrix]

I sympathize with the desire for traditional package managers. But I want to clarify a few things that the linked forum thread didn't go into enough detail about.

I did not go into further details for those aspects as you did not ask me to and stopped participating in the discussion but I am happy to respond to these details now and expand upon the great response from the previous person.

so I don’t need to compile anything.

You already don't have to compile anything -- we can do that for you with our build server (as a courtesy) or you can use xcaddy locally. (It wraps all the compilation for you.)

"we can do that for you with our build server" which goes even more against your personal philosophy that sysadmins should have total control over the binaries. I trust large distributions with reproducible builds and large enterprises and communities backing them a lot more than the build server of a significantly smaller company.
With xcaddy I still have to perform the compilation, it's just abstracted away from me.
And it is not only the initial compilation. I would also have to regularly update the binaries, roll them out to our servers etc. All the stuff which is already solved by the distributions package manager.

Note the hugely significant drawbacks documented by the Go plugin package:

The previous comment already addresses many of these though I would like to expand upon that for a few points.

• Plugins are currently supported only on Linux, FreeBSD, and macOS, making them unsuitable for applications intended to be portable.

Those would also be the platforms for which these would make the most sense as Linux and BSD have package managers. As a reminder, my proposal's intention was to enable usage of Caddy plugins with the binaries shipped by distributions.

• Applications that use plugins may require careful configuration to ensure that the various parts of the program be made available in the correct location in the file system (or container image). By contrast, deploying an application consisting of a single static executable is straightforward.

To ensure this would be the responsibility of the distributions' package maintainers, not caddy itself.

• Reasoning about program initialization is more difficult when some packages may not be initialized until long after the application has started running.
• Bugs in applications that load plugins could be exploited by an attacker to load dangerous or untrusted libraries.

The above is already true for the current Caddy plugins. Usage of go plugins would reuse the exact same initialization procedure! This is also why my proposed patch is so simple with only about 10 lines, mostly for scanning one directory for files to load.

• Runtime crashes are likely to occur unless all parts of the program (the application and all its plugins) are compiled using exactly the same version of the toolchain, the same build tags, and the same values of certain flags and environment variables.
• Similar crashing problems are likely to arise unless all common dependencies of the application and its plugins are built from exactly the same source code.
• Together, these restrictions mean that, in practice, the application and its plugins must all be built together by a single person or component of a system. In that case, it may be simpler for that person or component to generate Go source files that blank-import the desired set of plugins and then compile a static executable in the usual way.

Not a problems for distributions as everything is compiled inside a controlled environment. Even if people compile plugins locally with the distributions provided go toolchain they would also have a matching environment (though explicit support for this scenario was out of the scope of my proposal).

In the end, even the package docs recommend just going back to what we're doing.

Sure, xcaddy is not doing anything different than what would be done by using plugins. However, the drawbacks do not really apply for the use case desired by the community in this issue (and my forum post) as packages build by distributions provide exactly this controlled environment where everything is matched correctly. So we gain all of the advantages without being bothered by the disadvantages.

It would be useful if package managers could execute trusted/verified installer scripts (for example, something like xcaddy or even a signed bash script) that could take the (signed) vanilla Caddy source and add a couple import lines then run go build. Unfortunately they do not seem to have this capability.

This would be something which AUR or gentoo would do: Manually build everything on the machine locally. Though there are good reasons not to do this, such as the requirement that the whole golang toolchain would suddenly be required for caddy.

What one could do is abuse post-install scripts or package xcaddy but this does not remove the drawbacks of requiring the golang toolchain, making updates take longer, etc.

I'll close this, but feel free to continue discussion if there's anything actionable we can or should do about this; but I think it's a combination of deficiencies in package managers and technical hurdles in Go (which was always designed to be a statically linked).

I currently fail to see the disadvantages of including go plugin support (besides likely a small increase in binary size). The prototype I shared is only about 10 lines which need to be changed to get this to work. A proper solution would likely make the directory where plugins are looked for configurable at compile time but otherwise this does not require any big changes.
The existing plugin mechanism is already compatible with how go plugins are be loaded, so there is basically no maintenance burden. Label it as a feature for advanced users and package maintainers and everybody is happy.
If this required completely rethinking how plugins work in caddy I would understand your aversion to this proposal though that is not the case. It is actually a fairly simple change which greatly benefits distributions and more advanced users.

PS: No one is suggesting to replace the current plugin mechanism if that seems to be your impression. Simply leave xcaddy and the build server as is for those who prefer that.

[septatrix]

As an aside I also recently discovered another area this would have been really helpful. My company has been working on a few IoT devices which have an immutable root and where different features and configurations can be installed as overlay filesystems (like extension images). In those cases one would not want to update the base image but instead simply drop in an extension for the /usr partition which contains some plugins.

So this is not only interesting for distros but also advanced users where replacing a file under /usr/bin might be easy to say but impossible to perform. Using xcaddy or anything else in that case would not be possible.

[hslatman]

WebAssembly modules might be a great (alternative) approach (to Go plugins) for extending Caddy's capabilities (at runtime) for the following reasons:

• Unlike Go plugins, they're supported on more operating systems, depending on the WebAssembly runtime used by the host.
• Plugin functionality and system extensibility are often mentioned as major application areas of WebAssembly modules.
• The proxy-wasm specification defines an ABI for L4 and L7 proxies making use of WebAssembly modules to provide custom functionality. The spec was developed for Envoy and was initiated for similar reasons as posed in this issue. It's not a "standard-standard", but it can be used with other proxies (and is already used with Istio too, for example).
• HTTP-wasm defines an ABI for HTTP handlers implemented in WebAssembly. There's prior work supporting this in Caddy in https://github.com/anuraaga/caddy-wasm (and a WASI variant here: https://github.com/brendandburns/caddy-wasm).
• By leveraging WebAssembly, you get the benefits of WebAssembly, a.o. isolated execution (code executed in a virtual machine), maintainability/portability (multiple languages support WebAssembly as a target, so can be supported relatively easily) and security (the execution environment can be constrained using capabilities, reducing potential attack surface).

WebAssembly is still a relatively young technology, so changes in certain parts are somewhat expected. The nature of the technology has resulted in several distinct runtimes, each with their own pros and cons (and supported functionalities), as well as several "extensions" to WebAssembly in the form of the WebAssembly System Interface (WASI), WASIX, etc. So when considering to go this route, there's certainly some choices to be made that may rule out certain things (now), but all in all I think it's a very promising direction to go if one's considering runtime extensibility.

I think Caddy already does a phenomenal job at being able to extend its functionalities (at compile time), and I don't think those capabilities should be removed, but I do think that being able to provide plugins at runtime could make Caddy even greater than it already is, and for that purpose I think WebAssembly is a great way to implement it.

[francislavoie]

@hslatman I've not dabbled, but wouldn't WASM mean that all existing plugins would not be compatible with this approach? We rely directly on Go's init() for allowing plugins to self-register themselves.

I don't think there's any appetite for designing a whole new system for loading plugins. It would be a massive change to Caddy's architecture, adding a lot of complexity. Maybe we could consider it for Caddy v3, but we're not there yet.

But again, maybe I'm misunderstanding how WASM modules interact with Go.

[hslatman]

@francislavoie you're absolutely right that supporting extensibility through Wasm modules is not (immediately) compatible with Caddy's current module/plugin architecture, but I don't think that's the full issue at hand here, per se. For compile time extensibility, the current implementation works great, and it also aligns closely with the things mentioned before. But when someone's looking to extend Caddy at runtime (and to do so in a way that doesn't require Caddy to be recompiled and thus restarted), an alternative approach using WebAssembly could be a good way to do it. I would argue using WebAssembly is better than using Go plugins, or some other mechanism, such as the model used by Hashicorps go-plugin, because of the things I mentioned before.

To be able to use Wasm modules with Caddy, Caddy would have to become the host providing the runtime (providing ways to (down)load modules, cache modules, execute the module, etc). This is essentially what the caddy-wasm projects do, specifically for providing HTTP handler functionality. For the proxy-wasm functionality, a bit more may be required and I haven't looked into the specifics of that, but I think that also would make sense for Caddy, as it can be used as a reverse proxy for L7 (and L4 through the additional module).

Currently I see this as an additional way for extending Caddy's functionality, specifically to do so at runtime, and maybe just for some specific functionalities. Making Caddy depend on WebAssembly for all of its plugins sounds interesting too, and might also be possible, but I agree that that would make for a totally different project. In terms of making existing plugins compatible with this approach, things may not be that far off, considering modules are already written in Go, which can be compiled to Wasm/WASI. There's the obvious downside that multiple distinct modules written in Go would result in a considerably larger total size (every module would have its own Go runtime, essentially), but compiling using TinyGo might result in smaller binaries. There's also functionalities that are not (yet) supported, so that's something to keep in mind too.

All that said, people can extend Caddy today using WebAssembly already, as long as they use one of the caddy-wasm repositories (or maybe there's others too), or provide their own implementation of loading Wasm modules and calling into those from whatever type of module functionality they would like to offer.

[FranklinYu]

so I don’t need to compile anything.

You already don't have to compile anything -- we can do that for you with our build server (as a courtesy) or you can use xcaddy locally. (It wraps all the compilation for you.)

"we can do that for you with our build server" which goes even more against your personal philosophy that sysadmins should have total control over the binaries. I trust large distributions with reproducible builds and large enterprises and communities backing them a lot more than the build server of a significantly smaller company.

I would like to echo this. https://reproducible-builds.org is a real effort of the Linux community. Both Debian and Arch Linux are working hard on this.

[FranklinYu]

so I don’t need to compile anything.

You already don't have to compile anything -- we can do that for you with our build server (as a courtesy) or you can use xcaddy locally. (It wraps all the compilation for you.)

If I use the build server, how do I update the binary? (I assume that the “build server” here refers to https://caddyserver.com/download.)

[mholt]

@FranklinYu

If I use the build server, how do I update the binary?

Just download the binary again (is that what you're asking?) then overwrite the current one.

@septatrix

"we can do that for you with our build server" which goes even more against your personal philosophy that sysadmins should have total control over the binaries. I trust large distributions with reproducible builds and large enterprises and communities backing them a lot more than the build server of a significantly smaller company.

The build server doesn't take away control over the binaries on your server. It just makes them more convenient to get. That said, the previous version of our build server (before Caddy 2) signed the custom binaries. We could possibly look into doing that with a business/enterprise sponsorship. (It's tedious to ensure it is done correctly.)

With xcaddy I still have to perform the compilation, it's just abstracted away from me.

I mean, abstracting it away from you is the whole point of other distribution mechanisms as well. It's just that instead of downloading pre-built artifacts, a single command builds your binary for you on the spot. I don't see how that's different in terms of abstraction from a package manager.

my proposal's intention was to enable usage of Caddy plugins with the binaries shipped by distributions.

I'm just not aware of any Linux distribution that ships with Caddy in the first place.

Not a problems for distributions as everything is compiled inside a controlled environment.

The problem is we can't control how/where people would use this feature, I can foresee leading to lots of confusion and pain.

In those cases one would not want to update the base image but instead simply drop in an extension for the /usr partition which contains some plugins.

I do worry about runtime production vulnerabilities here where malicious executable libraries could inadvertently be deployed. Sure, it's preventable with certain processes, but I don't trust enough of the user base to do that, and right now it's really nice to be able to explain how/why Caddy is not susceptible to those kinds of attacks (you lose the cryptographic safety the Go toolchain provides when you start frankensteining your binary together in production).

@francislavoie

I think there's a lot of value in trying to explore this possibility further, and of course we would include an "EXPERIMENTAL" disclaimer for the forseeable future.

Ok, fair enough. I suppose we (or someone) can experiment with this on the side, but I don't feel like investing my or our energy into it right now -- unless you personally would like to tackle it. There's just a lot on my plate that has higher priority at the moment. And there's no guarantees that even if it does work, in some way, for some people, in some environments, that it'll end up "feeling right" for the project. (I don't know how to better articulate that, other than the stated values/goals and drawbacks above.)

[septatrix]

my proposal's intention was to enable usage of Caddy plugins with the binaries shipped by distributions.

I'm just not aware of any Linux distribution that ships with Caddy in the first place.

Fedora(+EPEL Repo for RHEL derivatives), Debian(+Raspbian), Ubuntu, Arch(+Manjaro), Alpine, Gentoo, and NixOS/nixpkg all ship recent versions of caddy: https://repology.org/project/caddy/badges
To my knowledge, however, most of these ship without any or only a very small set of the plugins.

In those cases one would not want to update the base image but instead simply drop in an extension for the /usr partition which contains some plugins.

I do worry about runtime production vulnerabilities here where malicious executable libraries could inadvertently be deployed. Sure, it's preventable with certain processes, but I don't trust enough of the user base to do that, and right now it's really nice to be able to explain how/why Caddy is not susceptible to those kinds of attacks (you lose the cryptographic safety the Go toolchain provides when you start frankensteining your binary together in production).

In the case of sysexts these have to be signed when secure boot is enabled.

@francislavoie

I think there's a lot of value in trying to explore this possibility further, and of course we would include an "EXPERIMENTAL" disclaimer for the forseeable future.

Ok, fair enough. I suppose we (or someone) can experiment with this on the side, but I don't feel like investing my or our energy into it right now -- unless you personally would like to tackle it. There's just a lot on my plate that has higher priority at the moment. And there's no guarantees that even if it does work, in some way, for some people, in some environments, that it'll end up "feeling right" for the project. (I don't know how to better articulate that, other than the stated values/goals and drawbacks above.)

I feel like the best way to go forward with this is to add a compile time option for a directory in which to search for such plugins. If this is unset (default) this mechanism is not used. Distributions could then set that option to enable the behaviour.

[mholt]

Fedora(+EPEL Repo for RHEL derivatives), Debian(+Raspbian), Ubuntu, Arch(+Manjaro), Alpine, Gentoo, and NixOS/nixpkg all ship recent versions of caddy: https://repology.org/project/caddy/badges

Oh, right, I know about package managers that ship Caddy, but I don't know of any distributions that ship it.

I'd like for the first experiments to be carried out in a separate repository for now, and as more people try it and work out the kinks and figure out the pinch points in practice, we can consider its utility for adoption into the project.

[mholt]

After reviewing the patch more, I'm willing to try the patch proposed in that forum thread in our repo as a strictly experimental functionality.

It does look like this will require all the plugins to make a main though.

[francislavoie]

It does look like this will require all the plugins to make a main though.

I think xcaddy can shim this in pretty easily. I think we'd need a command like xcaddy build-plugin <package> which would make a main.go which imports the plugin, and builds that.

We'll need to see what happens to caddy build-info when loading plugins like this, etc. Bunch of little details to iron out.