enable custom plugins/middlewares for Traefik

[migueleliasweb]

After seeing the Go1.8 new plugin feature I though that this could help a lot o people to add specific functionalities to Traefik.

Instead of building/compiling/shipping a custom-made version of Traefik to enable a custom functionality it would be possible to write way simpler custom-made middlewares with this approach, doesn't it ?

Try imagine creating a package that receives the request at a parameter without having to recompile the whole Traefik repository just to add a small change. Does it sounds like a middleware ? Because for me it is ! It's just a go1.8-plugin-based-middleware !

What do you guys think ?

[timoreimann]

@migueleliasweb thanks for the suggestion. That does sound like a pretty interesting idea.

I haven't worked with Go's new plugin framework yet, so I can't judge on what it's like to work with it from a practical point of view. One concern I'm seeing is that right now, plugins are only supported on Linux. While this should probably not be too much of an issue when running Traefik in production (most people supposedly use Linux or Docker anyway), it's going to make development on any other platform much harder. You'd have to revert to tricks like running your builds off of a Docker container with the source code being bind-mounted in.

I'm not saying that we shouldn't do it because of these restrictions, but that we have to make sure the development workflow is acceptably usable for people not running Linux natively.

[migueleliasweb]

Hey @timoreimann ! I'm aware of the possible limitation since the Go plugin feature is still in it's early phase of development but soon I hope this feature will be released for all platforms and architectures.

Just keep in mind this is a cool feature and this could lead a much faster development of modules and extra features for traefik from the community ;D.

[timoreimann]

Full ACK, the potential is pretty huge.

[sandstrom]

Plugins to transform request/response would be great! The ability to adjust headers (add/remove/edit) alone would go a long way.

[prologic]

Is anybody working on this? Do we have a set of requirements?

[jntakpe]

At my company we are considering several API gateways and this feature is a must have

[tscibilia]

I would love to see some development of a WAF or fail2ban or some sort of DOS security from attackers.

[SuperSandro2000]

@tscibilia you can always install fail2ban on the host machine

[letian0805]

I am using caddy because it supports custom plugins. But I really like some of the features provided by traefik, it would be better if traefik could support custom plugins.

[canselcik]

This would be huge. I am thinking something between Apache Traffic Server and Proxygen but in Go.

[zlepper]

I'm willing to take a shot doing a transform request/response middleware, if there is some sort of requirements for it? I assume with the new middleware infrastructure, it should be relatively easy to do.

Mostly i would assume what can be done in Microsoft IIS URL rewrite would be enough: Replacing using regexes, and their match groups. And considering that Go already has pretty good support for regexes, that should be relatively trivial.

[prologic]

I'm willing to take a shot doing a transform request/response middleware, if there is some sort of requirements for it? I assume with the new middleware infrastructure, it should be relatively easy to do.

Mostly i would assume what can be done in Microsoft IIS URL rewrite would be enough: Replacing using regexes, and their match groups. And considering that Go already has pretty good support for regexes, that should be relatively trivial.

As far as I'm concerned one just needs to use the Go plugin support that was introduced in the compiler some time go. Plugins then should implement the Middleware interface that already exists in Traefik.

The hard part is actually loading the plugins, discovering them and calling them.

[emilevauge]

@prologic FYI, we have been exploring this way for a long time (#1370, then #1865), and we concluded that plugins support introduced in Go was not ready yet. And sadly, it will probably never be.
But we have been working on something else in the meantime. Stay tuned ;)

[e-nikolov]

Native Go plugins have a lot of gotchas at the moment. We use them at work and had to come up with a whole lot of workarounds and our own toolchain around them.

Just to list a few issues:

1. Plugins don't work on windows (perhaps doesn't matter since you can use them in a docker container)
2. The base application and all plugins need to be compiled with the same version of Go
3. If the plugins and the base have shared dependencies, they need to be compiled with the exact same version of said dependency.
4. plugin: gopath type != vendor type golang/go#18827 - if both the base and the plugin vendor a shared dependency, those will be considered as different packages because for Go, those two packages are different:
   $GOPATH/path-to-base/vendor/path-to-package
$GOPATH/path-to-plugin/vendor/path-to-package

Points 3. and 4. have two implications:

• if the base application and a plugin need to share a type that isn't in the standard library, but in a separate package that is a shared dependency, then that package cannot be vendored in neither the base nor the plugin and must be taken from the $GOPATH
• If both the base and a plugin independently vendor the same dependency (but don't care about communicating with types from it), and that dependency does some "init()" magic to set itself up, it could panic because that init will be executed twice (e.g. net/trace or net/http - x/net/trace: init registers an endpoint on the http.DefaultServeMux, causing a panic if vendored in some cases golang/go#24137)

Additionally, it is still unclear to me if the introduction of go modules would fix some of those issues or make them worse (mostly because I've been too scared to test).

[santiagopoli]

The Lua solution seems OK. Even if it adds a little latency, is an opt-in feature. I also think that is better to have a working solution rather than no solution at all, since a lot of people are asking for this feature. I have yet to test @negasus solution, but It looks great!

Best case scenario would be to allow creating middlewares in both scripts (Lua) and webhooks (GRPC, HTTP).

[bitsofinfo]

def would like this for integrating w./ something like modsecurity

[guilhermeaiolfi]

When we talk about plugins/middlewares, will it be possible for me to create a middleware that query a database and based on that information decides which backend to use?

I would like to use traefik to determine which docker container to use depending on the client accessing. For example, client-1 would access application container v1.0 and client-2 would access v2.0, which way to go depending on the data stored in the database.

Is there a workaround for that scenario currently?

[torarnv]

Here's one use-case for a middleware that has a lua solution for ngnix: Working around Safari's lack of client certificate support for websockets. Hopefully it would be possible to do with traefik's future plugin system:

local HMAC_SECRET = "hunter2"
local crypto = require "crypto"
 
function ComputeHmac(msg, expires)
  return crypto.hmac.digest("sha256", string.format("%s%d", msg, expires), HMAC_SECRET)
end
 
verify_status = ngx.var.ssl_client_verify
 
if verify_status == "SUCCESS" then
  client = crypto.digest("sha256", ngx.var.ssl_client_cert)
  expires = ngx.time() + 3600
 
  ngx.header["Set-Cookie"] = {
    string.format("AccessToken=%s; path=/", ComputeHmac(client, expires)),
    string.format("ClientId=%s; path=/", client),
    string.format("AccessExpires=%d; path=/", expires)
  }
  return
elseif verify_status == "NONE" then
  client = ngx.var.cookie_ClientId
  client_hmac = ngx.var.cookie_AccessToken
  access_expires = ngx.var.cookie_AccessExpires
 
  if client ~= nil and client_hmac ~= nil and access_expires ~= nil then
    hmac = ComputeHmac(client, access_expires)
 
    if hmac ~= "" and hmac == client_hmac and tonumber(access_expires) > ngx.time() then
      return
    end
  end
end
 
ngx.exit(ngx.HTTP_FORBIDDEN)

https://blog.christophermullins.com/2017/04/30/securing-homeassistant-with-client-certificates

[torarnv]

But we have been working on something else in the meantime. Stay tuned ;)

@emilevauge Can you expand a bit on the plan you hinted at earlier? -^

Seems there are a few options:

• Adding custom middelware code and building trafik yourself
  • This is what Caddy does as far as I can tell
• A middleware that supports a scripting language (Lua e.g.)
  • Would allow transitioning existing scripts for e.g ngnix
• A middleware that exposes some kind of RCP mechanism

[andrewsav-bt]

There is Hashicorp gRPC based plugin system here

[ldez]

@andrewsav-datacom #2362

[AndrewSav]

@ldez ah, so you already know.

[thewilli]

as @dduportal didn't like my idea of having a dedicated issue #5591 to keep this one clean, let me allow to present you my proposal on how to move things forward before (if ever) a final, all-satisfying solution to introduce plugins can be found.

Issue

We want to extend Traefik, but an all-agreed plugin system seems far away. That's why additional plugins can only be compiled with Traefik. As adding new plugins requires code changes of Traefik itself, it is hard to maintain them or for users to have several of those plugins merged together.

Proposal

Restructure Traefik, so additional Middlewares (defined as Go Modules) can be compiled alongside Traefik without having to touch Traefik's source files. This would allow to automate custom Traefik builds with a custom set of builtin Middlewares (e.g. the Lua one) even from different sources, and for Middleware developers to maintain their code independent of Traefik. Please see #5591 for additional (high-level) details.

What do you think?

[negasus]

I will try make grpc plugin for traefik2 just for fun. Its interesting

[agjini]

Hello what can we do to have this feature added to traefik? Whathever the implementation is, to be able to execute a custom action from a middleware to interact with a request. Maybe some tests have been made with lua or go or grpc. The first use case I see for this kind of custom middleware is a WAF (i.e. calling mod security) which I admit I have no idea how it is pluggable.
I propose my help to get this feature a available but I need status on the actual progress status of this issue.

Thanks!

[negasus]

Meanwhile i did update a repo https://github.com/negasus/traefik2-luascript for latest version of Traefik2 :)

[danmx]

I saw that envoy supports web assembly extensions. Maybe it would be a good idea to use this instead language specific plugins (Go, Lua, etc.). There are some wasm VMs written in Go e.g. https://github.com/perlin-network/life

[Vad1mo]

one could even go a step further with WebAssembly packaged plugins similar to https://webassemblyhub.io/

[ionutzp]

currently using an open source api gateway that has the option of adding middlewares via go plugins built in: https://www.krakend.io/docs/extending/writing-plugins/
having a bit of trouble with their architecture atm, but i think it's a good idea to be able to add middlewares without having to recompile the binary.

[AndrewSav]

Latest new from containous on the topic: https://community.containo.us/t/custom-middleware-for-validating-jwt/5806/4?u=zespri