It would be great if Traefik supported wasm plugins

[jicvi]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you expect to see?

Envoy WASM plugins：
https://github.com/envoyproxy/envoy/tree/master/examples/wasm-cc

[tfny]

Hiya! Thanks for your suggestion. We are interested in this issue but are unsure about the use case and the traction it will receive, so we are going to leave the status as "kind/proposal" to give the community time to let us know that they would like this.

We will reevaluate as people respond.

If you or another community member would like to build it before that happens, let us know and we will work with you to make sure you have all the information needed.

Let us know here before you start. We prefer to work with our community members at the beginning of the design process to ensure that we are aligned and can move quickly with the review and merge process.

[acouvreur]

I'd be really interested about this feature too. Right now creating plugins for multiple reverse proxies is a pain because of all the different API each reverse proxy propose.

WasmProxy seems to be a very well fit usage.

[jcchavezs]

I am glad this issue already exists on this repository. I have been working with Wasm, proxy-wasm and proxies' features for a long time and I really like to see a way to extend traefik through Wasm soon.

There are a few reasons why Wasm is a good match for this use case, the first one is the fact that currently using yaegi is the only way to extend Traefik with custom functionality, however it comes with some limitations:

1. it needs to catch up with Go language syntax on every release
2. is limited to Go
3. Panics
4. Libraries incompatibility
5. Lack of profiling tooling

As opposed Wasm present several advantages on these matters:

1. Polyglot
2. Sandboxed (hence no panics)
3. More mature compatibility matrix for different libraries
4. Profiling tooling
5. More mature ecosystem, broader community

that said, we have three alternatives to pursue this goal:

1. proxy-wasm: dead standard used in envoy, focused on envoy and with unexisting community. Still there are some cool projects built on top of it, although since the standard is mainly based in Envoy, some of the features won't be portable. (Spoiler alert, I worked a lot with proxy-wasm)
2. http-wasm: new alternative with a host available in Go that bring several improvements from the main issues with proxy-wasm (less envoy focused request/response lifecycle, works as middleware and proxy feature, more performant, etc).
3. custom ABI: this is an alternative but very laborous to implement as it would require us to write an own host, define an ABI and write a TCK for host and guests SDK. While this is possible I would rather not reinvent the wheel.

I am happy to contribute with a PR supporting http-wasm. Any thoughts on this?

[emilevauge]

Thanks for your feedback @jcchavezs, very much appreciated.
http-wasm looks promising, but young, and from the author, not production ready:

The current maturity phase is early draft

But it would make things a lot easier to integrate.
@jcchavezs how far have you been experimenting with http-wasm?
Any other known alternative?

[zetaab]

At least I would like to see WASM support in Traefik. It could bring a lot more possibilities to plugin ecosystem (read: broader community and hopefully other products as well which will use WASM). If those 3 are the alternatives I would choose http-wasm and try to experiment that?

[juliens]

Some time ago, I attempted to integrate Proxy-WASM, and my experience left me somewhat dissatisfied. Regrettably, I couldn't achieve the seamless functionality of a proper Envoy plugin. It seemed like a substantial amount of effort would be necessary to address the missing components.

While I haven't explored HTTP-WASM, it appears to be a more attractive alternative. It seems relatively straightforward to integrate with Traefik.
If someone has already developed a plugin for Envoy, such as CorazaWaf, they might need to create another plugin, but I'm not sure it's a real problem.

With the Yaegi plugin mechanism, you gain extensive access to the underlying Go objects, including the complete ResponseWriter.
In theory, you could even hijack the connection, among other possibilities.
I'm curious if the HTTP-WASM "interface" will provide the required functionality for various middleware plugins. What kinds of middleware can be implemented with it? Perhaps we should consider implementing real middleware, similar to some of the existing Yaegi plugins, or even adapting the CorazaWaf middleware.

[mmatur]

Hello,

I recently conducted some tests using proxy-wasm, but unfortunately, I encountered challenges in getting it to work effectively. I share the sentiment with you, @jcchavezs, that proxy-wasm is mainly based on Envoy. I'm not sure we will be able to have current Envoy Wasm filters working with Traefik.

On a more positive note, I found http-wasm to be a much simpler and more straightforward option, particularly when it comes to integration with Traefik. In fact, I even worked on a small project to explore its capabilities, which you can find here.

I'm in agreement with @juliens about implementing a real middleware with http-wasm. While it may not cover all the use cases that Yaegi currently handles, I view it as a promising initial step. Moreover, it can seamlessly coexist with plugin middleware.

[zetaab]

@mmatur nice work! the actual change does not look even big! of course that is missing configurable parameters and kubernetes part etc. Anyway looks promising

[jcchavezs]

Really nice work @mmatur. I wonder if we could start a PR on this repo attempting to add http-wasm for traefik. There are a few things I want to discuss on this, for starting I would also suggest we look at existing impl. in dapr which also supports http-wasm (see https://github.com/dapr/docs/blob/v1.11/daprdocs/content/en/reference/components-reference/supported-middleware/middleware-wasm.md).

Some of the questions are:

• Do we support only local wasm files or also remote. If we don't support remote, are we expecting to use config map to pass the wasm binaries? Some binaries could take much longer than 1MB
• How do we pass config to the middlewares

[juliens]

@jcchavezs I'm curious if attempting to incorporate it into the existing plugin system might provide a solution. We've already established a mechanism for handling configuration, and for addressing the remote or local issue, this could potentially be a viable option. Although I'm uncertain about how straightforward it would be to integrate it into the current catalog system. Perhaps @ldez could share their perspective on this matter.

[ldez]

I can see solutions to integrate WASM plugins inside the catalog and Traefik (even if I still think that the best use of WASM is not for plugins).
It is useless to talk about that now, this is a detail that we will talk about when something will work locally inside Traefik (during the PR).

[mmatur]

@jcchavezs,

I believe Traefik should support both local and remote Wasm files. Perhaps this support could be integrated into https://plugins.traefik.io for better accessibility.

Regarding the middleware configuration, I conducted some research and found that http-wasm-host-go offers a guestConfig option, which appears suitable for passing the config to the middleware.

https://github.com/http-wasm/http-wasm-host-go/blob/4171a60674e821aa956c985fede9beb295d026c9/handler/options.go#L25-L30.

@ldez, could you please provide more details about your vision for the Wasm integration in Traefik?

(even if I still think that the best use of WASM is not for plugins)

What are your ideas for utilizing Wasm in Traefik?

[ldez]

What are your ideas for utilizing Wasm in Traefik?

This is off-topic: this topic is about WASM and plugins, we can talk about it somewhere else.

[ldez]

Maybe I was not clear: it's easy to integrate a different type of plugin if the repository follows the same convention as the existing plugins.
Some changes will be needed inside the remote plugin services and Traefik but, for now, based on the current information, I don't see any blockers.

[emilevauge]

All in all, here are important questions that still need to be answered regarding the WASM plugin implementation:

• can we reuse the same plugin middleware contract?
• can we reuse exsiting plugins?

Even if that's not the case, I think it's worth exploring this WASM plugin implementation, but this would definitely make things easier.

[jcchavezs]

Thanks for the feedback.

http-wasm plugins are agnostic so ideally you could use any http-wasm binary into traefik which in the end is beneficial for everyone and they can be hosted in traefik plugin system (if they allow to download a file) or any other artifact published somewhere.

I was thinking of something like this for syntax for using http-wasm middleware with custom binaries

# example for https://plugins.traefik.io/plugins/628c9f11108ecc83915d7772/traefik-token-middleware
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: http-wasm
spec:
    file: https://plugins.traefik.io/plugins/628c9f11108ecc83915d7772/traefik-token-middleware-wasm
    config:
        queryParam: token
        secret: secret

can we reuse the same plugin middleware contract?

I think so.

command:
  - --experimental.plugins.traefik-token-middleware.modulename=url-for-wasm-plugin
  - --experimental.plugins.traefik-token-middleware.version=v0.1.4

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: my-jwtauth
spec:
  plugin:
    traefik-token-middleware:
      queryParam: token
      secret: secret

and the traefik.yml:

displayName: Traefik Token Middleware
type: http-wasm-middleware
binary: https://github.com/corazawaf/coraza-proxy-wasm/releases/download/{version}/coraza-proxy-wasm-{version}.zip
summary: 'Verify JWT Token from query Param'

testData:
  encodedHeader:
    - "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWUsImp0aSI6ImY1NzVlNDZlLTU2ZDQtNGU0Zi04MDQ1LTcwMWJmYjViM2RmNCIsImlhdCI6MTYwODE1MDg1MSwiZXhwIjoxNjA4MTU0NDUxfQ.g_XGqgyR-vo1DZzHiw-HwPQJpJYSsYMvI9rdeMRJEg8"

can we reuse exsiting plugins?

Some work needs to happen to recompile them in wasm (using tinygo)

[ldez]

There are 4 plugin contracts:

• the repository convention
  • .traefik.yml
  • readme.md
• the programmatic "interface":
  • A type type Config struct { ... }.
  • A function func CreateConfig() *Config.
  • A function func New(ctx context.Context, next http.Handler, config *Config, name string) (http. Handler, error).
• the static configuration

experimental:
  plugins:
    example:
      moduleName: github.com/traefik/plugindemo
      version: v0.2.1
# OR
experimental:
  localPlugins:
    example:
      moduleName: github.com/traefik/plugindemo

• the dynamic configuration

  middlewares:
    my-plugin:
      plugin:
        example:
          headers:
            Foo: Bar

The repository convention should be kept to ease the integration inside the catalog.

The programmatic "interface" could be only partially different because it should follow the middleware signature.

The static configuration should be changed to use another field than moduleName to avoid confusion and ease the handling of the current plugins.

The dynamic configuration will stay the same.

---

The 2 plugin systems can live together: we don't control the code of the current plugins so we cannot really compile for them, a compilation of plugins inside the infrastructure will lead to security issues, and the 2 systems can be handled without major difficulties.

[emilevauge]

we don't control the code of the current plugins so we cannot really compile for them, a compilation of plugins inside the infrastructure will lead to security issues, and the 2 systems can be handled without major difficulties.

Wouldn't be possible to provide a cli to compile yaegi plugins to wasm binaries?

[geraldcroes]

Hey there,

Before we dig deeper and spend time finding proper implementation, I'd like to do a quick poll to understand who supports integrating WASM as a plugin engine and who is against

Please 👍 (in favor) 👎 (against) in the reaction! (I'm pinging other maintainers, making sure they see this poll as well, but anyone, feel free to add weight)

[zetaab]

I did some tests, it looks like its possible to support yaegi and wasm plugins by using same "ecosystem".

zetaab@ce47cf1

So what I did? I used yaegi "demoexample" plugin and basically little bit modified @mmatur http-wasm example.

the response

% curl -H "Host: powpow.demo.traefiklabs.tech" 127.0.0.1:8000
Hostname: 28cf73dc0e09
IP: 127.0.0.1
IP: 172.17.0.3
RemoteAddr: 172.17.0.1:37502
GET / HTTP/1.1
Host: powpow.demo.traefiklabs.tech
User-Agent: curl/8.1.2
Accept: */*
Accept-Encoding: gzip
Foo: Bar
X-Powpow: hello
X-Forwarded-For: 127.0.0.1
X-Forwarded-Host: powpow.demo.traefiklabs.tech
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: xxxx
X-Real-Ip: 127.0.0.1

like we can see, both plugins are working (foo:bar from demoexample and x-powpow:hello from wasm plugin). However, there are still some questions:

1. how we could correctly insert configuration to wasm (as guestconfig or ModuleConfig?)
2. how we could correctly insert logger to wasm (https://github.com/http-wasm/http-wasm-host-go/blob/main/handler/options.go#L41 ?)
3. Example made by @mmatur contains only request middleware, we need better example which contains also example for response. Example should contain: request, response, return access denied back to user, using logger, use some basic configuration (like in traefik demo example). Error handling in general
4. is https://github.com/http-wasm/http-wasm-host-go way to go? it says its in "early draft".

requirements for yasm plugin repo: .traefik.yml file (which contains the manifest) and then path to compiled yasm file. Or should we hardcode that to like plugin.wasm? However, its not perhaps the best to have builded artifact in git repo. Better place for instance github release as an artifact?

Or should this be even in git ecosystem? What if we have for instance S3 object storage (or any other http webserver that can server static files) which have two files: .traefik.yml and wasm file itself? Then the structure could be

https://myobjectstorage/plugin1/.traefik.yml 
https://myobjectstorage/plugin1/foobar.wasm
https://myobjectstorage/plugin2/.traefik.yml 
https://myobjectstorage/plugin2/example.wasm

in that case .traefik.yml should contain path to github repository, where people can fill issues and see the code itself.

[zetaab]

Update:

traefik code v3.0...zetaab:traefik:feature/httpwasm
plugin code https://github.com/zetaab/traefik-wasm-demo

logs from traefik:

hello from handleRequest
2023-10-22T15:01:11+03:00 DBG pkg/server/service/loadbalancer/wrr/wrr.go:188 > Service selected by WRR: 75327ea3b04bcda5
hello from handleResponse

the response

% curl -H "Host: powpow.demo.traefiklabs.tech" 127.0.0.1:8000
Hostname: 28cf73dc0e09
IP: 127.0.0.1
IP: 172.17.0.3
RemoteAddr: 172.17.0.1:48202
GET / HTTP/1.1
Host: powpow.demo.traefiklabs.tech
User-Agent: curl/8.1.2
Accept: */*
Accept-Encoding: gzip
Foo: Bar
Heh: hah
Huh: foobar
X-Forwarded-For: 127.0.0.1
X-Forwarded-Host: powpow.demo.traefiklabs.tech
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: xx
X-Real-Ip: 127.0.0.1

So what is solved here:

1. Configurations can be passed as guestconfig
2. there is logger (see logs from traefik and code from plugin). However, the problem is that we cannot just pass zerolog https://github.com/zetaab/traefik/blob/feature/httpwasm/pkg/plugins/middlewares.go#L79 because it does not implement https://github.com/http-wasm/http-wasm-host-go/blob/main/api/api.go#L59 function. I am just thinking why this logger is done like people cannot use existing logger solutions in golang? Also there is no way to panic currently https://github.com/zetaab/traefik-wasm-demo/blob/main/header.go#L25 (or exit?). If I use panic() the code will not work at all.

[zetaab]

One possible problem that I see in this format, that before this people could write http.Handler by themselves either in yaegi or in traefik middleware. However, using http-wasm this is divided to two different functions. It makes it little bit different to implement existing middlewares to it. For instance, for coraza there is http.handlers or http.handlerwrappers ready, but these cannot be used for http-wasm as-is.

Example using http-wasm:

// handleRequest implements a simple request middleware.
func (mw *Middleware) handleRequest(req api.Request, resp api.Response) (next bool, reqCtx uint32) {
	handler.Host.Log(api.LogLevelInfo, "hello from handleRequest")
	mw.Start = time.Now()
	for k, v := range mw.Config.Headers {
		req.Headers().Add(k, v)
	}
	// proceed to the next handler on the host.
	next = true
	return
}

// handleResponse implements a simple response middleware.
func (mw *Middleware) handleResponse(reqCtx uint32, req api.Request, resp api.Response, _ bool) {
	handler.Host.Log(api.LogLevelInfo, fmt.Sprintf("request took %v %v [%s] %s", time.Since(mw.Start), req.Headers().GetAll("host"), req.GetMethod(), req.GetURI()))
}

vs

func (mw *Middleware) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
	handler.Host.Log(api.LogLevelInfo, "hello from handleRequest")
	mw.Start = time.Now()
	for k, v := range mw.Config.Headers {
		req.Headers().Add(k, v)
	}

	mw.next.ServeHTTP(rw, req)

	handler.Host.Log(api.LogLevelInfo, fmt.Sprintf("request took %v %v [%s] %s", time.Since(mw.Start), req.Headers().GetAll("host"), req.GetMethod(), req.GetURI()))
}

@jcchavezs I am just thinking could there be a way to override https://github.com/http-wasm/http-wasm-host-go/blob/main/handler/nethttp/middleware.go#L111? It would be really nice to use standard http.Handler to write middlewares. It would make things much easier and people could basically copy paste things.

[jcchavezs]

Many thanks @zetaab

how we could correctly insert configuration to wasm (as guestconfig or ModuleConfig?)

yes, guestconfig it is.

how we could correctly insert logger to wasm (https://github.com/http-wasm/http-wasm-host-go/blob/main/handler/options.go#L41 ?)

Mainly adapters. Logging is a sensitive piece as it can lead to several allocations if not done carefully (see https://networkbuilders.intel.com/blog/coraza-performance-optimization).

Example made by @mmatur contains only request middleware, we need better example which contains also example for response. Example should contain: request, response, return access denied back to user, using logger, use some basic configuration (like in traefik demo example). Error handling in general

in this module we exercise most of the features: request, response access denial and mutation https://github.com/jcchavezs/coraza-http-wasm.

is https://github.com/http-wasm/http-wasm-host-go way to go? it says its in "early draft".

Yes, that is the way to go. We came up with a minimal host implementation that cover the use cases we wanted to support but projects like dapr used it and helped to improve it (see https://github.com/dapr/components-contrib/blob/master/go.mod#L71)

I am just thinking why this logger is done like people cannot use existing logger solutions in golang?

Yes, as pointed out before logging is very sensitive. Ideally we should provide a traefik implementation that implements the http-wasm logger so library maintainers can consume it.

Also there is no way to panic currently https://github.com/zetaab/traefik-wasm-demo/blob/main/header.go#L25 (or exit?). If I use panic() the code will not work at all.

What is the use case for panic? Do we want to crash traefik? I think logging and initializing nops on the interceptors is enough.

[jcchavezs]

@jcchavezs I am just thinking could there be a way to override https://github.com/http-wasm/http-wasm-host-go/blob/main/handler/nethttp/middleware.go#L111? It would be really nice to use standard http.Handler to write middlewares. It would make things much easier and people could basically copy paste things.

This is by design. Notice ABI is language/platform agnostic so designing the ABI targeting standard http.Handler isn't a goal for http-wasm. Also technically talking no wasm implementation (targeting WASI) can support net/http package so that is also a technical reason to not to do this. Finally, net/http is very opinionated and works for go but not for a general purpose cross language, notice now traefik can benefit from libraries written in ANY language and compiled to wasm.

[zetaab]

What is the use case for panic? Do we want to crash traefik? I think logging and initializing nops on the interceptors is enough.

I am just thinking if for instance parsing config fails here https://github.com/zetaab/traefik-wasm-demo/blob/main/header.go#L27 basically it should mean that whole plugin should not work. Now it will just print error to logs and continue running the plugin

[jcchavezs]

I am just thinking if for instance parsing config fails here https://github.com/zetaab/traefik-wasm-demo/blob/main/header.go#L27 basically it should mean that whole plugin should not work. Now it will just print error to logs and continue running the plugin

I get that but in practice what does that mean for traefik? crashing or logging and becoming a noop?.

HEADS UP: I am not saying we shouldn't support panic, we can. I am just asking what is the use case.

[zetaab]

In my opinion from Traefik point of view: it should not crash Traefik, but it could for instance make the route not working? It is little bit scary if people assumes that "yeah we do have WAF in place" but they forgot to check error logs and they have live route in production without WAF?

[jcchavezs]

We should design the behaviour here but still you can achieve the same without panic by passing a always deny hook onRequest if bootstrapping fails. I mean, ideally a wasm plugin being sandboxed (one of major benefits of Wasm runtimes) does not need to crash because that crash signal will never be bubbled up to the host so I don't see a real need for panic.

[geraldcroes]

In my opinion from Traefik point of view: it should not crash Traefik, but it could for instance make the route not working? It is little bit scary if people assumes that "yeah we do have WAF in place" but they forgot to check error logs and they have live route in production without WAF?

The philosophy we've been following until now is that when anything middleware that a router uses is malfunctionning, the whole route goes down, but the rest is unaffected.

Indeed, as you mentioned, the idea is to avoid the user thinking they're all set and secure ... when they are not (that's the same idea with authentication middleware)

[jcchavezs]

Awesome. Then we have feature definition for this.

…

On Sun, 22 Oct 2023, 19:29 Gérald Croës, ***@***.***> wrote: In my opinion from Traefik point of view: it should not crash Traefik, but it could for instance make the route not working? It is little bit scary if people assumes that "yeah we do have WAF in place" but they forgot to check error logs and they have live route in production without WAF? The philosophy we've been following until now is that when anything middleware that a router uses is malfunctionning, the whole route goes down, but the rest is unaffected. Indeed, as you mentioned, the idea is to avoid the user thinking they're all set and secure ... when they are not (that's the same idea with authentication middleware) — Reply to this email directly, view it on GitHub <#9552 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXOYAWEKDVANSNX6OH7W6TYAVJV7AVCNFSM6AAAAAASL3LEFWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZUGE2TEMBWGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[zetaab]

seems that os.Exit(1) handles that case https://github.com/zetaab/traefik-wasm-demo/blob/main/header.go#L29 it will print the error to logs and make the route unavailable because of broken middleware.

from traefik log

2023-10-22T23:14:09+03:00 ERR pkg/server/router/router.go:129 > error="wasm: error instantiating guest: module closed with exit_code(1)" entryPointName=web routerName=customer1@file

[zetaab]

@geraldcroes I think at least I have quite good picture how this could work. However, I would like to see that someone from Traefik side will implement this as PR, because you will define how you want to plugin system really work. I mean that you have catalog ecosystem etc, you will define the rules where the files can be stored and what is the structure for that.

Feel free to use my code that I did. It works, but its not the most beautiful yet!

[geraldcroes]

Hey @zetaab ! We discussed this with other maintainers during the triage process yesterday, and we would very much rather help you finish the PR than writing it ourselves (based on your work and effort).

For the integration within the rest of the plugin system (the catalog for example), we're looking into opening it to the community so that everyone can contribute as well.

If finishing the PR sounds appealing to you, please open one, and one maintainer will take the time to ramp you up and get it to the project's standards (if need be).

[traefiker]

Closed by #10189.