[feature] Support self compiling workflow with custom middlewares (high memory and cpu usage) #12213
Description
Welcome!
- Yes, I've searched similar issues on GitHub and didn't find any.
- Yes, I've searched similar issues on the Traefik community forum and didn't find any.
What did you expect to see?
Feature request
Document and add out of the box support to make it easier to compile custom/community middleware with core itself.
Motivations
This is my story. We've had memory and performance issues with Traefik since day 1. The problem was clearly middlewares (using Yaegi not WASM), so I thought to attempt at profiling without success because of this:
Turns out (unless someone shines some light on how to do it) that it is almost impossible to profile middleware code that was interpreted by Yaegi.
Next move was to compile Traefik with the middlewares embedded (compiled with core itself). Not too difficult. A small patch to core itself, a few go package redirections, and we were all set to deploy and profile to find the performance issues.
Suprise.... memory issues went away:
CPU usage improved notably:
All without changing a single line of code in the middlewares themselves.
Yes of course we profiled after that and found a few minor leaks and issues with the middlewares, but I was not expecting Yaegi to be the culprit of poor performance. Yaegi plugins might be OK for code that runs every once in a while, but for middlewares that get hammered with thousands of requests per second it might not be the best option.
The patches required to adapt core to hardcode middleware integrations I used are here:
https://github.com/david-garcia-garcia/traefik-with-plugins/tree/main/traefik/embedded-plugins
Feature details
- Make it easier to compile traefik with custom middlewares. I am not 100% sure that a zero patch approach could be accomplished by using reflection to introspect the middleware constructors, and have the middleware info be passed through config. As per my patches, this might be feasible.
- Document this