High memory usage when using custom error pages

[aisbergg]

Do you want to request a feature or report a bug?

Bug

What did you do?

• Running Traefik as a reverse proxy in front of containerized micro services
• Traefik itself is run in a Docker container as part of the Docker Swarm (only one node)
• creating some traffic (140MBit/s) by downloading data

What did you expect to see?

• without load the memory footprint is around ~30MB
• when some traffic is generated it should not peek as much as it does

What did you see instead?

• very high memory usage (450MB) until the memory limit of the cgroup was reached and traefik got killed by the OOM killer

Output of traefik version: (What version of Traefik are you using?)

Version:      v1.7.7
Codename:     maroilles
Go version:   go1.11.4
Built:        2019-01-08_10:21:03AM
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

{
 "LifeCycle": {
  "RequestAcceptGraceTimeout": 0,
  "GraceTimeOut": 0
 },
 "GraceTimeOut": 0,
 "Debug": false,
 "CheckNewVersion": false,
 "SendAnonymousUsage": false,
 "AccessLogsFile": "",
 "AccessLog": null,
 "TraefikLogsFile": "",
 "TraefikLog": null,
 "Tracing": null,
 "LogLevel": "INFO",
 "EntryPoints": {
  "dashboard": {
   "Address": "xxxx",
   "TLS": {
    "MinVersion": "VersionTLS12",
    "CipherSuites": [],
    "Certificates": null,
    "ClientCAFiles": null,
    "ClientCA": {
     "Files": null,
     "Optional": false
    },
    "DefaultCertificate": null,
    "SniStrict": false
   },
   "Redirect": null,
   "Auth": {
    "basic": {}
   },
   "WhitelistSourceRange": null,
   "WhiteList": {},
   "Compress": false,
   "ProxyProtocol": null,
   "ForwardedHeaders": null
  },
  "healthcheck": {
   "Address": "xxxx",
   "TLS": null,
   "Redirect": null,
   "Auth": null,
   "WhitelistSourceRange": null,
   "WhiteList": null,
   "Compress": false,
   "ProxyProtocol": null,
   "ForwardedHeaders": null
  },
  "http": {
   "Address": "xxxx",
   "TLS": null,
   "Redirect": null,
   "Auth": null,
   "WhitelistSourceRange": null,
   "WhiteList": null,
   "Compress": true,
   "ProxyProtocol": null,
   "ForwardedHeaders": {
    "Insecure": false,
    "TrustedIPs": []
   }
  },
  "https": {
   "Address": "xxxx",
   "TLS": {
    "MinVersion": "VersionTLS12",
    "CipherSuites": [],
    "Certificates": null,
    "ClientCAFiles": null,
    "ClientCA": {
     "Files": null,
     "Optional": false
    },
    "DefaultCertificate": null,
    "SniStrict": false
   },
   "Redirect": null,
   "Auth": null,
   "WhitelistSourceRange": null,
   "WhiteList": null,
   "Compress": true,
   "ProxyProtocol": null,
   "ForwardedHeaders": {
    "Insecure": false,
    "TrustedIPs": []
   }
  }
 },
 "Cluster": null,
 "Constraints": [],
 "ACME": {
  "Email": "xxxx",
  "Domains": null,
  "Storage": "xxxx",
  "StorageFile": "",
  "OnDemand": false,
  "OnHostRule": false,
  "CAServer": "",
  "EntryPoint": "xxxx",
  "KeyType": "xxxx",
  "DNSChallenge": null,
  "HTTPChallenge": null,
  "TLSChallenge": null,
  "DNSProvider": "",
  "DelayDontCheckDNS": 0,
  "ACMELogging": false,
  "OverrideCertificates": false,
  "TLSConfig": null
 },
 "DefaultEntryPoints": [
  "http",
  "https"
 ],
 "ProvidersThrottleDuration": 2000000000,
 "MaxIdleConnsPerHost": 200,
 "IdleTimeout": 0,
 "InsecureSkipVerify": true,
 "RootCAs": null,
 "Retry": {
  "Attempts": 20
 },
 "HealthCheck": {
  "Interval": 60000000000
 },
 "RespondingTimeouts": {
  "ReadTimeout": 0,
  "WriteTimeout": 0,
  "IdleTimeout": 360000000000
 },
 "ForwardingTimeouts": {
  "DialTimeout": 30000000000,
  "ResponseHeaderTimeout": 0
 },
 "AllowMinWeightZero": false,
 "KeepTrailingSlash": false,
 "Web": null,
 "Docker": {
  "Watch": true,
  "Filename": "",
  "Constraints": null,
  "Trace": false,
  "TemplateVersion": 0,
  "DebugLogGeneratedTemplate": false,
  "Endpoint": "xxxx",
  "Domain": "",
  "TLS": null,
  "ExposedByDefault": false,
  "UseBindPortIP": false,
  "SwarmMode": true,
  "Network": "",
  "SwarmModeRefreshSeconds": 20
 },
 "File": {
  "Watch": true,
  "Filename": "",
  "Constraints": null,
  "Trace": false,
  "TemplateVersion": 0,
  "DebugLogGeneratedTemplate": false,
  "Directory": "",
  "TraefikFile": ""
 },
 "Marathon": null,
 "Consul": null,
 "ConsulCatalog": null,
 "Etcd": null,
 "Zookeeper": null,
 "Boltdb": null,
 "Kubernetes": null,
 "Mesos": null,
 "Eureka": null,
 "ECS": null,
 "Rancher": null,
 "DynamoDB": null,
 "ServiceFabric": null,
 "Rest": null,
 "API": {
  "EntryPoint": "dashboard",
  "Dashboard": true,
  "Debug": false,
  "CurrentConfigurations": null,
  "Statistics": null
 },
 "Metrics": null,
 "Ping": {
  "EntryPoint": "healthcheck"
 },
 "HostResolver": null,
 "ConfigFile": "/container/traefik.toml"
}

The Configuration on each Docker Container looks like:

labels:
  - traefik.enable=true
  - traefik.docker.network=xxx
  - traefik.port=8080
  - traefik.weight=10
  - traefik.frontend.errors.network.backend=error
  - traefik.frontend.errors.network.query=/{status}.html
  - traefik.frontend.errors.network.status=400,401,405,408,429,500-599
  - traefik.frontend.headers.STSIncludeSubdomains=true
  - traefik.frontend.headers.STSPreload=true
  - traefik.frontend.headers.STSSeconds=315360000
  - traefik.frontend.passHostHeader=true
  - traefik.frontend.priority=10
  - traefik.frontend.redirect.entryPoint=https
  - traefik.frontend.redirect.permanent=true
  - traefik.frontend.rule=Host:xxx
  - traefik.backend.loadbalancer.method=drr
  - traefik.backend.loadbalancer.stickiness.cookieName=xxx
  - traefik.backend.loadbalancer.stickiness=true
  - traefik.backend.loadbalancer.swarm=true

---

I will try to create a pprof statistic. I have to check how this is done first.

[aisbergg]

I tested it on my local machine and the result was the same. When serving big files with Nginx as a webserver and Traefik in front of it, then traefik will kneel down. The CPU usage is very high and the memory usage peaked at 2 GB. The reason seems to be the custom error pages that I defined on every webservice. Without the custom error page definition the memory stays as it is. Here are the pprof samples: pprof.zip

If you need a setup to try it yourself, then I provide you with a vagrant setup that includes everything needed.

The heap visualized:

[jlevesy]

Hello @aisbergg,

Thanks for opening the issue and for the insightful pprof sample, I'm having a look at this right now !

[jlevesy]

Hello @aisbergg,

Can you try to disable the custom error pages for your backend and see if it improves the situation ?

Thanks

[aisbergg]

Yes, without the error pages on a Docker based service the memory issue does not occur.
I created a little test environment that demonstrates the issue: https://github.com/Aisbergg/Traefik-memory-usage-demonstration . The README.md will tell you more about what I've done.