Pilot and Hub services are down and Traefik doesn't start

[alexdelprete]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

Configured a plugin, restarted Traefik, but pilot service (hub too) is down, so Traefik doesn't start.
Duplicate of #8617 that was closed as fixed.

What did you see instead?

Duplicate of #8617 that was closed as solved.

What version of Traefik are you using?

Version:      2.8.3
Codename:     vacherin
Go version:   go1.19
Built:        2022-08-12T14:24:34Z
OS/Arch:      linux/amd64

What is your environment & configuration?

# N/A

Add more configuration information here.

If applicable, please paste the log output in DEBUG level

No response

[ldez]

Hello,

Thank you and we are working on it.

traefik/traefik-helm-chart#624

https://twitter.com/traefik/status/1560291970760822786

[alexdelprete]

Thanks Ludovic.

Is there a way to configure Traefik to start in any case? The dependency from these remote services should not affect so severely local functionality.

[ldez]

@alexdelprete a workaround can be to use local plugins: https://github.com/traefik/plugindemo#local-mode

You will have to clone plugin repos (I can help to find it) and mount that.

[alexdelprete]

Good advice, thanks a lot. And the hub service dependency is blocking like the plugins, or it would start in that case?

[ldez]

And the hub service dependency is blocking like the plugins, or it would start in that case?

I don't think so.

[alexdelprete]

I don't think so.

Ok, so I guess I'll disable it for now. Thanks.

I'll check the local plugins setup now...

[rjchicago]

We experienced this issue. We disabled plugins for now and upgraded from 2.7.0 to 2.8.3 and Traefik has come back up for us. We will look into local-mode to re-enable plugins.

[ldez]

We're currently restoring our cluster, we are doing the quickest we can.

[alexdelprete]

I don't think so.

Ludovic, the local plugin is working perfectly, this is the config I used:

experimental:
  # Enable the Traefik Hub Experimental feature.
  hub: true
  http3: true
  localPlugins:
    traefik-get-real-ip:
      moduleName: github.com/Paxxs/traefik-get-real-ip

I didn't need to disable the hub, Traefik starts anyway, and I see no errors in the log.

[ldez]

If this can help, the list of plugins:

plugin	clone
17media/plugin-allowpath	git clone git@github.com:17media/plugin-allowpath.git
23deg/jwt-middleware	git clone git@github.com:23deg/jwt-middleware.git
aarlint/pathauth	git clone git@github.com:aarlint/pathauth.git
acouvreur/traefik-modsecurity-plugin	git clone git@github.com:acouvreur/traefik-modsecurity-plugin.git
acouvreur/traefik-ondemand-plugin	git clone git@github.com:acouvreur/traefik-ondemand-plugin.git
adyanth/traefik-container-manager	git clone git@github.com:adyanth/traefik-container-manager.git
agence-gaya/traefik-plugin-blockuseragent	git clone git@github.com:agence-gaya/traefik-plugin-blockuseragent.git
alex-held/traefik-plugin-rerouter	git clone git@github.com:alex-held/traefik-plugin-rerouter.git
alexandrebouthinon/traefik-kuzzle-auth	git clone git@github.com:alexandrebouthinon/traefik-kuzzle-auth.git
alexandreh2ag/traefik-ipfilter-basicauth	git clone git@github.com:alexandreh2ag/traefik-ipfilter-basicauth.git
apwe/headerproxy	git clone git@github.com:apwe/headerproxy.git
argyle-engineering/copy-header-value-traefik-plugin	git clone git@github.com:argyle-engineering/copy-header-value-traefik-plugin.git
AustinHellerRepo/TraefikOutboundLimiter	git clone git@github.com:AustinHellerRepo/TraefikOutboundLimiter.git
axiaoxin/traefikplugindemo	git clone git@github.com:axiaoxin/traefikplugindemo.git
bay1ts/SiriusGeo	git clone git@github.com:bay1ts/SiriusGeo.git
Beanow/traefik-plugin-rawdata	git clone git@github.com:Beanow/traefik-plugin-rawdata.git
benoitg31/traefik-forced-body-plugin	git clone git@github.com:benoitg31/traefik-forced-body-plugin.git
BetterCorp/cloudflarewarp	git clone git@github.com:BetterCorp/cloudflarewarp.git
bukukasio/super-rate	git clone git@github.com:bukukasio/super-rate.git
carnage-sh/sessionmapper	git clone git@github.com:carnage-sh/sessionmapper.git
che-incubator/header-rewrite-traefik-plugin	git clone git@github.com:che-incubator/header-rewrite-traefik-plugin.git
CorentinMAG/geoip	git clone git@github.com:CorentinMAG/geoip.git
dariopb/traefikServiceFabricPlugin	git clone git@github.com:dariopb/traefikServiceFabricPlugin.git
dariusandz/header-transmute	git clone git@github.com:dariusandz/header-transmute.git
darkweak/souin	git clone git@github.com:darkweak/souin.git
dclairac/traefik-plugin-headers	git clone git@github.com:dclairac/traefik-plugin-headers.git
dgzlopes/traefik-fault-injection	git clone git@github.com:dgzlopes/traefik-fault-injection.git
Dimoniq/jwtvalidator	git clone git@github.com:Dimoniq/jwtvalidator.git
dimorder/dimdanredirect	git clone git@github.com:dimorder/dimdanredirect.git
dkijkuit/azurejwttokenvalidation	git clone git@github.com:dkijkuit/azurejwttokenvalidation.git
dkijkuit/checkheadersplugin	git clone git@github.com:dkijkuit/checkheadersplugin.git
domainesia/traefik-plugin-reformatheader	git clone git@github.com:domainesia/traefik-plugin-reformatheader.git
dragosnutu/traefik-plugin	git clone git@github.com:dragosnutu/traefik-plugin.git
EasySolutionsIO/traefikxrequeststart	git clone git@github.com:EasySolutionsIO/traefikxrequeststart.git
evolves-fr/traefik-plugin-redirect	git clone git@github.com:evolves-fr/traefik-plugin-redirect.git
fdufault/mapheaders	git clone git@github.com:fdufault/mapheaders.git
FinalCAD/TraefikRegionalPlugin	git clone git@github.com:FinalCAD/TraefikRegionalPlugin.git
florentinl/websitepathconverter	git clone git@github.com:florentinl/websitepathconverter.git
galaxias/traefik-vault-auth	git clone git@github.com:galaxias/traefik-vault-auth.git
GiGInnovationLabs/traefikgeoip2	git clone git@github.com:GiGInnovationLabs/traefikgeoip2.git
GiGInnovationLabs/traefikuseragent	git clone git@github.com:GiGInnovationLabs/traefikuseragent.git
Hongbo-Miao/traefik-plugin-disable-graphql-introspection	git clone git@github.com:Hongbo-Miao/traefik-plugin-disable-graphql-introspection.git
igoooor/conteo-traefik-cache	git clone git@github.com:igoooor/conteo-traefik-cache.git
irotem/jwt-rewrite	git clone git@github.com:irotem/jwt-rewrite.git
iYUYUE/traefik-forwarded-real-ip	git clone git@github.com:iYUYUE/traefik-forwarded-real-ip.git
jackhillman/pluginesi	git clone git@github.com:jackhillman/pluginesi.git
jerrywoo96/AddForwardedHeader	git clone git@github.com:jerrywoo96/AddForwardedHeader.git
jerrywoo96/AddMissingHeaders	git clone git@github.com:jerrywoo96/AddMissingHeaders.git
jpxd/torblock	git clone git@github.com:jpxd/torblock.git
kevtainer/denyip	git clone git@github.com:kevtainer/denyip.git
kingjan1999/traefik-plugin-exception-authbasic	git clone git@github.com:kingjan1999/traefik-plugin-exception-authbasic.git
kucjac/traefik-block-ua	git clone git@github.com:kucjac/traefik-block-ua.git
kucjac/traefik-plugin-geoblock	git clone git@github.com:kucjac/traefik-plugin-geoblock.git
kumina/headers-by-request	git clone git@github.com:kumina/headers-by-request.git
kumina/traefik-routing-plugin	git clone git@github.com:kumina/traefik-routing-plugin.git
kzmake/traefik-plugin-forward-request	git clone git@github.com:kzmake/traefik-plugin-forward-request.git
Lepkem/traefik-plugin-response-code-override	git clone git@github.com:Lepkem/traefik-plugin-response-code-override.git
lion7/traefik-jwt-headers-plugin	git clone git@github.com:lion7/traefik-jwt-headers-plugin.git
lixianliang/traefik-plugin-return-response	git clone git@github.com:lixianliang/traefik-plugin-return-response.git
LyuHe-uestc/traefik-plugin-check	git clone git@github.com:LyuHe-uestc/traefik-plugin-check.git
LyuHe-uestc/traefik-plugin-token-auth	git clone git@github.com:LyuHe-uestc/traefik-plugin-token-auth.git
mayankkumar2/traefik-plugin-securum-exire	git clone git@github.com:mayankkumar2/traefik-plugin-securum-exire.git
mrdrelar/traefik-plugin-rewriteheader	git clone git@github.com:mrdrelar/traefik-plugin-rewriteheader.git
muhgumus/traefik-token-middleware	git clone git@github.com:muhgumus/traefik-token-middleware.git
nese/forwardcookie	git clone git@github.com:nese/forwardcookie.git
nilskohrs/environmentheader	git clone git@github.com:nilskohrs/environmentheader.git
nilskohrs/headerblock	git clone git@github.com:nilskohrs/headerblock.git
nilskohrs/pathauth	git clone git@github.com:nilskohrs/pathauth.git
nilskohrs/regex2redirect	git clone git@github.com:nilskohrs/regex2redirect.git
nilskohrs/reproxied	git clone git@github.com:nilskohrs/reproxied.git
nilskohrs/stripcookie	git clone git@github.com:nilskohrs/stripcookie.git
n0m4dz/jwt-cors	git clone git@github.com:n0m4dz/jwt-cors.git
nscuro/traefik-plugin-geoblock	git clone git@github.com:nscuro/traefik-plugin-geoblock.git
packruler/rewrite-body	git clone git@github.com:packruler/rewrite-body.git
packruler/traefik-themepark	git clone git@github.com:packruler/traefik-themepark.git
PascalMinder/geoblock	git clone git@github.com:PascalMinder/geoblock.git
PavloZastavnyi/headerstransformation	git clone git@github.com:PavloZastavnyi/headerstransformation.git
Paxxs/traefik-get-real-ip	git clone git@github.com:Paxxs/traefik-get-real-ip.git
pierre-verhaeghe/traefik-replace-response-code	git clone git@github.com:pierre-verhaeghe/traefik-replace-response-code.git
pipe01/plugin-requestid	git clone git@github.com:pipe01/plugin-requestid.git
portofrotterdam/environmentheader	git clone git@github.com:portofrotterdam/environmentheader.git
portofrotterdam/headerblock	git clone git@github.com:portofrotterdam/headerblock.git
portofrotterdam/pathauth	git clone git@github.com:portofrotterdam/pathauth.git
portofrotterdam/regex2redirect	git clone git@github.com:portofrotterdam/regex2redirect.git
portofrotterdam/reproxied	git clone git@github.com:portofrotterdam/reproxied.git
portofrotterdam/stripcookie	git clone git@github.com:portofrotterdam/stripcookie.git
pvalletbo/traefik-blocklist	git clone git@github.com:pvalletbo/traefik-blocklist.git
pvalletbo/traefik-forwarded-real-ip	git clone git@github.com:pvalletbo/traefik-forwarded-real-ip.git
qxsugar/request-dispatch	git clone git@github.com:qxsugar/request-dispatch.git
qxsugar/request-mark	git clone git@github.com:qxsugar/request-mark.git
Rajabalian/ipclient	git clone git@github.com:Rajabalian/ipclient.git
rocdove/replacepathfromurlregex	git clone git@github.com:rocdove/replacepathfromurlregex.git
romracer/traefik-get-real-ip	git clone git@github.com:romracer/traefik-get-real-ip.git
Russia9/body-size-limit	git clone git@github.com:Russia9/body-size-limit.git
scrazy77/plugin-simplecache-nocache	git clone git@github.com:scrazy77/plugin-simplecache-nocache.git
snapt/traefik-nova-plugin	git clone git@github.com:snapt/traefik-nova-plugin.git
soulbalz/correlationid	git clone git@github.com:soulbalz/correlationid.git
soulbalz/traefik-check-body	git clone git@github.com:soulbalz/traefik-check-body.git
soulbalz/traefik-real-ip	git clone git@github.com:soulbalz/traefik-real-ip.git
s00rk/firebase-verify-token	git clone git@github.com:s00rk/firebase-verify-token.git
sunalwaysknows/redirect2https	git clone git@github.com:sunalwaysknows/redirect2https.git
suteqa/plugin_record	git clone git@github.com:suteqa/plugin_record.git
team-carepay/traefik-jwt-plugin	git clone git@github.com:team-carepay/traefik-jwt-plugin.git
team-carepay/traefik-opa-plugin	git clone git@github.com:team-carepay/traefik-opa-plugin.git
tnt-sbab/jwt-verifier	git clone git@github.com:tnt-sbab/jwt-verifier.git
tnt-sbab/token-translator	git clone git@github.com:tnt-sbab/token-translator.git
togettoyou/traefik-timer-plugin	git clone git@github.com:togettoyou/traefik-timer-plugin.git
tomMoulard/fail2ban	git clone git@github.com:tomMoulard/fail2ban.git
tomMoulard/htransformation	git clone git@github.com:tomMoulard/htransformation.git
traefik/plugin-blockpath	git clone git@github.com:traefik/plugin-blockpath.git
traefik/plugin-log4shell	git clone git@github.com:traefik/plugin-log4shell.git
traefik/plugin-rewritebody	git clone git@github.com:traefik/plugin-rewritebody.git
traefik/plugin-simplecache	git clone git@github.com:traefik/plugin-simplecache.git
traefik/plugindemo	git clone git@github.com:traefik/plugindemo.git
traefik/pluginproviderdemo	git clone git@github.com:traefik/pluginproviderdemo.git
trinnylondon/lowercase	git clone git@github.com:trinnylondon/lowercase.git
trinnylondon/redirector	git clone git@github.com:trinnylondon/redirector.git
trinnylondon/traefik-add-trace-id	git clone git@github.com:trinnylondon/traefik-add-trace-id.git
Uscreen-video/traefik-plugin-rewritehost	git clone git@github.com:Uscreen-video/traefik-plugin-rewritehost.git
vaspapadopoulos/traefik-cookie-handler-plugin	git clone git@github.com:vaspapadopoulos/traefik-cookie-handler-plugin.git
vidosits/header-pattern-proxy	git clone git@github.com:vidosits/header-pattern-proxy.git
vincentinttsh/cloudflareip	git clone git@github.com:vincentinttsh/cloudflareip.git
vincentinttsh/rewriteheaders	git clone git@github.com:vincentinttsh/rewriteheaders.git
virtualzone/rewriteheaders	git clone git@github.com:virtualzone/rewriteheaders.git
vtacquet/redbase-plugin	git clone git@github.com:vtacquet/redbase-plugin.git
wiltonsr/ldapAuth	git clone git@github.com:wiltonsr/ldapAuth.git
worldline-go/traefik-plugin-hello	git clone git@github.com:worldline-go/traefik-plugin-hello.git
XciD/traefik-plugin-rewrite-headers	git clone git@github.com:XciD/traefik-plugin-rewrite-headers.git
0xDagal/charon-passto	git clone git@github.com:0xDagal/charon-passto.git
ytyuanxi/userinfo	git clone git@github.com:ytyuanxi/userinfo.git
ZeroGachis/traefik-auth-middleware	git clone git@github.com:ZeroGachis/traefik-auth-middleware.git
ZeroGachis/traefik-magic-jwt	git clone git@github.com:ZeroGachis/traefik-magic-jwt.git

[alexdelprete]

If this can help, the list of plugins:

I got the git clone line from the plugin repository, but the list is helpful for unexperienced people.

Good that the hub active config is not stopping the service.

I was thinking: maybe Traefik could automatically cache a local copy of the plugin, and use that, then periodically it can refresh the local cloned repo. So you don't have to distinguish between local/remote plugins.

[ldez]

maybe Traefik could automatically cache a local copy of the plugin

Traefik works with a local copy but we have to be sure about the plugin integrity is right.
Traefik calls a service to check the integrity, if there is an integrity problem (or something else) the plugin is removed and Traefik will download again the plugin.

[alexdelprete]

Traefik works with a local copy but we have to be sure about the plugin integrity is right.

If the plugin is run at least once, you are sure the integrity was checked positively. You can do a local check for local corruption eventually, storing a hash of the local plugin. If at the next run the local hash is correct, you can start the local copy of the plugin even if the remote plugin service is down.

[ldez]

Having a local hash and local check is not secure because both can be corrupted.
If you put a lock and the key in the same place you don't have a lock 😉

I already think about that when I created the plugin system.