Traefik does not register backend services when any service has a configuration error

[littleninja]

Do you want to request a feature or report a bug?

BUG

What did you do?

1. Deploy a service (BorkService) to Service Fabric cluster with an invalid ServiceManifest.xml

<StatelessServiceType ServiceTypeName="BorkServiceType" UseImplicitHost="true">
  <Extensions>
    <Extension Name="Traefik">
      <Labels xmlns="http://schemas.microsoft.com/2015/03/fabact-no-schema">
        <Label Key="traefik.frontend.rule.borkservice">PathPrefixStrip: /bork</Label>
        <Label Key="traefik.enable">true</Label>
        <Label Key="traefik.frontend.entryPoints">["https","http"]</Label>
      </Labels>
    </Extension>
  </Extensions>
</StatelessServiceType>

2. Deploy another service (GoodService) to cluster with a valid ServiceManifest.xml
3. Deploy Traefik to cluster.

What did you expect to see?

Traefik should register GoodService but not BorkService.

What did you see instead?

Neither backend service registers. Traefik Dashboard UI is entirely blank except for header. Errors are not visible unless viewing Traefik logs on server.

Output of traefik version: (What version of Traefik are you using?)

1.6.0

What is your environment & configuration (arguments, toml, provider, platform, ...)?

Provider: Service Fabric v6.1.480.9494
Platform: Windows Server 2016 Datacenter

################################################################
# Global configuration
################################################################

InsecureSkipVerify = true

# Enable debug mode
#
# Optional
# Default: false
#
debug = true

# Traefik logs file
# If not defined, logs to stdout
#
# Optional
#

[traefikLog]
  filePath = "traefik.log"

# Log level
#
# Optional
# Default: "ERROR"

logLevel = "ERROR"

# Entrypoints to be used by frontends that do not specify any entrypoint.
# Each frontend can specify its own entrypoints.
#
# Optional
# Default: ["http"]
#
defaultEntryPoints = ["https","http"]

# Entrypoints definition
#
# Optional
# Default:
[entryPoints]
  [entryPoints.http]
  address = ":30080"
  [entryPoints.https]
  address = ":30443"
    [entryPoints.https.proxyProtocol]
      insecure = true
    [entryPoints.https.tls]
      [entryPoints.https.tls.ClientCA]
      optional = true
      [[entryPoints.https.tls.certificates]]
      certFile = "localhost.cert"
      keyFile = "localhost.key"
[entryPoints.traefik]
address = ":38080"


################################################################
# API definition
################################################################

[api]
  # Name of the related entry point
  #
  # Optional
  # Default: "traefik"
  #
  entryPoint = "traefik"

  # Enabled Dashboard
  #
  # Optional
  # Default: true
  #
  dashboard = true

################################################################
# Service Fabric provider
################################################################

# Enable Service Fabric configuration backend
[servicefabric]

# Service Fabric Management Endpoint
clustermanagementurl = "https://localhost:19080"

# Service Fabric Management Endpoint API Version
apiversion = "3.0"

# Enable TLS connection.
#
# Optional
#
[serviceFabric.tls]
  cert = "traefikcert.crt"
  key = "traefikkey.key"
  insecureskipverify = true
  caoptional = true

If applicable, please paste the log output at DEBUG level (--logLevel=DEBUG switch)

time="2018-05-11T19:54:07Z" level=error msg="Provider connection error: Near line 133 (last key parsed 'frontends.frontend-fabric:/BorkService/BorkService.entryPoints'): expected a comma or array terminator ']', but got 'h' instead; retrying in 1.008273491s"

[ldez]

Thanks for your interest in Traefik !

It's the expected behavior.

You can join the Traefik community Slack

---

Note, your configuration contains errors:

InsecureSkipVerify = true

# Enable debug mode
#
# Optional
# Default: false
#
debug = true

# Traefik logs file
# If not defined, logs to stdout
#
# Optional
#

[traefikLog]
  filePath = "traefik.log"

# Log level
#
# Optional
# Default: "ERROR"

logLevel = "ERROR"

# Entrypoints to be used by frontends that do not specify any entrypoint.
# Each frontend can specify its own entrypoints.
#
# Optional
# Default: ["http"]
#
defaultEntryPoints = ["https","http"]

# ...

I remove comments:

InsecureSkipVerify = true
debug = true

[traefikLog]
  filePath = "traefik.log"
logLevel = "ERROR"
defaultEntryPoints = ["https","http"]

# ...

TOML is not base on indentation.

You must write your configuration like that:

InsecureSkipVerify = true
debug = true
logLevel = "ERROR"
defaultEntryPoints = ["https","http"]

[traefikLog]
  filePath = "traefik.log"

# ...

And the correct syntax for the entry points label is:

 <Label Key="traefik.frontend.entryPoints">https,http</Label>

[littleninja]

Thanks and good find, @ldez. We fixed our toml and redeployed but we're still seeing this issue. We think the ServiceManifest.xml caused the issue, as removing the entrypoints label allowed the backend services to register.

Edit to clarify: We expect a mis-configured service not to register, but instead it seems to block all services from registering and this seems like a bug. Could you take another look?

Here is our current toml:

################################################################
# Global configuration
################################################################

InsecureSkipVerify = true
logLevel = "ERROR"
defaultEntryPoints = ["https","http"]

#debug = true
#[traefikLog]
#  filePath = "traefik.log"

[entryPoints]
  [entryPoints.http]
  address = ":30080"
    [entryPoints.http.proxyProtocol]
      insecure = true
  [entryPoints.https]
  address = ":30443"
    [entryPoints.https.proxyProtocol]
      insecure = true
    [entryPoints.https.tls]
      [entryPoints.https.tls.ClientCA]
      optional = true
      [[entryPoints.https.tls.certificates]]
      certFile = "localhost.cert"
      keyFile = "localhost.key"
[entryPoints.traefik]
address = ":38080"

################################################################
# API definition
################################################################

[api]
  entryPoint = "traefik"
  dashboard = true

################################################################
# Service Fabric provider
################################################################

[servicefabric]
clustermanagementurl = "https://localhost:19080"
apiversion = "3.0"

[serviceFabric.tls]
  cert = "traefikcert.crt"
  key = "traefikkey.key"
  insecureskipverify = true
  caoptional = true

[littleninja]

@ldez I see your edit, thanks. The issue is this: a single misconfigured service breaks Traefik and puts all services at risk of an outage. I would expect Traefik to gracefully recover and register other backends, but that's not happening. Can we agree that sounds like a bug, is it expected behavior?

[jcleavel]

This is kind of a big deal.
Does anyone have any idea when this might get looked at?

[johnib]

I hope this behavior would change in the near future.

Applications in service fabric can be unrelated to one another, developed by different teams running different products.

If this behavior remains intact then it puts me in position where my availability depends on others configuring their apps right.
The more apps running on the cluster - the less available I am.

We would be happy to have this behavior modified.

[dzoech]

Any updates on this issue?

[iambryancs]

Experienced the same running multiple services via stack on a swarm.
Two services were mis-configured and have the same traefik port.

[emil-nasso]

We have run into this issue too.

We deployed a new service with a bad configuration. The name of the router/service was missing, ie: traefik.http.services..loadbalancer.server.port=80. This is just plain broken and a mistake from our side, can't blame traefik for that. 😄

The provider scans all services and when it finds this error, it stops, not providing configuration for any services.

In the perfect world, it would be nice if the provider looked at the labels for each service individually and if a syntax error was detected on one of the services was detected, it would be skipped but the other services would still be considered valid.

If that was the case our cluster would still be up and running with almost all our services and only the misconfigured service would be down (we can't expect traefik to read minds, after all 😅 ).

Are you able to confirm if there are any limitations that forces this behavior or if this should be considered a feature request or a bug @ldez ? This would be a great addition to an already rock solid product. 🙂

Can I provide any more information?