invalid middleware "default-code-server-sablier@kubernetescrd" configuration

[H4CK3R-01]

Hi :-)

Describe the bug
I followed your code server guide but it's not working.
I'm using k3s with default traefik installation.
If I deploy only the middleware without an ingress there are no errors visible in the traefik dashboard.
But if I apply the ingress with the annotation to use the middleware there is an error in traefik: invalid middleware "default-code-server-sablier@kubernetescrd" configuration: invalid middleware type or middleware does not exist.

What can be wrong?

Context

• Sablier version: 1.3.0
• Provider: kubernetes
• Reverse proxy: 2.9.4
• Sablier running inside a container? - In deployment like in the tutorial

Expected behavior
It should work, haha ;-)

Additional context

[acouvreur]

I just tried on my local machine and indeed I have the following error:

time="2023-03-30T05:57:50Z" level=info msg="Configuration loaded from flags."
time="2023-03-30T05:57:55Z" level=error msg="Plugins are disabled because an error has occurred." error="failed to download plugin github.com/acouvreur/sablier: failed to call service: Get \"https://plugins.traefik.io/public/download/github.com/acouvreur/sablier/v1.3.0\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
time="2023-03-30T06:01:58Z" level=error msg="invalid middleware \"default-code-server-sablier@kubernetescrd\" configuration: invalid middleware type or middleware does not exist" routerName=default-code-server-ingress-localhost@kubernetes entryPointName=metrics
time="2023-03-30T06:01:58Z" level=error msg="invalid middleware \"default-code-server-sablier@kubernetescrd\" configuration: invalid middleware type or middleware does not exist" entryPointName=web routerName=default-code-server-ingress-localhost@kubernetes
time="2023-03-30T06:01:58Z" level=error msg="invalid middleware \"default-code-server-sablier@kubernetescrd\" configuration: invalid middleware type or middleware does not exist" entryPointName=websecure routerName=websecure-default-code-server-ingress-localhost@kubernetes

I'll have to chat with the @traefik team

[acouvreur]

Now after a reboot of my traefik instance it works, because if you try it yourself you can actually download https://plugins.traefik.io/public/download/github.com/acouvreur/sablier/v1.3.0/.

Can you please do kubectl get middleware. Because the error you have states the the referenced middleware does not exist.

As it is a CRD by Traefik, and mentioned here https://github.com/acouvreur/sablier/blob/main/docs/guides/code-server-traefik-kubernetes.md#5-routing-code-server-through-traefik-with-the-sablier-plugin-middleware at step 5.

You should be able to create the middleware. Maybe you have forgotten or skip this step?

[H4CK3R-01]

Thank you for the quick response.
The middleware exists:

> kubectl get middlewares --all-namespaces | grep code
default       code-server-sablier       22h

> kubectl describe middlewares code-server-sablier
Name:         code-server-sablier
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  traefik.containo.us/v1alpha1
Kind:         Middleware
Metadata:
  Creation Timestamp:  2023-03-29T08:41:10Z
  Generation:          1
  Managed Fields:
    API Version:  traefik.containo.us/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:plugin:
          .:
          f:sablier:
            .:
            f:dynamic:
              .:
              f:displayName:
              f:refreshFrequency:
              f:showDetails:
              f:theme:
            f:names:
            f:sablierUrl:
            f:sessionDuration:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2023-03-29T08:41:10Z
  Resource Version:  5061624
  UID:               91e865fc-184e-4af3-ab5d-6cbe4628d517
Spec:
  Plugin:
    Sablier:
      Dynamic:
        Display Name:       Code Server Demo
        Refresh Frequency:  5s
        Show Details:       true
        Theme:              hacker-terminal
      Names:                deployment_default_code-server-deployment_1
      Sablier URL:          http://sablier:10000
      Session Duration:     30m
Events:                     <none>

In traefik's dashboard the middleware is also visible

If I remove the ingress, so that no ingress is using the middleware there is no error:

[Edmund-Blackadder]

Hi, I looked through my logs in Graylog and found the following:

message
2023-03-30T23:25:02.858101395+02:00 stdout F time="2023-03-30T23:25:02+02:00" level=error msg="Plugins are disabled because an error has occurred." error="mkdir plugins-storage: read-only file system"

So I looked online to find which folder was required and found this: traefik/traefik-helm-chart#280

Which says there should be a /plugin-storage folder.

I gave my Traefik deployment an emptydir for that /plugin-storage path. After restarting the pods I can see that the subfolders got populated with the plugin, and tested again. This time the dashboard shows that everything is OK and the plugin works as expected.

[acouvreur]

Very glad to here that!

Did you have a different initial configuration for the Guide then ?

Or did you use some custom values for the helm chart?

[H4CK3R-01]

Thank you!
I've got it working now.
I had the same issue with the read-only file system. But I missed the log entry :-(

[Edmund-Blackadder]

@acouvreur

I do not use a Helm chart for Traefik, I used the official Helm chart with helm template to create all the manifest files, while referencing a slightly modified values file. I did not change anything in the values for the storage.

This was just over a week ago, since then I also updated to Traefik 2.9.9, so maybe something changed on their end.

@H4CK3R-01

Great to hear!

I also missed that log at first, was banging my head against the wall for about a hour.