New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ACME] How to bootstrap using KV? #927

Closed
Mika56 opened this Issue Dec 2, 2016 · 22 comments

Comments

Projects
None yet
@Mika56
Copy link

Mika56 commented Dec 2, 2016

When bootstraping a new Traefik cluster using a KV store to store configuration and ACME certificates, is there any way to correctly bootstrap the KV store?
When I started my cluster, I had to launch a Traefik not connected to the KV store, then use the storeconfig subcommand to upload my account certificate.

If a way already exist, I was not able to find it in doc. If it isn't documented, please use this ticket to track the documentation update.
If Traefik is expected to handle this correctly, we can use this ticket to try to understand why it didn't in my case.

@Mika56

This comment has been minimized.

Copy link
Author

Mika56 commented Dec 2, 2016

I've seen (and used) the storeconfig command, but you have to start a single Traefik instance and let it generate the ACME config, which isn't documented.
To be clear, what I mean is that if you start a Traefik cluster with the following keys:

and don't have the acme/account/object set, none of the Traefik instances will generate an account and store it there, at least that's what happened with me.

@Shark

This comment has been minimized.

Copy link

Shark commented Dec 18, 2016

Can confirm this. I also had to manually remove the traefik/acme/storageFile key (had an empty value) created by storeconfig. I didn't set storageFile in my traefik.toml. It worked after I removed said key.

The relevant log output for reference:

time="2016-12-18T12:14:31Z" level=info msg="Starting server on :80"
time="2016-12-18T12:14:31Z" level=info msg="Starting server on :4443"
time="2016-12-18T12:14:33Z" level=error msg="Error creating TLS config Empty Store, please provide a key for certs storage"
time="2016-12-18T12:14:33Z" level=fatal msg="Error preparing server: Empty Store, please provide a key for certs storage"
@Berndinox

This comment has been minimized.

Copy link

Berndinox commented Feb 23, 2017

Can confirm, you have to remove the storagefile key.

@CyrilPeponnet

This comment has been minimized.

Copy link

CyrilPeponnet commented Feb 23, 2017

Same here I had to remove the storagefile key.

@ArchiFleKs

This comment has been minimized.

Copy link

ArchiFleKs commented Feb 24, 2017

Same here, it is working now thanks

@borsboom

This comment has been minimized.

Copy link

borsboom commented Mar 12, 2017

This caused me so much frustration until I figured out the right steps to bootstrap both the K/V store and ACME together in an HA configuration for use as a Kubernetes Ingress controller. What I actually wanted was to just store the ACME certificates in the KV store, but not the configuration, but that doesn't seem to be possible, and it took me a while to figure that out.

In any case, this is what I'm doing now. The info below isn't new and can be found in this issue, #725, and the documentation, but I'm just spelling out in case it helps anyone else:

  1. Create traefik.toml with the K/V configuration and ACME configuration (including storage = "traefik/acme/account"). All subsequent commands use this same traefik.toml. This is what mine looks like:
    debug = true
    defaultEntryPoints = ["http","https"]

    [entryPoints]
      [entryPoints.http]
      address = ":80"
        [entryPoints.http.redirect]
        entryPoint = "https"
      [entryPoints.https]
      address = ":443"
        [entryPoints.https.tls]

    [kubernetes]

    [consul]
    endpoint = "consul:8500"

    [web]

    [acme]
    email = "censored@example.com"
    storage = "traefik/acme/account"
    entryPoint = "https"
    acmeLogging = true
    onDemand = false
    onHostRule = true
  1. Run traefik storeconfig to upload the of the configuration to the K/V store. I used a Kubernetes Job similar to that in #725 (comment).

  2. Remove the traefik/acme/storagefile key from the K/V store (e.g. consul-cli kv delete traefik/acme/storagefile).

Then you can run traefik normally in an HA configuration.

If you miss step 2, the very first run of traefik works fine, but any subsequent runs get this error (this confused me so much because it worked once and then didn't work again and it wasn't at all clear why it was ignoring the ACME configuration that I was providing in the config file):

level=error msg="Error creating TLS config Unknown entrypoint for ACME configuration"
level=fatal msg="Error preparing server: Unknown entrypoint for ACME configuration"

If you miss step 3, you get this error:

level=error msg="Error creating TLS config Empty Store, please provide a key for certs storage"
level=fatal msg="Error preparing server: Empty Store, please provide a key for certs storage"

@pickfire

This comment has been minimized.

Copy link

pickfire commented Mar 31, 2017

It works after removing traefik/acme/storageFile key from consul manually after storeconfig.

@bitsofinfo

This comment has been minimized.

Copy link
Contributor

bitsofinfo commented Dec 11, 2017

I see the same results as @pickfire and have experienced the same as @borsboom (frustration)

For me the process is:

  1. curate your TOML w/ acme and storage (no storageFile)
  2. storeconfig for the config.
  3. delete the auto-generated storageFile entry .
  4. Now start up Traefik pointing to your KV, all works fine.
@Overbryd

This comment has been minimized.

Copy link

Overbryd commented Dec 12, 2017

I cannot get traefik with etcd backend running here.

traefik version 1.5.0-rc2 against etcd v3 with useAPIV3 = true.

I followed the process @bitsofinfo described above, but still get the following error when starting traefik:

$ traefik --web --kubernetes --etcd=true --etcd.endpoint=etcd-cluster-client:2379 --etcd.useAPIV3=true
time="2017-12-12T14:21:02Z" level=error msg="Error creating TLS config: Key not found in store"
time="2017-12-12T14:21:02Z" level=fatal msg="Error preparing server: Key not found in store"

I have deleted the traefik/acme/storagefile key in etcd prior to starting traefik.

@Berndinox

This comment has been minimized.

Copy link

Berndinox commented Dec 12, 2017

Some time ago, but i can remember a bug where you have to remove a space in the ETCD Database of ACME... may you can find something with that peace of information. BR

@Overbryd

This comment has been minimized.

Copy link

Overbryd commented Dec 14, 2017

I mean, jokes aside, how do I get traefik in a h/a + acme configuration running?

I cannot be that this project is solely used by small heroku like projects that run behind a single reverse proxy? What is happening?
My user journey with this project so far is a very frustrating one.
Something is definitely wrong, either the project should not promote to support any number of backends or it should fix every backend it promotes.

If there is something I can help with (testing wise) I am happy to help. My Go skills are not sufficient to look into the code.
I do not yet want to give up on traefik, as I like its feature set and modern integration very much.

@bitsofinfo @emilevauge

@bitsofinfo

This comment has been minimized.

Copy link
Contributor

bitsofinfo commented Dec 14, 2017

@Overbryd the above works for me: #927 (comment) ... kludgy and not intuitive? yes

@igoratencompass

This comment has been minimized.

Copy link

igoratencompass commented Dec 16, 2017

@Overbryd exactly the same issue here with 1.5.0-rc2 but etcd v2 backend:

time="2017-12-16T09:16:11Z" level=error msg="Error creating TLS config: Key not found in store" 
time="2017-12-16T09:16:11Z" level=fatal msg="Error preparing server: Key not found in store" 

@bitsofinfo I followed the same steps and it is not working as you can see. Shame, I have it all working with nginx-ingress-controller and kube-lego, really trying here to give Traefik a chance but ... I think I wasted enough time with something that should be working as advertised.

@nmengin

This comment has been minimized.

Copy link
Member

nmengin commented Dec 18, 2017

@Overbryd @igoratencompass

I tried to reproduce your problem with Docker thanks to the environment given in example here but I cannot.

Can you turn your log level to Debug logLevel = DEBUG and send both the logs and your whole Træfik configuration?

Note: If you want to test the cluster example, you have to use the shell script as desbrided here :

./manage_cluster_docker_environment.sh --start --etcd3.

@igoratencompass

This comment has been minimized.

Copy link

igoratencompass commented Dec 18, 2017

@nmengin I am running in debug mode. Unfortunately for the output to be helpful these:

time="2017-12-16T09:16:11Z" level=error msg="Error creating TLS config: Key not found in store" 
time="2017-12-16T09:16:11Z" level=fatal msg="Error preparing server: Key not found in store"

should tell us exactly which key is it complaining about. Here is my log:

$ kc logs traefik-external-ingress-proxy-5b76dd8c54-btlnv
time="2017-12-18T23:02:14Z" level=warning msg="The ETCD API V2 is deprecated. Please use API V3 instead" 
time="2017-12-18T23:02:27Z" level=info msg="Using TOML configuration file /etc/traefik/traefik.toml" 
time="2017-12-18T23:02:27Z" level=warning msg="web provider configuration is deprecated, you should use these options : api, rest provider, ping and metrics" 
time="2017-12-18T23:02:27Z" level=info msg="Traefik version v1.5.0-rc2 built on 2017-12-06_03:07:42PM" 
time="2017-12-18T23:02:27Z" level=info msg="
Stats collection is disabled.
Help us improve Traefik by turning this feature on :)
More details on: https://docs.traefik.io/basic/#collected-data
" 
time="2017-12-18T23:02:27Z" level=debug msg="Global configuration loaded {"LifeCycle":{"RequestAcceptGraceTimeout":0,"GraceTimeOut":0},"GraceTimeOut":0,"Debug":true,"CheckNewVersion":true,"SendAnonymousUsage":false,"AccessLogsFile":"","AccessLog":null,"TraefikLogsFile":"","TraefikLog":null,"LogLevel":"DEBUG","EntryPoints":{"http":{"Network":"","Address":":80","TLS":null,"Redirect":{"EntryPoint":"https","Regex":"","Replacement":""},"Auth":null,"WhitelistSourceRange":null,"Compress":true,"ProxyProtocol":null,"ForwardedHeaders":{"Insecure":true,"TrustedIPs":null}},"https":{"Network":"","Address":":443","TLS":{"MinVersion":"VersionTLS12","CipherSuites":["TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA","TLS_RSA_WITH_AES_128_GCM_SHA256","TLS_RSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_128_CBC_SHA","TLS_RSA_WITH_AES_256_CBC_SHA"],"Certificates":[{"CertFile":"/ssl/tls.crt","KeyFile":"/ssl/tls.key"}],"ClientCAFiles":null,"ClientCA":{"Files":null,"Optional":false}},"Redirect":null,"Auth":null,"WhitelistSourceRange":null,"Compress":false,"ProxyProtocol":null,"ForwardedHeaders":{"Insecure":true,"TrustedIPs":null}},"traefik":{"Network":"","Address":":8080","TLS":null,"Redirect":null,"Auth":null,"WhitelistSourceRange":null,"Compress":false,"ProxyProtocol":null,"ForwardedHeaders":{"Insecure":true,"TrustedIPs":null}}},"Cluster":{"Node":"ad79ae71-1316-4b85-86e6-2392bb6b982c","Store":{"Store":{},"Prefix":"/traefik"}},"Constraints":[],"ACME":{"Email":"igorc@encompasscorporation.com","Domains":[{"Main":"virtual.local","SANs":["nodejs-app.virtual.local","encompass.virtual.local"]}],"Storage":"traefik/acme/account","StorageFile":"","OnDemand":true,"OnHostRule":true,"CAServer":"https://acme-staging.api.letsencrypt.org/directory","EntryPoint":"https","DNSProvider":"","DelayDontCheckDNS":0,"ACMELogging":true,"TLSConfig":null},"DefaultEntryPoints":["http","https"],"ProvidersThrottleDuration":2000000000,"MaxIdleConnsPerHost":200,"IdleTimeout":0,"InsecureSkipVerify":false,"RootCAs":null,"Retry":null,"HealthCheck":{"Interval":30000000000},"RespondingTimeouts":null,"ForwardingTimeouts":null,"Web":{"Address":":8080","CertFile":"","KeyFile":"","ReadOnly":false,"Statistics":null,"Metrics":null,"Path":"/","Auth":null,"Debug":false},"Docker":null,"File":null,"Marathon":null,"Consul":null,"ConsulCatalog":null,"Etcd":{"Watch":true,"Filename":"","Constraints":[],"Trace":false,"DebugLogGeneratedTemplate":false,"Endpoint":"192.168.0.151:2379,192.168.0.152:2379,192.168.0.153:2379","Prefix":"/traefik","TLS":null,"Username":"","Password":"","UseAPIV3":false},"Zookeeper":null,"Boltdb":null,"Kubernetes":{"Watch":true,"Filename":"","Constraints":[],"Trace":false,"DebugLogGeneratedTemplate":false,"Endpoint":"","Token":"","CertAuthFilePath":"","DisablePassHostHeaders":false,"EnablePassTLSCert":false,"Namespaces":null,"LabelSelector":"public-lb=traefik"},"Mesos":null,"Eureka":null,"ECS":null,"Rancher":null,"DynamoDB":null,"ServiceFabric":null,"Rest":null,"API":{"EntryPoint":"traefik","Dashboard":true,"Debug":true,"CurrentConfigurations":null,"Statistics":null,"Stats":null,"StatsRecorder":null},"Metrics":null,"Ping":{"EntryPoint":"traefik"}}" 
time="2017-12-18T23:02:27Z" level=info msg="Preparing server http &{Network: Address::80 TLS:<nil> Redirect:0xc4207e5170 Auth:<nil> WhitelistSourceRange:[] Compress:true ProxyProtocol:<nil> ForwardedHeaders:0xc4206054e0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s" 
time="2017-12-18T23:02:27Z" level=info msg="Preparing server https &{Network: Address::443 TLS:0xc4207f2d00 Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc420605500} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s" 
time="2017-12-18T23:02:27Z" level=debug msg="Add certificate for domains *.virtual.local" 
time="2017-12-18T23:02:27Z" level=info msg="Starting server on :80" 
time="2017-12-18T23:02:40Z" level=error msg="Error creating TLS config: Key not found in store" 
time="2017-12-18T23:02:40Z" level=fatal msg="Error preparing server: Key not found in store"

and here is my config:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: default

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: default

---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: traefik-external-ingress-proxy
  name: traefik-external-ingress-proxy
  namespace: default
spec:
  selector:
    app: traefik-external-ingress-proxy
  type: NodePort
  externalTrafficPolicy: Local	# Pass client ip to backends
  ports:
  - name: http
    protocol: TCP
    port: 80
    nodePort: 31285
  - name: https
    protocol: TCP
    port: 443
    nodePort: 31287
  - name: http-admin
    protocol: TCP
    port: 8080
    nodePort: 31286
  selector:
    app: traefik-external-ingress-proxy

---
# Set kubernetes.io/ingress.class to "traefik" on Ingress to select this LB
# Only Ingresses labeled with public-lb=traefik will be monitored
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: traefik-external-ingress-proxy
  namespace: default
  labels:
    app: traefik-external-ingress-proxy
spec:
  replicas: 1
  revisionHistoryLimit: 3
  template:
    metadata:
      labels:
        app: traefik-external-ingress-proxy
        name: traefik-external-ingress-proxy
    spec:
      terminationGracePeriodSeconds: 60
      serviceAccountName: traefik-ingress-controller
      containers:
      - image: traefik:v1.5.0-rc2
        name: traefik-ingress-lb
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            cpu: 200m
            memory: 2000Mi
          requests:
            cpu: 50m
            memory: 50Mi
        ports:
        - containerPort: 80
        - containerPort: 443
        - containerPort: 8080
        args:
        - --web
        - --kubernetes
        - --configfile=/etc/traefik/traefik.toml
        volumeMounts:
        - name: ssl
          mountPath: /etc/ssl
          readOnly: true
        - name: tls
          mountPath: /ssl
          readOnly: true
        - name: config
          mountPath: /etc/traefik
          readOnly: true
      volumes:
      - name: ssl
        hostPath:
          path: /etc/kubernetes/ssl
      - name: tls
        secret:
          secretName: traefik-cert
      - name: config
        configMap:
          name: traefik-external-ingress-proxy-config

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: traefik-external-ingress-proxy-config
  namespace: default
data:
  traefik.toml: |-
    logLevel = "DEBUG"
    debug = true
    defaultEntryPoints = ["http", "https"]

    [entryPoints]
      [entryPoints.http]
      address = ":80"
      compress = true
      [entryPoints.http.redirect]
      entryPoint = "https"
      [entryPoints.https]
      address = ":443"
      [entryPoints.https.tls]
      MinVersion = "VersionTLS12"
      CipherSuites = ["TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA", "TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA", "TLS_RSA_WITH_AES_128_GCM_SHA256", "TLS_RSA_WITH_AES_256_GCM_SHA384", "TLS_RSA_WITH_AES_128_CBC_SHA", "TLS_RSA_WITH_AES_256_CBC_SHA"]
      [[entryPoints.https.tls.certificates]]
      CertFile = "/ssl/tls.crt"
      KeyFile = "/ssl/tls.key"

    [etcd]
    endpoint = "192.168.0.151:2379,192.168.0.152:2379,192.168.0.153:2379"
    watch = true
    prefix = "/traefik"

    [acme]
    email = "igorc@encompasscorporation.com"
    storage = "traefik/acme/account"
    entryPoint = "https"
    onHostRule = true
    onDemand = true
    acmeLogging = true
    # For Staging, comment out to go to Prod
    caServer = "https://acme-staging.api.letsencrypt.org/directory"
    [[acme.domains]]
    main = "virtual.local"
    sans = ["nodejs-app.virtual.local", "encompass.virtual.local"]

    [web]
    address = ":8080"

    [kubernetes]
    # only monitor Ingresses with this label
    labelselector = "public-lb=traefik"

The following job was used to populate the etcdv2 store:

---
apiVersion: batch/v1
kind: Job
metadata:
  name: traefik-etcd-config
spec:
  backoffLimit: 3
  activeDeadlineSeconds: 100
  template:
    metadata:
      name: traefik-etcd-config
    spec:
      containers:
      - name: storeconfig
        image: traefik:v1.5.0-rc2
        imagePullPolicy: IfNotPresent
        args: [ "storeconfig", "-c", "/etc/traefik/traefik.toml" ]
        volumeMounts:
        - name: config
          mountPath: /etc/traefik
          readOnly: true
        - name: tls
          mountPath: /ssl
          readOnly: true
      restartPolicy: Never
      volumes:
      - name: tls
        secret:
          secretName: traefik-cert
      - name: config
        configMap:
          name: traefik-external-ingress-proxy-config
@nmengin

This comment has been minimized.

Copy link
Member

nmengin commented Dec 19, 2017

Hello @igoratencompass .

I reproduced your problem.

It's due to a problem when we migrate data from configuration file to KV store (thanks to the storeconfig command) : if the field acme.storageFile is missing, all the key needed by ACME are not created.

I will fix that as soon as possible but you can use a workaround :

  • Create a file acme.json which just contains {}
  • Add it to a configMap and mount ithis configMap into the Træfik Deployment (at the path /acme/acme.json for example)with the right 600 (needed by ACME)
  • Add the field storageFile=/acme/acme.json into the [acme] part of your Træfik configuration of your configMap traefik-external-ingress-proxy-config
  • Execute the storeconfig Job
  • Delete the traefik/acme/storageFile entry from ETCD

Don't hesitate to contact us on our Slack for more information.

@igoratencompass

This comment has been minimized.

Copy link

igoratencompass commented Dec 20, 2017

@nmengin I tried as instructed, the key I'm seeing though in the acme dir:

$ etcdctl ls traefik/acme/
/traefik/acme/caserver
/traefik/acme/dnsprovider
/traefik/acme/onhostrule
/traefik/acme/storage
/traefik/acme/storagefile
/traefik/acme/acmelogging
/traefik/acme/delaydontcheckdns
/traefik/acme/domains
/traefik/acme/email
/traefik/acme/entrypoint
/traefik/acme/ondemand

as you can see the key is not traefik/acme/storageFile but /traefik/acme/storagefile, maybe this is the reason I'm still seeing the same error? By the way this key was being created anyway even when I didn't have the storageFile = "/acme/acme.json" setting in teh ConfigMap and that's the key I was deleting.

@nmengin

This comment has been minimized.

Copy link
Member

nmengin commented Dec 20, 2017

@igoratencompass

Sorry, it was a mistake, I talked about the /traefik/acme/storagefile. I know this key is always inserted for now.

But if you don't provide the field storageFile = "/acme/acme.json", Træfik is not able to initiliaze correctly the acme keys into the KV store as you can see here.

Indeed, in the keys yous listed, /traefik/acme/account/xx keys are missing, that's why you can not start correctly your Træfik instance.

Morevover, there is a problem in the arguments given to Træfik : you have to replace

        args:
        - --web
        - --kubernetes
        - --configfile=/etc/traefik/traefik.toml

by

        args:
        - --web
        - --kubernetes
        - --etcd
        - --etcd.endPoint="192.168.0.151:2379,192.168.0.152:2379,192.168.0.153:2379"

In the way to be sure to start Træfik with configuration from KV store.

@igoratencompass

This comment has been minimized.

Copy link

igoratencompass commented Dec 20, 2017

Hi @nmengin

Yeah I changed the args but still same problem I guess due to the missing /traefik/acme/account/xx as you pointed out. Anything else that can be done?

@traefiker

This comment has been minimized.

Copy link

traefiker commented Dec 20, 2017

Closed by #2598.

@traefiker traefiker added this to the 1.5 milestone Dec 20, 2017

@traefiker traefiker closed this Dec 20, 2017

@igoratencompass

This comment has been minimized.

Copy link

igoratencompass commented Dec 20, 2017

Confirming, this is now working with v1.5.0-rc3

$ etcdctl ls /traefik/acme --recursive
/traefik/acme/acmelogging
/traefik/acme/entrypoint
/traefik/acme/ondemand
/traefik/acme/onhostrule
/traefik/acme/caserver
/traefik/acme/delaydontcheckdns
/traefik/acme/dnsprovider
/traefik/acme/domains
/traefik/acme/domains/0
/traefik/acme/domains/0/main
/traefik/acme/domains/0/sans
/traefik/acme/domains/0/sans/0
/traefik/acme/domains/0/sans/1
/traefik/acme/email
/traefik/acme/storage
/traefik/acme/account
/traefik/acme/account/lock
/traefik/acme/account/object
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment