Allow Traefik to function in HA with Flatfile

[schemen]

Do you want to request a feature or report a bug?

Feature

What did you expect to see?

With #2940 and #3487 making it really hard to run Traefik in HA for Docker Swarm, I would love to see Traefik HA to be able to use ACME.json and a simple config file over a shared Filesystem (Which I assume most of the swarm clusters have).

Currently, I think you guys are saying that it is absolutely not recommendet to start HA (--global) as there are difficulties regarding the flat file. Because of that, I run Traefik with a single container still using that flat file.

Is there any chance that only the master of the N Traefik containers accesses the flat file RW and the others only R? Why is it that we only can use a KV store?

[dtomcej]

Hello @schemen,

The feature you are requesting is one that gets requested often.

You are correct, that we do not recommend using a flat file for multiple Traefik instances with LetsEncrypt.

There are a few reasons for this:

1. The acme.json file is not read by Traefik except on startup. It is used to provide persistence, in case of an app crash, or a reboot. Making changes to the file while Traefik is running has no effect, and will be overwritten by Traefik as data changes.
2. Since the acme.json is not read, when one instance writes a challenge token to the file, none of the other instances will be aware of this, and will fail the challenge. It is possible that the same Traefik instance handles the challenge and the subsequent LE request, in which case, the certificate will be generated, but it is somewhat of luck or chance that this succeeds.
3. Since each Traefik instance is not aware of any others, all instances will attempt to renew and update certificates at the same time. This will cause the same challenges to be overwritten, and them all to fail, and potentially be LE rate limited due to the failures.

Using a KV store solves these problems by allowing leader election to take place between the Traefik instances. This ultimately means that one instance is elected to be responsible for updating certificates, and all nodes have access to the same challenge data via KV key lookup.

We have discussed at length the pros and cons of modifying the code to allow a flat file to be used for HA, but ultimately we have decided that we will not pursue it due to being sub-optimal from a design perspective:

1. Flat files do not have a consistant form of locking. Waiting on file I/O is poor from a design perspective
2. Flat files (especially on a shared file system) do not have consistent inotify properties, to let Traefik know that updates have been made
3. Flat files have no optimization for searching/querying/etc, so using them to back an HA environment will run into performance issues in larger environments.

I hope this gives a bit more clarity to the situation.

For these reasons, I will be closing this issue. Please feel free to reach out to us on slack: traefik.slack.com if you have any further questions or concerns about this issue.

Thanks!

[schemen]

Heh! Thank you for clarifying this for me, sorry for asking a much asked Request yet again :)

I hope that we'll get updated documentation regarding Docker Swarm then, I am currently not able to get a running example sadly.

Let me know if I can help with that, somehow.