New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Caddy v2? #130
Comments
To be honest, I don't know, and I didn't spend any time to play with caddy v2 yet. I wonder if there is still value in improving this project, given that:
Could you please describe your use-case? I presume that you use this project already.
|
I do agree with you that Docker Swarm is loosing some popularity and lot of people are migrating to Kubernetes. Including me, I plan to switch to Kubernetes when Caddy will have a proper ingress controller for Kubernetes. I think it would be kind a waste of time to support the second version of Caddy because it would serve a niche of people due to the fact that the second version of Caddy still haven't any (good) plugins and the first version of Caddy is still very usable. |
I do not, I'm evaluating Caddy.
It would be for production on a single server(quadcore with 16GB RAM), only around 1k active users during peak traffic.
I do not use swarm but have kept an eye on it since release. Just using containers with a few compose configs.
Eventually yes. This is volunteer work in my spare time and the amount of terminology/knowledge required to migrate to k8s has offset that for now. At least it seems like it has a higher learning curve than getting other volunteer devs accustomed to k8s vs compose.
I have tried traefik and I like it,. but v2 does make dealing with redirect to HTTPS a tad annoying(compared to v1). Brotli support has been requested for some time, but no response from developers, OCSP stapling support is another one among a few other TLS support features, again lacking much activity from the developers currently while they bring v2 to release. Caddy afaik addresses all of that, and this project helps get a more traefik like experience, but I'd like to know if the functionality will be available with v2 release next year. I could alternatively look at just using Caddy for TLS termination and forwarding to Traefik for routing, which would also alleviate my concerns with Traefik. |
I like that swarm is that simple and just works. compose syntax is easy and I haven't found a as simple and extensive documentation for kubernetes... |
Traefik doesn't allow (or badly) HA, while Caddy supports it very well thanks to its plugins (see my blog post about that). The use of Kubernetes belongs to everyone, but I agree with @pwFoo about the simplicity of writing under Swarm. I didn't really read up on the new features of Caddy version 2 but I think it would be interesting, anyway, to port this plugin to version 2. Currently, I work with about 150 clients that I host on the same server. On the other side, I have several servers containing the main services offered (database, etc.). These were using Traefik because it initially offered a wide variety of options for easy configuration. However, nowadays and taking into account my needs, Caddy corresponds better to my expectations. I don't use K8S and my containers need to be accessible from the outside. If one day I have to switch to version 2 of Caddy, I will still have the same need, namely to be able to listen to the events of the Swarm in order to generate the associated routings. |
Was that referring to v1.x? Your issue appeared to be with Consul for K/V use(which I've also seen others have bad experiences with), and I think that only recently became available in v2(2.1)? I haven't used consul personally or HA myself, so I might not be understanding the issue right. Your blog mentioned the size limit with consul, and how Traefik was managing/storing certs in one large JSON file?(which doesn't sound like it'd be suitable with Consul, especially with the possibility of growing beyond that 512KB limit) While your solution with Caddy + Consul was better because certs were separate entries for consul to manage. I haven't looked into the topic too much, but regarding HA, on v1.x consul or not, there was issues with LetsEncrypt, that was removed from v2 and made available again, but only via EE version. HA in general though is meant to be fine for Traefik:
So your HA issue with Traefik was specifically about keeping certs in sync across Traefik instances? Or the 2nd issue raised in the blog post regarding the error, which based on the github issue linked, indicates it was more of a problem with LetsEncrypt interacting with the same instance for renewing a cert(stateful). A DNS challenge probably would have worked better than HTTP in that case? It's not clear what Caddy is doing differently here, other than the Consul plugin having might smaller entries to sync/update, which if Caddy isn't doing anything differently, may just happen to sync the change across all instances in a timely manner for the LetsEncrypt HTTP challenge to work if it hits a different instance? With v2 of Traefik, afaik it focuses on handling the data plane of a Service Mesh, while Maesh acts as a control plane. So in this case, isn't Docker Swarm providing your HA with multiple Traefik/Caddy instances? EDIT: It's now also possible to avoid Traefik EE and deal with LetsEncrypt externally, currently there are docs for setting it up with k8s. Granted, not as nice as it's one more thing to setup, however I think your blogpost misunderstood why Traefik ran into the issues when Caddy didn't? |
Just adding a response to your questionnaire to provide an extra data point.
Yes.
With Docker Swarm.
No such plans right now. It's definitely an option and I'm well aware that it won the orchestration war, but Swarm so far is more than sufficient for our limited needs and is much simpler to manage. I would want to have at least two fully-trained K8S admins on staff before moving over.
I was aware of Traefik, but we'd already had a stellar experience with Caddy in non-Docker environments and we were familiar with Caddyfiles configuration, so it was our first choice, and since it handled all our requirements fine on the first try we didn't spend time on alternatives. Having said that, in the last few months I've had two show-stopping failures - one was #104, and another was a mysterious case of the Caddy service somehow filling up the entire disk space that I can't pin down any more specifically than an unhelpful "it goes away if i restart the service by scaling it down to 0 and back up again". This is perfectly understandable for an open-source project of this scope, of course, but it does mean that I now intend to find the time to give Traefik a try. |
@lucaslorentz Docker Swarm may have less interest but it still is a huge user group. And believe me, there are tons of exact use cases like this. Secondly, your plugin still has a very interesting feature and value that is unique in the market, with the exeption of Traefik. Anyway, I'm in the difficult scenario at the moment of also trying Traefik and bagging my head against the wall for several weeks and now decided to give Caddy2 a try. Could you give an estimation on how much time you need to make this possible? It would be a real shame if something like this gets abandoned as long as Docker swarm is around, which I believe will still be for at least several years. Thanks for your great value and creating this plugin! I really hope you can find the goodwill to bring a v2 soon. If I can help you out in any way, please let me know. |
I love the power of that proxy plugin because of the configuration flexibility! It would be awesome to configure caddy webserver and caddy proxy just with docker labels! |
Agreed. I use Caddy because Traefik removing the on-demand tls feature. There's alternative OpenResty+Nginx, but I don't think it's good enough. Moving to kubernetes also might not be feasible for our team right now. Aside that, i got few understanding about this plugin, so I will probably try to update it for v2. |
+1 We use Traefik but sadly 1.7 suffers from issues with distributed stores and 2.X removed the ability to store certificates in KV Stores in favor of their Enterprise Edition so we'd very much like to move to Caddy2 and this addon would be perfect for our needs. |
Subscribing to this thread 👀 I do think this plugin is super cool, and I want to see some functionality like it for v2. Caddy 2 is extremely flexible and extensible, and although we're wrapping up the 2.0 release in the next few days or maybe 2 weeks at the most, there's still a lot we can add to it after that. (Btw, we're looking for help to finish building the ingress controller; it's not my cup of tea! Let me know if you're interested.) I might as well mention a few things while I'm here: Caddy 2 has a really flexible and intuitive API which can be used for any sort of dynamic, on-line config changes like adding upstreams and stuff: https://caddyserver.com/docs/api -- so you can implement any logic around that API without even needing a Caddy plugin.
That's quickly changing! I know of about 12-14 plugins in development currently (aside from config adapters): WebDAV handler, DynamoDB storage, Redis storage, Consul storage, several HTTP authentication/authorization plugins, git client, a distributed cache handler, secure HTTP/2 forward proxy, several DNS providers, dynamic DNS app, and several others. (That's also not counting the fact that many plugins that were necessarily plugins in v1 aren't needed in v2 at all anymore; for example, the realip, ipfilter, and cors plugins can mostly be done natively.) Give the plugin ecosystem a few months to mature -- but mature it will, as long as we keep reminding ourselves that v1 will eventually/soon be discontinued. 🙂 We're trying to encourage people to upgrade here.
I will say that is not a bad idea, to at least put Caddy in front; Caddy has objectively better TLS handling (see: recent PKI events such as extended OCSP responder outages and mass revocation events and you'll notice that only Caddy was unaffected -- among many other good reasons) and is more flexible in the long run. We've also significantly improved the on-demand TLS feature in Caddy 2.
Caddy coordinates certificate management across a cluster for all instances that are configured with the same storage backend. Deploy 2 or 20 or 2000 Caddy instances sharing the storage, and they'll share certificates, OCSP staples, and even TLS session ticket keys if you want. Take a few instances down, no problem, the site will go on. It's automatic, and not an enterprise-only feature like with Traefik. That's the primary difference. Also, as I said above, Caddy's cert logic is superior in many ways and will keep your site online when other servers... well... don't. ProposalOur goal is fewer moving parts, so less to maintain, less sunk costs, and less that can go wrong. So although Caddy 2's API can already do these kinds of dynamic changes, it would require another moving part. (The API is not bad -- it is actually very good and you can totally use it! But when possible, we can bake it in.) So, here is my proposal for how this might work in v2 without needing its API: If all you need is to dynamically update the list of proxy upstreams/backends (see the "upstreams" array), I could imagine a config where instead of specifying a static list of upstreams, you specify an upstream source. An upstream source could be a module that simply keeps the list of upstreams updated. (Under the hood, we'd refactor things so that the static list of upstreams actually is an upstream source, so the singular source of truth is the upstream source module.) One such upstream source module could use Docker labels. In other words, instead of:
you'd do something like:
This config is more static, and simply keeps the list of upstreams current according to Docker labels. I don't know how all the Docker stuff works though. So I'd need some help from any/all of you. I could help with the changes needed in Caddy's reverse proxy, you'd just have to implement the code that gets the labels from Docker, which this plugin already does I suppose. Please let me know your thoughts. |
@mholt Here are my 2 cents for the ingress controller: About the list of upstream, I don't think that would support additional configuration, like declaring a new website to be hosted and configuring http filters unrelated to proxy. Being able to configure the full caddyfile in a distributed way (in different docker services) is key for this project. Traefik have a good architecture for this, you can plug in a single or multiple source of configuration. Consul, Kubernetes, Docker and so on. And if I configure the same website on Consul and Docker, their config should be merged and it should proxy to both back-ends. I think the concept of multiple source of config should be baked in caddy, including merging config from those multiple sources. Then this plugin would only convert docker labels to caddy config. If we go down the Caddy V2 API path, and we have a cluster of caddy instances. Should we call the API in every instance to keep them in sync? Or are they able to sync configuration automatically? |
Thanks for the explanation @lucaslorentz .
I think it can be done. The admin endpoint is extensible. We just need a piece of code that lives in a Caddy module that can keep track of all the different config sources, assemble them into one structure, and call I don't know how all those tools work, but I'm willing to help people integrate the code that interacts with them into Caddy.
Caddy 2 is capable of this, but it's not yet implemented. We'll see if/when it's needed. But I think we won't need this to support auto-updating configs; as I said above, I think we can make a Caddy module that takes care of it within Caddy. |
I'm no expert in all of this, but I wanted to jump in and add my vote for these features. To be able to configure labels/env on a docker-compose file and have it automatically detect and add them to Caddy2 would be very helpful. I have been using a version of this functionality using Nginx (https://github.com/Verisage/docker-nginx-letsencrypt-odoo) But to move this to a clean simple Caddy2 setup with one container would be awesome. |
@andrewdhastings Awesome, I think we can make it happen -- we got your email and will be in touch! |
I started working on caddy v2 integration for this plugin. My plan so far is to create a caddy v2 app called "docker". That app would convert caddy labels to docker v2 json and apply them using caddy API. API urls would be configurable and would default to local API. User can chose to run docker app and http server in separate containers or together. Will push a branch when I have something more concrete. @mholt what do you think? |
@lucaslorentz I think that will work pretty well -- but as long as the docker app is generating the entire Caddy config to be used, then you can easily pass it into |
FYI, this plugin is getting some very positive attention in a reddit thread: https://www.reddit.com/r/selfhosted/comments/gj550w/been_looking_for_a_new_resvere_proxy_to_use_with/fqinyrt/ I think there'll be a lot of interest in a v2 equivalent of this; so, definitely ping me if I can help, where I am able (despite my lack of Docker knowledge). @lucaslorentz If you want an invite to our dev slack, if having that kind of feedback cycle would be helpful to you, let me know which email to invite. |
Thanks @mholt I Implemented what I described above, generating JSON and as caddy app, and learned with it. Now I'm implementing it focused on Caddyfile, like the current version. A migration path for users from caddy v1 to v2 with docker should be quite straight forward, as all existing features and basic labels will stay the same. And instead of an app, I'm doing it as a caddy command. I made good progress so far, I will probably create a PR tomorrow, then feedback will be very welcome. |
@lucaslorentz Thanks for the update -- invite sent! (Again, I don't know much about Docker, but a command sounds good to me too, since plugins can also add commands to the CLI. 👍 ) |
I'm happy to do some testing and feedback when you have something ready.
…On Sat, May 16, 2020 at 11:27 AM Matt Holt ***@***.***> wrote:
@lucaslorentz <https://github.com/lucaslorentz> Thanks for the update --
invite sent!
(Again, I don't know much about Docker, but a command sounds good to me
too, since plugins can also add commands to the CLI. 👍 )
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#130 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AHYGNQ2K2OVDOCDB2YJPRDTRR3EIPANCNFSM4J2HDDZQ>
.
--
Andrew Hastings
*The Catalyst*
Dream Mountain Services
*P:*
*M:*
(801) 770-4869
(435) 890-0526
*W:*
*E:*
DreamMtn.Services <http://dreammtn.services/>
Andrew@DreamMtn.Services <andrew@dreammtn.services>
<https://www.facebook.com/DreamMtnServices/>
<https://twitter.com/DreamMtnService>
<https://www.linkedin.com/in/andrewdhastings>
<https://www.instagram.com/dreammtnservices/>
Request a Free Odoo/ERP Consultation <http://dreammtn.services/contact-us>
|
PR is there. PR is big and has a lot of refactorings. For now, I don't have docker images for testing, once I merge the PR the images tagged with CI will have the latest changes. |
Sorry, this is still work in progress |
Now we have the first docker images for testing: Readme is also updated with new caddy v2 instructions: Keep in mind that special labels (address, targetport...) were removed. Please, play with it a bit and provide feedback. PS: we might still do major changes before releasing a final V2 version |
I was able to run some tests successfully. Super excited for this! Will be doing some more extensive testing in the coming weeks. I did have a question, is there a way to set global variables with environment or labels under the caddy service? https://caddyserver.com/docs/caddyfile/options Also, not a big deal, but I think you have your volumes and networks names swapped in this file. https://github.com/lucaslorentz/caddy-docker-proxy/blob/master/examples/efs-volume.yaml |
As noted in the PR, I think you set global options by having the labels be first, and setting them like this: Actually, looking at the test here https://github.com/lucaslorentz/caddy-docker-proxy/blob/master/plugin/caddyfile/fromlabels_test.go#L130 this looks incorrect. Global options must be within braces, like this:
So I think that might be incorrectly implemented... but I'd need to dig deeper. |
To be honest, I just saw the global block documentation for the first time: It was a coincidence I used the term global as well. And my reply on that PR is wrong. I was using the term global to refer to directives outside any server block, like:
I will adjust implementation to do:
As that's probabbly just code removal, because that would be the natural behaviour because you didn't set keys for caddy |
Where should the global parameters be defined ? Since they don't belong in the scope of a proxied container I assume you would read labels on the caddy container/service itself ? If we do that it means Caddy starts with a blank conf and serves the default caddy welcome page until the docker proxy addon kicks in and replace its caddyfile conf What we did with Traefik is pass all the global static config as parameters on Traefik itself and let it merge the dynamic config (labels) on top of that :
AFAIK Caddy don't support that kind of mechanism |
@rgdev caddy-docker-proxy/examples/example.yaml Line 21 in 63a38c9
|
I think we can probably close this issue now @lucaslorentz, what do you think? Anything remaining can be made new issues and tracked separately. Of course, we'd appreciate any testing/feedback from anyone so we can continue to improve this! 😁 |
This is perfect. Also, sorry if this is a noob question, but is there a difference between:
and
|
@andrewdhastings are you sure that's how you wanted to write? one single line? |
No, sorry, it was multiple lines but when I posted the comment it put them
on one line...
…On Tue, May 19, 2020 at 9:13 AM Paul Tötterman ***@***.***> wrote:
@andrewdhastings <https://github.com/andrewdhastings> are you sure that's
how you wanted to write? one single line?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#130 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AHYGNQ47A5XEXTEXXEFYPGTRSKOYNANCNFSM4J2HDDZQ>
.
--
Andrew Hastings
*The Catalyst*
Dream Mountain Services
*P:*
*M:*
(801) 770-4869
(435) 890-0526
*W:*
*E:*
DreamMtn.Services <http://dreammtn.services/>
Andrew@DreamMtn.Services <andrew@dreammtn.services>
<https://www.facebook.com/DreamMtnServices/>
<https://twitter.com/DreamMtnService>
<https://www.linkedin.com/in/andrewdhastings>
<https://www.instagram.com/dreammtnservices/>
Request a Free Odoo/ERP Consultation <http://dreammtn.services/contact-us>
|
Use ``` instead of just a single backtick, on their own lines before and after the code to preserve newlines. Single backticks are for inline code I updated the comment for you (click edit to see what I did 😅) Anyways, yes, I think those should be the same. Personally I'd put a space before the |
Thanks for both tips :) |
Is this project going to remain targeting v1 alone, or is there plans to support v2 (currently in Beta, aimed to release early 2020) at a later date?
The text was updated successfully, but these errors were encountered: