implement the On-Demand TLS feature #63

did · 2024-11-01T23:18:18Z

Big fan of Kamal & Kamal proxy here. Great job 👏

However, for my 2 Rails projects (website hosting for LocomotiveCMS & Maglev), I was missing the On-Demand TLS feature in Caddy.

It turns out it was easy to implement it in Kamal-Proxy since the autocert.HostPolicy type is just a function returning an error if the host is not allowed to get a certificate.

So, just by setting a new config option (--tls-on-demand-url), we can test dynamically if a host can get a TLS certificate, just by calling the --tls-on-demand-url endpoint.
It just has to return a 200 HTTP code (http://my-api-end-point/any-path?host=).

In order to test it on my server, I implemented a little Sinatra service to test but I didn't include in my PR because it was in Ruby and the single example in your repository was written in Go. Besides, this is a super niche feature, so it might not require an example.

require 'sinatra'

ALLOWED_HOSTS = %w(paul.nocoffee.fr sacha.nocoffee.fr)

get '/' do
  <<-HTML 
<html>
  <body>
    <h1>Hello #{request.host}!</h1>
  </body>
</html>
  HTML
end

get '/up' do
  200
end

get '/check' do
  ALLOWED_HOSTS.include?(params[:host]) ? [200, ['ok']] : [406, ['fail']]
end

My next step is to build a Kamal proxy docker image and use it in the Kamal deployment of one of my projects.

Let me know if you want me to re-work the PR to match your PR rules.

Thanks!

did · 2024-11-03T22:10:25Z

alright, I've made it work with my own fork of kamal.

The modifications were super light, actually it was just a matter of accepting a new config option for kamal-proxy (+ point to my own Docker image of kamal-proxy).

So, my config/deploy.yml looks like:

proxy:
  ssl: true
  hosts: [""]
  tls_on_demand_url: "http://staging-app-web-latest:3000/locomotive/api/allowed_host"
  app_port: 3000
  forward_headers: true

Notes:

hosts: [""] is important. Basically, it tricks the kamal-proxy router by allowing the * (wildcard) route (originally, impossible with ssl: true). I'd rather put hosts: "*" instead but it would require some extra work on the kamal-proxy app.
staging-app-web-latest is the host of my app, following the Kamal app name pattern <service_name>-web-<version>. I had to set a specific version to make it work across deployments (export VERSION=latest).

did · 2024-11-29T09:45:32Z

I also needed this feature to run the LocomotiveCMS hosting platform behind Kamal-Proxy 👉 did#1

internal/server/service.go

did · 2024-12-02T23:11:37Z

thanks @kwilczynski for the code review!

kevinmcconnell

Hey @did,

Thanks for getting this started! This does seem like a useful addition.

I left a couple of comments on the details. Let me know what you think about addressing those. (I'm also happy to make the changes myself if you don't have time).

README.md

kevinmcconnell · 2024-12-16T12:15:42Z

internal/server/service.go

+		return autocert.HostWhitelist(hosts...), nil
+	}
+
+	_, err := url.ParseRequestURI(options.TLSOnDemandUrl)


I'd expect that the on demand URL will usually point to an endpoint in the application that's deployed, rather than to some other external app. In which case, it would be simpler for this to be a path rather than an absolute URL. We'd then automatically call it on the currently deployed target (a bit like how the health check paths work).

That way you don't have to worry about having a stable hostname to reach for all versions of the app, etc., because the proxy takes care of that for you.

I'm not sure if there's a common enough need to support an external on demand URL as well, but for simplicity's sake it would be nice to have this be path-only if possible.

What do you think?

👌 I haven't thought about it!!!

In my production case, it would work perfectly because my Rails app was always in charge of returning that list of hostnames (even when I was hosting it with k8s).
Indeed, it'd have saved me a lot of time, trying to figure out the hostname of my endpoint.

Perhaps (it's a guess), we should keep the URL as well for developers who prefers to move the responsibility of this endpoint to another app (and probably deployed by Kamal too) for performance or architecture reasons.

Let's keep it simple in a first time so let's use the path only :-)

(I will make the modifications)

Sounds good!

I'd vote for full path as perhaps someone would want to host the check in a Kamal accessory instead?

We can always allow both: if you supply a path we'll call that on the app target, but if you supply a URL to some other endpoint we'll use that.

That way we get to keep the simpler configuration when the app is responsible for checking, which I suspect would be the more common case.

Sounds good to me :)

@kevinmcconnell I've implemented it. Let me know if it's okay for you. I also need to re-test this new implementation and see if it doesn't break my app.

internal/server/service.go

brendon · 2025-03-06T21:03:40Z

This looks great! Keep up the good work. We run a CMS where we can set up new sites while the app is running. Our current solution is Passenger via OpenResty (Nginx with Lua) and https://github.com/auto-ssl/lua-resty-auto-ssl. The last project on the list there appears abandoned though so I'm looking at other options :)

did · 2025-03-06T21:09:07Z

This looks great! Keep up the good work. We run a CMS where we can set up new sites while the app is running. Our current solution is Passenger via OpenResty (Nginx with Lua) and https://github.com/auto-ssl/lua-resty-auto-ssl. The last project on the list there appears abandoned though so I'm looking at other options :)

Thanks! Funny, Passenger + OpenResty is exactly the solution I'm using to run the LocomotiveCMS hosting SaaS platform!
The staging env uses Kamal with my patches and it runs great.

I need to find some free time to work on Kevin's feedbacks.

brendon · 2025-03-06T21:33:33Z

This looks great! Keep up the good work. We run a CMS where we can set up new sites while the app is running. Our current solution is Passenger via OpenResty (Nginx with Lua) and https://github.com/auto-ssl/lua-resty-auto-ssl. The last project on the list there appears abandoned though so I'm looking at other options :)

Thanks! Funny, Passenger + OpenResty is exactly the solution I'm using to run the LocomotiveCMS hosting SaaS platform! The staging env uses Kamal with my patches and it runs great.

I need to find some free time to work on Kevin's feedbacks.

Yep, I've never had any problems with this setup. I wrote a custom Lua script to look at our database and find the authorised hostnames. Didn't even think about just hitting an endpoint on the app itself. So much simpler! :)

jmadkins · 2025-07-02T04:19:15Z

Thanks for all your work on this, @did! I also need this feature and am happy to help finish this up. Could I help?

did · 2025-07-09T16:04:24Z

thanks @jmadkins! Alright I'm back to this PR and I've just pushing come code. I won't say no to some help ☺️

My next task is to use either a path or an URL (TlsOnDemandURL).
In the case of the path, it's still blurry for me if I had to get a hostname or just naively the localhost.
And perhaps, we will have to rename the config name as well since TlsOnDemandUrl is a bit weird if by default, we expect a path. Any thoughts?

…e tests for the service.createAutoCertHostPolicy function

…is present