feat: Let's Encrypt ACME certificate integration (HTTP-01 + DNS-01 challenges)

[phaus]

Summary

Flynn currently generates self-signed TLS certificates during bootstrap (pkg/certgen) and stores them in PostgreSQL. There is no integration with any ACME certificate authority. This feature adds automatic Let's Encrypt certificate provisioning and renewal, supporting both HTTP-01 (webserver-based) and DNS-01 (DNS record-based) challenges.

Uses go-acme/lego (v4) as the ACME client library, including lego's built-in autodns provider for DNS-01 challenges via the AutoDNS JSON API (api.autodns.com/v1/). No separate AutoDNS library is needed.

---

Current Certificate Architecture

pkg/certgen → self-signed X.509 (RSA 2048, 5yr)
    ↓
pkg/tlscert → CA + leaf cert pair
    ↓
bootstrap → controller PostgreSQL (certificates + route_certificates tables)
    ↓
controller API → event stream → router (SNI GetCertificate callback)

Key files:

• pkg/certgen/certgen.go — self-signed cert generation
• pkg/tlscert/tlscert.go — wrapper (CA + leaf)
• controller/data/schema.go:707-721 — certificates + route_certificates tables
• controller/data/route.go:104-133 — cert CRUD with dedup (cert_sha256 unique index)
• router/http.go:352-417 — TLS listener with GetCertificate SNI callback
• router/types/types.go:8-22 — Certificate struct (ID, Cert, Key, Routes)

---

Proposed Design

1. New Package: pkg/autocert/

A new pkg/autocert/ package that encapsulates ACME certificate management:

pkg/autocert/
├── autocert.go       — Certificate manager (request, renew, store)
├── challenge.go      — Challenge HTTP handler (for HTTP-01)
├── dns.go            — DNS-01 challenge provider wiring (lego built-in providers)
├── storage.go        — Certificate storage interface (PostgreSQL via controller)
├── config.go         — Configuration types (CA URL, email, challenge type, DNS provider)
└── renewal.go        — Background renewal goroutine

2. Configuration

Add an autocert section to the controller config (or cluster environment variables):

type AutoCertConfig struct {
    Enabled       bool              `json:"enabled"`
    CAURL         string            `json:"ca_url"`          // default: Let's Encrypt production
    Email         string            `json:"email"`           // ACME account email
    ChallengeType string            `json:"challenge_type"`  // "http-01" or "dns-01"
    DNSProvider   string            `json:"dns_provider"`    // e.g. "autodns"
    DNSConfig     map[string]string `json:"dns_config"`      // provider-specific config
}

3. ACME Client Setup (lego)

import (
    "github.com/go-acme/lego/v4/lego"
    "github.com/go-acme/lego/v4/certificate"
)

func NewClient(config *AutoCertConfig, user *ACMEUser) (*lego.Client, error) {
    cfg := lego.NewConfig(user)
    cfg.CADirURL = config.CAURL
    client, err := lego.NewClient(cfg)
    if err != nil { return nil, err }

    switch config.ChallengeType {
    case "http-01":
        // lego uses built-in HTTP-01 solver
    case "dns-01":
        provider, err := createDNSProvider(config)
        if err != nil { return nil, err }
        client.Challenge.SetDNS01Provider(provider)
    }
    return client, nil
}

4. AutoDNS DNS-01 Provider (lego built-in)

lego ships a first-class AutoDNS provider (providers/dns/autodns, added in v3.2.0) that talks to the AutoDNS JSON API (https://api.autodns.com/v1/, overridable via AUTODNS_ENDPOINT). Use it directly — no custom adapter, no extra dependency:

import "github.com/go-acme/lego/v4/providers/dns/autodns"

provider, err := autodns.NewDNSProviderConfig(&autodns.Config{
    APIUser:            cfg.DNSConfig["api_user"],     // AUTODNS_API_USER
    APIPassword:        cfg.DNSConfig["api_password"], // AUTODNS_API_PASSWORD
    Context:            4,                             // 4 = production, 1 = test
    PropagationTimeout: 120 * time.Second,
    TTL:                600,
})
client.Challenge.SetDNS01Provider(provider)

Credentials are supplied via NewDNSProviderConfig from controller config (not env vars).

5. HTTP-01 Challenge Handler

For HTTP-01, the controller serves /.well-known/acme-challenge/<token> on port 80 (plain HTTP, before TLS termination):

func (m *AutoCertManager) HandleChallenge(w http.ResponseWriter, r *http.Request) {
    token := strings.TrimPrefix(r.URL.Path, "/.well-known/acme-challenge/")
    response, err := m.httpSolver.GetChallengeResponse(token)
    if err != nil { http.NotFound(w, r); return }
    w.Write([]byte(response))
}

6. Certificate Lifecycle

1. Initial provisioning: On route creation (or explicit command), request a certificate via the configured challenge type.
2. Storage: Store the issued certificate in the existing certificates table, linked to the route via route_certificates.
3. Router sync: Controller fires event → router picks up new certificate via existing event stream → GetCertificate serves the real cert.
4. Background renewal: Goroutine checks certificate expiry daily and renews at 30 days before expiry (Let's Encrypt certs are 90 days).
5. Renewal flow: lego Renew() → challenge re-issued → new cert stored → event fired → router hot-swaps.

7. API Changes

New controller endpoints:

POST   /certs/letsencrypt          — provision a Let's Encrypt cert
GET    /certs/letsencrypt/:id      — check provisioning status
DELETE /certs/letsencrypt/:id      — revoke a cert
GET    /certs/letsencrypt/config   — get/set ACME configuration

CLI commands:

flynn cert letsencrypt --domain example.com --challenge dns --dns-provider autodns
flynn cert letsencrypt --domain example.com --challenge http
flynn cert letsencrypt --status <cert-id>
flynn cert letsencrypt --revoke <cert-id>

8. Database Schema Change

ALTER TABLE certificates ADD COLUMN source text NOT NULL DEFAULT 'self-signed';
ALTER TABLE certificates ADD COLUMN expires_at timestamptz;
ALTER TABLE certificates ADD COLUMN acme_account_id text;
ALTER TABLE certificates ADD COLUMN domains text[];

---

Implementation Plan

Phase 1: Core ACME Library

• Create pkg/autocert/ with lego client setup
• Wire up lego's built-in autodns provider (autodns.NewDNSProviderConfig) for DNS-01
• Implement HTTP-01 challenge handler
• Certificate storage integration with existing certificates table
• Background renewal goroutine

Phase 2: Controller Integration

• New /certs/letsencrypt/* API endpoints
• Schema migration (add source, expires_at, acme_account_id, domains columns)
• ACME account management (store/retrieve account keys)
• Challenge endpoint on port 80 (HTTP-only)

Phase 3: Router Integration

• Verify /.well-known/acme-challenge/* is served on HTTP (port 80)
• Test hot-swap on certificate renewal

Phase 4: CLI

• flynn cert letsencrypt command
• Status and revoke subcommands

Phase 5: Bootstrap Integration

• Optional: auto-provision cluster domain cert during bootstrap
• ACME config in bootstrap manifest

Phase 6: Testing

• Unit tests for pkg/autocert/ (mock lego client)
• Integration test with Let's Encrypt staging
• Test HTTP-01 and DNS-01 challenge flows
• Test certificate renewal lifecycle
• Test hot-swap (renew cert while traffic is flowing)

---

Design Decisions

1. lego vs manual ACME: Use lego — handles account management, challenge orchestration, certificate parsing, and renewal logic.

2. lego's built-in autodns provider vs a custom adapter: lego ships a native providers/dns/autodns provider (since v3.2.0, PR Godeps: Update BurntSushi/toml. flynn/flynn#957, actively maintained upstream) using the AutoDNS JSON API (api.autodns.com/v1/). Use it via autodns.NewDNSProviderConfig and drop consolving/autodns.go (a separate XML-API binding). Fall back to an XML-API adapter only if the AutoDNS account lacks JSON API access.

3. Certificate storage: Reuse existing certificates table with new columns (source, expires_at, etc.) — keeps the existing route→certificate mapping and router sync unchanged.

4. Challenge handler location: Mount on the controller's HTTP port (port 80). Router needs to ensure /.well-known/acme-challenge/* reaches the controller.

5. Renewal timing: Renew at 30 days before expiry (Let's Encrypt certs are 90 days). Check daily via background goroutine.

---

Open Questions

• Should the ACME account private key be stored in PostgreSQL (encrypted) or a separate secret store?
• How to handle wildcard certificates (require DNS-01)?
• Support multiple DNS providers beyond AutoDNS, or AutoDNS-only for now?
• Port 80: does the Flynn router already handle HTTP on port 80, or do we need explicit HTTP listener support?
• Confirm the AutoDNS account has JSON API access (api.autodns.com/v1/); if not, fall back to consolving/autodns.go via the XML endpoint (gateway.autodns.com)
• Rate limits: how to handle Let's Encrypt rate limits (50 certs/week) in a multi-node cluster?