New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support to reverse proxy into Docker Containers #199

Closed
redbeard0x0a opened this Issue Jul 24, 2015 · 57 comments

Comments

@redbeard0x0a

redbeard0x0a commented Jul 24, 2015

A useful feature that would be good to use Caddy as the frontend server in front of 1+ Docker Hosts (or Docker Swarm).

The caddy configuration file would have one or more docker hosts specified so that specifically tagged/named docker container instances running on those hosts could be looked up based on the Host header requested via HTTP. When a request comes in, take the Host header and do a lookup (full, partial or regex) against the Docker API to match the Host header to a Docker Instance running on the docker hosts (load balancing if multiple instances are running).

This would make it easy to launch docker containers that are configured in a specific way (i.e. a container tagged as dockerhubacct/www.example.com -or- via environment variables). There are many different ways this could be setup, container names need to be unique per docker host, but tags and environment variables do not.

Docker Proxy - Possible Caddyfile Directives

docker_proxy / dockerhost1 dockerhost2 {
  # Use the same things as a normal proxy. docker_proxy would just be a smarter proxy module
}
@redbeard0x0a

This comment has been minimized.

redbeard0x0a commented Aug 6, 2015

Here is a very sketchy bit of code that could provide as an example of what is being accomplished. This code was prototyped on a mac, running boot2docker with specifically tagged docker containers. I had to setup extra hostnames in /etc/hosts to match the container tags.

package main

import (
    "log"
    "net/http"
    "net/http/httputil"
    "fmt"
    "strings"
    "os"
    "math/rand"
    "strconv"
    "github.com/fsouza/go-dockerclient"
)

func main() {
    endpoint := os.Getenv("DOCKER_HOST")
    path := os.Getenv("DOCKER_CERT_PATH")
    ca := fmt.Sprintf("%s/ca.pem", path)
    cert := fmt.Sprintf("%s/cert.pem", path)
    key := fmt.Sprintf("%s/key.pem", path)
    client, _ := docker.NewTLSClient(endpoint, cert, key, ca)
    // use client

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {

        requestedHost := strings.Replace(strings.Split(r.Host, ":")[0], ".localdomain", "", 1)

        var appName string
        if strings.Contains(requestedHost, ".") {
            appName = strings.Split(requestedHost, ".")[0]
        } else {
            appName = requestedHost
        }

        // Lookup Requested Host from Docker API (host name)
        imgs, _ := client.ListContainers(docker.ListContainersOptions{All: false})
        containers := []string{}
        fmt.Println("Running Containers\n------------------")
        for _, img := range imgs {
            fmt.Printf("%s [%s]\n", img.Image, img.ID)

            var imageName string
            if strings.Contains(img.Image, "/") {
                imageName = strings.Split(img.Image, "/")[1]
            } else {
                imageName = img.Image
            }
            if strings.ToLower(imageName) == strings.ToLower(appName) {
                containers = append(containers, img.ID)
            }
        }
        fmt.Printf("\nContainers that can fulfill request: \n%v\n", containers)

        if len(containers) <= 0 {
            w.Write([]byte("BACKEND DOWN"))
            return
        }
        containerID := containers[rand.Intn(len(containers))]
        fmt.Printf("\nPicking Container: %s\n", containerID)

        container, _ := client.InspectContainer(containerID)
        fmt.Printf("Ports: %v\n", container.NetworkSettings.Ports)
        port, _ := strconv.Atoi(container.NetworkSettings.Ports["3000/tcp"][0].HostPort)
        fmt.Printf("\n\n%#v\n", port)

        director := func(req *http.Request) {
            req = r
            req.URL.Scheme = "http"
            req.URL.Host = fmt.Sprintf("192.168.59.103:%d", port) //r.Host
        }
        proxy := &httputil.ReverseProxy{Director: director}
        proxy.ServeHTTP(w, r)
        fmt.Printf("Requested Host: %s\n", requestedHost)
        fmt.Printf("App Name: %s\n", appName)
    })
    log.Fatal(http.ListenAndServe(":8000", nil))
}
@redbeard0x0a

This comment has been minimized.

redbeard0x0a commented Aug 6, 2015

This is from memory, but to follow through this example, there would be some setup to do on your mac. You need boot2docker installed and running.

Edit /etc/hosts

127.0.0.1 www.example.com

Make a container that we can 'reverse proxy' into, listening on port 3000 with the url as a part of the tag.

FROM busybox

WORKDIR /

COPY ./testsvr_linux_amd64 /testsvr

EXPOSE 3000
CMD /testsvr

This would be the command to create the container with the right tag.

docker build -t username/www.example.com .

You should be able to go to the web browser and enter http://www.example.com:8000, which would then hit the example server (which should be replaced by caddy), it would lookup the hostname against the Docker Host API (from the servers specified in the Caddyfile), then reverse proxy to the IP:PORT combination that the container has been mapped to, giving you the same result if you had connected directly to the dockerhost:3000 web server.

@mholt

This comment has been minimized.

Owner

mholt commented Aug 7, 2015

Just wanted to chime in real quick to let you know that you're not speaking into a vacuum. 😄 I am paying attention to this feature suggestion, just traveling the next few weeks. (Anyone with Docker experience is invited to discuss here in the meantime!)

@abiosoft

This comment has been minimized.

Collaborator

abiosoft commented Aug 10, 2015

This will make a nice add-on.

If I get this right, the middleware will lookup ip and port of running containers (meeting a filter criteria) and reverse proxy into it (or load balance if multiple).

I'll try looking into this.

@Zenithar

This comment has been minimized.

Zenithar commented Sep 3, 2015

This feature can be done with docker-template with haproxy, registrator to handle docker events, consul as a container / service registry.

In fact you could replace haproxy by caddyserver, docker-template will fill a default Caddyfile then ask to caddyserver to restart.

Ref:
Using HAProxy and Consul for dynamic service discovery on Docker

@brodock

This comment has been minimized.

brodock commented Sep 10, 2015

awesome... was looking for something around that 👍

@captncraig

This comment has been minimized.

Collaborator

captncraig commented Oct 31, 2015

There is also https://github.com/jwilder/docker-gen which already does a lot of the work of watching docker socket, configuration and such. Unfortunately their code is not (yet) reusable as a library. We can look at their code to see most of the possible config options needed to contact docker in different environments.

I like the scheme that https://github.com/jwilder/nginx-proxy uses for detecting config. Namely VIRTUAL_HOST, VIRTUAL_PORT, and VIRTUAL_PROTO environment variables. I pretty much want that magic for caddy.

My thought was generate a file like docker.conf from running docker containers, and import that into your main caddyfile. When changes are detected, the plugin can regenerate all of docker.conf and restart caddy.

Magic Docker + Magic https would be a killer product.

@mholt

This comment has been minimized.

Owner

mholt commented Oct 31, 2015

Now that Caddy has restart capabilities (in the letsencrypt branch) this is possible. 😃

@zacheryph

This comment has been minimized.

zacheryph commented Nov 10, 2015

The idea of querying docker [on multiple hosts] for every request sounds like an extreme amount of overhead, and an absurd amount of added latency. I don't think caddy as an http server needs to care or remotely know where or what docker is. Next you will have people asking that it track when they create new droplets on digital ocean or instances on AWS and automatically proxy to them.

I would agree with @Zenithar and @captncraig. This should be handled by something like docker-gen or bigger such as registrator / consul / docker-template. containers specifically stating via env vars where/what they need to be doing and where they need to be getting injected. This is far less magic, less complexity, and less overhead.

@captncraig

This comment has been minimized.

Collaborator

captncraig commented Nov 10, 2015

I feel like the ideal for this would be some api to the proxy module that allows you to change the downstreams at runtime. Restarting the entire server seems tricky, and there are a lot of possible use cases for reconfiguring the downstream servers.

It would be nice if either a custom middleware, or an outside application could change the set of proxied servers without a full restart.

@mholt

This comment has been minimized.

Owner

mholt commented Nov 10, 2015

@captncraig

It would be nice if either a custom middleware, or an outside application could change the set of proxied servers without a full restart.

That's really, really hard to do right. It's inherently racy.

Restarting the entire server seems tricky

Restarts are pretty lightweight (I haven't tried doing dozens of restarts per second or anything like that, but they work well enough in succession). You may have a few processes hang around for a couple seconds while it closes connections, but each restart is graceful and race-free. I'm going for robust restarts here; you can trust them to work.

@zacheryph

The idea of querying docker [on multiple hosts] for every request sounds like an extreme amount of overhead, and an absurd amount of added latency. I don't think caddy as an http server needs to care or remotely know where or what docker is.

I agree with you -- the added latency would be unacceptable. What you might do instead (this coming from a guy who has no idea how Docker works, having barely ever used it) is have a background routine that polls (?) on occasion and keeps the backends cached for handlers to use on demand. This is still clunky, though, it's true.

@pierrebeaucamp

This comment has been minimized.

pierrebeaucamp commented Nov 10, 2015

I'm currently using confd to manage a very similar scenario with docker - but I haven't looked into graceful restarts yet. I think the should be possible with the new caddy API.

@joshmanders

This comment has been minimized.

joshmanders commented Dec 3, 2015

I use jwilder/nginx-proxy but if this was a directive or feature of Caddy, oh man, that'd be killer.

@PunKeel

This comment has been minimized.

PunKeel commented Dec 6, 2015

👍 :)

@BlackGlory

This comment has been minimized.

BlackGlory commented Dec 25, 2015

@joshmanders
I make caddy-proxy do this use docker-gen, but I cannot let its response newly containers, any advice?

@captncraig

This comment has been minimized.

Collaborator

captncraig commented Dec 25, 2015

Looks like you need to use the notify feature in docker gen to restart
caddy.
On Dec 25, 2015 6:57 AM, "BlackGlory" notifications@github.com wrote:

@joshmanders https://github.com/joshmanders
I make caddy-proxy https://github.com/BlackGlory/caddy-proxy do this
use docker-gen, but I cannot let its response newly containers, any advice?


Reply to this email directly or view it on GitHub
#199 (comment).

@pwFoo

This comment has been minimized.

pwFoo commented Dec 30, 2015

Interesting topic. Would love to use Caddy as http / https proxy fpr docker containers.

@pwFoo

This comment has been minimized.

pwFoo commented Dec 30, 2015

docker-gen should be a different docker container than caddy-proxy because docker-gen needs docker.sock.
I would try to build it with alpine linux.

@pwFoo

This comment has been minimized.

pwFoo commented Dec 31, 2015

My entrypoint / cmd (get from docker inspect)

        "Entrypoint": [
            "/usr/local/bin/docker-gen"
        ],
        "Cmd": [
            "-watch",
            "/home/caddy/conf/caddy.tmpl",
            "/home/caddy/conf/Caddyfile"
        ],

@BlackGlory
Live changes to config need a notify method to advice caddy to reload.
I removed the needed param because docker-gen not supports SIGUSR1. For testing I fired SIGUSR1 inside the container manually. Docker-gen "-notify-sighup" could kill the caddy process. Maybe it could be done (untested yet!) with:

-notify

I build a simple test with two containers, but some problems to solve / implement additional features...

  1. docker-gen can't send -SIGUSR1 to reload the config
  2. I try to add custom config with caddy import feature. For example to for set "tls off" or other stuff, but the file to import have to be created automatically to prevent "no such file or directory"

Instead of two containers maybe it could be secured with "sudo -u " or a chroot environment for caddy.

@zacheryph

This comment has been minimized.

zacheryph commented Dec 31, 2015

@BlackGlory awesome work on this. I've been meaning to get something like this going but alas, no time 😢. Right now every time i upgrade a container I have to rebuild/restart caddy as well.

docker-gen running alone or within (same way as nginx-proxy) I am ok. For the purpose of getting it working its probably easier to have them running together anyway.

Caddy is tricky in this scenario. For one, we can NOT just docker kill -s USR1 the caddy container or the caddy process. I repeat, we CANNOT do this.

USR1

Reloads the configuration file, then gracefully restarts the server. This spins up a process with a different process ID.

When you run caddy it runs in the foreground (yeah guy...) When sending caddy a USR1 it restarts, with a different process ID. This in turn means that caddy is no longer running in the foreground.

This will require some kind of wrapper to watch the pid file, gracefully hand off USR1 to the caddy process, and ... sleep? for so long waiting for the pid file to be updated with the new pid and in turn keeping an eye on the new process.

Are there any plans for caddy to be able to gracefully restart without starting a new process? I personally hope so as it would make the endeavor a whole lot easier.

@zacheryph

This comment has been minimized.

zacheryph commented Dec 31, 2015

I just got done looking at the issue list and found #416. Maybe the pidproxy tool @tpng mentioned can be used (or something similar that doesn't require supervisord?) can be used for this purpose.

@pwFoo

This comment has been minimized.

pwFoo commented Jan 16, 2016

I played a little bit with my caddy-proxy docker image (based on @BlackGlory build) with additional CADDY_OPTS.

Usage example

sudo docker run -d --name Flarum -e VIRTUAL_HOST=dev.flarum.lan -e CADDY_OPTS="tls off;log access.log;errors error.log" echo511/docker-flarum

Generated Caddyfile:

dev.flarum.lan {
      proxy / 172.17.0.9:80
      # Additional Caddy options by CADDY_OPTS
      tls off                                  
      log access.log                                  
      errors error.log                                  
}

Options like log or errors should be unique! Options added without modification. So all files would be created in the same directory.

It's combined with docker-gen because of the reload problem. Primary container process is docker-gen, so it's easy to reload caddy with a sigusr1 signal.

@joshmanders

This comment has been minimized.

joshmanders commented Feb 17, 2016

@pwFoo can I see the source of this? I'm looking at swapping out jwilder/nginx-proxy for Caddy.

@pwFoo

This comment has been minimized.

pwFoo commented Feb 19, 2016

Hi @joshmanders
at the moment I haven't a git / docker repo for that.

dockerfile

FROM alpine:3.3

ADD     files/* /tmp/

RUN /sbin/apk -U add sudo libcap ca-certificates && rm -rf /var/cache/apk/* \ 
    && mkdir -p /var/www/logs /var/www/.caddy \
    && /bin/chown -R nobody:nobody /var/www \
    && awk -F ":" 'BEGIN{OFS = ":"} /nobody/{$6="/var/www"}{ print}' /etc/passwd > /etc/passwd.nwe && mv /etc/passwd.nwe /etc/passwd \
    && /bin/chmod +x /tmp/caddy /tmp/docker-gen /tmp/caddy.sh \
    && mv /tmp/caddy /tmp/docker-gen /usr/local/bin/ \
    && mv /tmp/caddy.sh /caddy.sh \
    && mv /tmp/Caddyfile.tmpl /etc/Caddyfile.tmpl \
    && /usr/sbin/setcap cap_net_bind_service=+ep /usr/local/bin/caddy

EXPOSE 80
EXPOSE 443
WORKDIR /var/www

ENTRYPOINT ["/caddy.sh"]

Usage

docker run -d --name caddy-proxy -v /var/run/docker.sock:/var/run/docker.sock:ro -p 80:80 -p 443:443 caddy-proxy [additional caddy options]

caddy.sh start script

#!/bin/sh

set -x

WORKDIR=/var/www
PID=$WORKDIR/caddy.pid
CONF=/etc/Caddyfile
TMPL=/etc/Caddyfile.tmpl
CUSTOM=$WORKDIR/Caddyfile_custom
LOG=$WORKDIR/logs/caddy.log
PORT=80

# Update because of changed Caddyfile by Docker-Gen
if [ "$1" == "update" ]; then
    /bin/kill -SIGUSR1 `cat $PID`
else
    rm -f $LOG

    if [ ! -f $CONF ]; then
        touch $CONF
    fi

    if [ ! -f $CUSTOM ]; then 
        touch $CUSTOM
    fi

    # Caddy ReverseProxy
    /usr/bin/sudo -u nobody touch $LOG
    /usr/bin/sudo -u nobody /usr/local/bin/caddy -conf=$CONF -pidfile=$PID -log=$LOG -port=$PORT -agree=true $@ &

    # Docker-Gen
    /usr/local/bin/docker-gen -watch -notify="/caddy.sh update" $TMPL $CONF
fi

Caddyfile.tmpl

{{ range $host, $containers := groupByMulti $ "Env.VIRTUAL_HOST" "," }}
{{ $host }} {
  {{ range $index, $value := $containers }}
    {{ $addrLen := len $value.Addresses }}
    {{ if eq $addrLen 1 }}
      {{ with $address := index $value.Addresses 0 }}
      proxy / {{ $address.IP }}:{{ $address.Port }}
      {{ end }}
    {{ else if $value.Env.VIRTUAL_PORT }}
      {{ range $i, $address := $value.Addresses }}
        {{ if eq $address.Port $value.Env.VIRTUAL_PORT }}
        proxy / {{ $address.IP }}:{{ $address.Port }}
        {{ end }}
      {{ end }}
    {{ else }}
      {{ range $i, $address := $value.Addresses }}
        {{ if eq $address.Port "80" }}
        proxy / {{ $address.IP }}:{{ $address.Port }}
        {{ end }}
      {{ end }}
    {{ end }}
  {{ end }}

      # Additional Caddy options by CADDY_OPTS
      log logs/{{ $host }}-access.log {
        rotate {
          size 10 # Rotate after 100 MB
          age  30 # Keep log files for 14 days
          keep 5  # Keep at most 10 log files
        }
      }
      errors {
        log logs/{{ $host }}-error.log {
          size 10 # Rotate after 50 MB
          age  30 # Keep rotated files for 30 days
          keep 5  # Keep at most 5 log files
        }
      }

  {{ range $index, $value := $ }}
  {{ $opts := $value.Env.CADDY_OPTS }}
    {{ if $opts }}{{ range $index, $option := split $opts ";" }}        
      {{ $option }}                                  
    {{ end }}{{ end }}
  {{ end }}    
}
{{ end }}

import Caddyfile_custom

The dockerfile requires the caddy binary, docker-gen binary, Caddyfile.tmpl and caddy.sh in a folder "files"

It works for me (non production environment), but should be improved...
Tested it with some containers and also with load balancing.

Maybe I'll create a repo, but at the moment there is no time...

@cblomart

This comment has been minimized.

cblomart commented Jun 2, 2016

Hello, I always like the idea of automating things a bit.

I try to build trimed down container by building them from scratch.
And don't think it would play well with having docker-gen and caddy in the same container.

I must admit i liked the idea of having caddy directly lisening to docker events to get restarted/created/stopped container and be able to adapt itself. But if i get the feeling, it would be a bit too much specialisation for Caddy (why not aws and digitalocean then... -well because docker is docker not A paas-).

Additionally i don't like the idea of templates...
this should stay simple: label your container, they get proxied

I would think of very default options then standard 3 liners (forwarded proto/host and host header). Evetnualy a bit of log separation but... what for in a docker container.

So my two cents: a docker template/gen would need much customisation for the app (template mgmt, link to container, ...): shouldn't it basic reverse proxying be simplier ?

@ometra

This comment has been minimized.

ometra commented Jun 2, 2016

Hi All,

For this specific use case, you can look at traefik project (https://traefik.io/).
I haven't tested it for now but it seems promising.

@mholt

This comment has been minimized.

Owner

mholt commented Jun 2, 2016

Traefik is a good option if dynamically reverse proxying into Docker is your primary goal. That's a very specific use case and it's good at that.

I'm pretty confident Caddy will be able to do this in the future without too much trouble; Caddy 0.9 will be much more extensible than Caddy currently is, which may make this possible with a plugin.

If someone would like to make a concrete proposal that is very specific so that it could be implemented from those instructions, maybe somebody could write a plugin to do it. And I could help make sure it's possible to plug into Caddy like that. Maybe a new directive is all that's needed, or extending the proxy directive somewhat, like #564.

@dimitrovs

This comment has been minimized.

dimitrovs commented Jul 1, 2016

It looks like @BlackGlory got it working! Caddy restarts automatically when a new docker container with -e VIRTUAL_HOST is created. Good job @BlackGlory.

@ibmendoza

This comment has been minimized.

ibmendoza commented Jan 20, 2017

No need to support this as far as Docker Swarm Mode is concerned.

To illustrate, please read https://github.com/ibmendoza/go-examples/tree/master/docker

The gist of the matter is this: You direct or proxy incoming request from Caddy to backend(s) located at Docker Swarm manager node(s). Docker Swarm takes care of relaying those requests to the underlying worker nodes.

From my example above, all incoming requests from Caddy will be proxied to port 8080 at any of three (typically) Docker Swarm manager nodes. Docker Swarm then takes care of load balancing and high availability amongst the worker nodes. Great separation of concerns!

@lucaslorentz

This comment has been minimized.

Contributor

lucaslorentz commented Jun 13, 2017

I think this would be a good case for a Caddyfile loader plugin: https://github.com/mholt/caddy/wiki/Writing-a-Plugin:-Caddyfile-Loader

I stearted this plugin some days ago: https://github.com/lucaslorentz/caddy-docker-proxy
It's a way I have to learn golang 😄

@mholt I'm using signals to reload caddy configuration, it's working fine. I'm looking forward for more instructions on "Dynamic Reloads" wiki section.

Traefik is awesome for proxying and service discovery, but caddy has more http features. I also had issues with Traefik gzip compression when container returns already compressed data. Caddy is working fine for me so far.

@mholt

This comment has been minimized.

Owner

mholt commented Jun 15, 2017

Glad to hear! (And yeah, I'll get around to that sooner or later. Gonna see how the API plays into it.)

@pwFoo

This comment has been minimized.

pwFoo commented Jul 10, 2017

@lucafavatella
Nice, I'll take a look into it soon :)

@krishamoud

This comment has been minimized.

Collaborator

krishamoud commented Oct 9, 2017

I forked a dynamic caddy reloader backed by Consul here: https://github.com/krishamoud/caddy-consul

The use case was actually exactly dynamic proxying to Docker containers. I have been using this setup in production a docker hosting platform I run.

Heres a demo: https://www.youtube.com/watch?v=uKJJgRxoo8o

@mholt

This comment has been minimized.

Owner

mholt commented Oct 11, 2017

@krishamoud That's cool! Thanks for sharing. I noticed Caddy's 404 near the end there. :)

@mholt

This comment has been minimized.

Owner

mholt commented Feb 17, 2018

Does Docker Swarm use DNS? The proxy directive now supports SRV backends.

@whitestrake

This comment has been minimized.

Contributor

whitestrake commented Feb 19, 2018

It does use DNS, but I understand it simply makes A record resolution possible using the service name as a host name, just like Compose (e.g. caddy. 600 IN A 172.18.0.7).

I think Kubernetes uses SRV records, though.

@lbguilherme

This comment has been minimized.

Contributor

lbguilherme commented Feb 19, 2018

I use Docker Swarm with Caddy in production. Swarm provides a DNS for A records of service names. So I can just mention the service name on the Caddyfile and it all works.

@ibnesayeed

This comment has been minimized.

Contributor

ibnesayeed commented Feb 22, 2018

Recently I was setting up a server for our research projects. Our requirement was to be able to host new service stacks in a Docker Swarm cluster and make them accessible on specified sub-domains automatically. Being a contributor of Caddy, I was very interested in using it as the reverse proxy. We had a similar setup using popular jwilder/nginx-proxy image in the past, but we wanted a simpler and more modern approach to it. However, making it work in Caddy (as of now) would have required us to ave some custom code and plumbing in place.

We ended up using Traefik, which has solved the dynamic service discovery and loadbalancing in Docker Swarm pretty nicely. When defining a Docker service stack, we add a few labels in the Stackfile (such as the domain name that is intended to proxy requests to the service) and create the service. Traefik runs in a container which has Docker's socket file mounted, so it can react to any changes, scan through all the containers, filter against certain labels, and automatically generate a reverse proxy configuration file (behind the scene) to reroute traffic.

Once we have similar capabilities baked into Caddy, we might reconsider it as it has some features that Traefik is lacking.

@mholt

This comment has been minimized.

Owner

mholt commented Feb 22, 2018

@ibnesayeed Thanks for writing your story -- it's really handy to know how people are using the software!

Once we have similar capabilities baked into Caddy

Which capabilities, exactly? Your post talked about a lot of stuff and I want to be sure I understand exactly what you'd need Caddy to do. (I assume using Swarm's DNS A records is not an option or something?)

@ibnesayeed

This comment has been minimized.

Contributor

ibnesayeed commented Feb 22, 2018

Which capabilities, exactly? Your post talked about a lot of stuff and I want to be sure I understand exactly what you'd need Caddy to do. (I assume using Swarm's DNS A records is not an option or something?)

I have described it in the second paragraph of my previous comment, but I can explain it more. Exploiting Swarm's DNS records will work as it adds an entry with the name of each defined service. However, this would require us to update a Caddyfile manually, each time a new service is added or removed (or taken down temporarily) in the Swarm. This manual setup would work great if there are just a few services to be deployed and they do not change very often. However, in an academic research lab like ours, we come up with new ideas every other day, build a tool around it, deploy it for internal testing, advertise to masses when we feel comfortable sharing it with the world, and take them down when we can't afford to maintain them. In a situation like this, we really want something that automates things as much as possible to minimize manual configuration and lower the learning barrier for new researchers joining the team and taking the torch from those who are graduating.

Here is roughly how I would envision Caddy to solve this issue. Say, we have a Caddy plugin for Docker that provides this functionality. The plugin would by default know the well-known location of the Docker socket, but can be configured to read from a TCP URL if necessary. Having access to the socket, plugin will be notified about every change that happens under the connected Docker engine. Using well-documented Docker API, the plugin can query all the running containers along with all their metadata at anytime. Whenever a new service is added or existing services die, the plugin reacts to the notification and scans all the services currently present to identify those matching certain criteria to be selected for being proxy routed to. The criteria would be some well-known labels (defined by the plugin) that are added to those services. Using values in those labels, a config file is generated and loaded. This process will happen each time there is a change in the Docker Swarm that adds or removes services.

Consider the following sample Stackfile.

version: "3"

services:
  assets:
    image: httpd
    volumes:
      - /web/root/directory:/usr/local/apache2/htdocs
    networks:
      - public
    deploy:
      replicas: 3
      labels:
        - "caddy.vhost=static.example.com"
        - "caddy.port=80"

networks:
  public:
    external: true

When this stack is deployed (say, with the name static), it adds a new service named static_assets in the Swarm. The service has three replicas, but each of them will have the two labels as defined above. The Docker plugin of Caddy gets notification of the change, it scans services again and finds one that has the caddy.vhost label defined, so it adds a new backend with the name of the service, static_assets (this is where utilizing the Swarm's DNS automatically will be helpful) which can be accessed when a request comes with Host: static.example.com header. The caddy.port label tells which port number of of the container is the listening port. In this case, basically we are going to route traffic to static_assets:80. Docker Swarm will take care of loadbalancing among replicas of the same service. We can add some other labels to allow more configuration options, such as disabling TLS for certain routes or enabling automatic redirects to corresponding HTTPS URL.

@whitestrake

This comment has been minimized.

Contributor

whitestrake commented Feb 22, 2018

Caddy supports graceful reloading and custom Caddyfile loaders, so I'd wager that the Traefik approach of adapting to Docker socket changes with hot reloads is a very approachable feature with plugins; the annoying part, I imagine, would be writing a Docker-specific Caddyfile loader that interprets results from the Docker API.

Also, for your consideration, a while back I tinkered with jwilder/docker-gen (the Docker-monitoring and template-writing half of jwilder/nginx-proxy) and graceful reloading based on environmental variables, and packaged it with Caddy in a single container. Conceptually it's identical to nginx-proxy, but with Caddy - as a proof-of-concept, it's pretty solid: https://github.com/Whitestrake/caddy-gen

@ibnesayeed

This comment has been minimized.

Contributor

ibnesayeed commented Feb 22, 2018

Also, for your consideration, a while back I tinkered with jwilder/docker-gen (the Docker-monitoring and template-writing half of jwilder/nginx-proxy) and graceful reloading based on environmental variables, and packaged it with Caddy in a single container. Conceptually it's identical to nginx-proxy, but with Caddy - as a proof-of-concept, it's pretty solid: https://github.com/Whitestrake/caddy-gen

Even I experimented a while ago with the same combination of jwilder/docker-gen + Caddy to replicate what happens in jwilder/nginx-proxy and I was able to get things working, but I did not use it in production, because such a setup would be brittle from maintenance perspective in the academic research environment I work. Currently, there is just an official Docker image for Traefik that we use and there is a config file associated with it. Any changes in Traefik are not expected to break current setup anytime soon. When a plugin for Caddy is built with the similar promise, we can use it with more comfort.

@mholt

This comment has been minimized.

Owner

mholt commented Feb 22, 2018

Thanks for the details, @ibnesayeed -- I think that's helpful. In the meantime, anyone is welcome to write a plugin that does what you describe!

@lucaslorentz

This comment has been minimized.

Contributor

lucaslorentz commented Feb 22, 2018

@ibnesayeed
Have you tried https://github.com/lucaslorentz/caddy-docker-proxy ? I posted it some comments above.
It does watch for new services on docker swarm, it generates in-memory caddyfiles and reload caddy using signals.
It supports any configuration for caddy by converting docker service labels into caddy directives.

You can include the plugin on your own build, like this:
https://github.com/lucaslorentz/caddy-docker-proxy/blob/master/main.go#L5
Or just use the docker image I provided.

I've been using it for months in production. I still need to add it to caddy pages, but I really suck at writing docs 😄 . Any help would be welcome.

@ibnesayeed

This comment has been minimized.

Contributor

ibnesayeed commented Feb 22, 2018

@lucaslorentz this is fabulous. I casually looked at its code. It just needs some polish and documentation, then it should be added in the official Caddy docs.

@mholt

This comment has been minimized.

Owner

mholt commented Feb 22, 2018

@lucaslorentz Great! Let me know if I can help get it on the Caddy website for others to use. 👍

@lucaslorentz

This comment has been minimized.

Contributor

lucaslorentz commented Feb 22, 2018

Thanks @mholt @ibnesayeed
I will improve readme, tests and build, then I will open PRs to add it to caddy website.
I'm sure an experienced golang developer would see a lot to improve. So, review and PRs are welcome.

@ibnesayeed

This comment has been minimized.

Contributor

ibnesayeed commented Feb 22, 2018

I have reported a couple of issues in your repo @lucaslorentz.

@lucaslorentz

This comment has been minimized.

Contributor

lucaslorentz commented Feb 22, 2018

@ibnesayeed Awesome! Thanks for taking time to do that.

@lucaslorentz

This comment has been minimized.

Contributor

lucaslorentz commented Feb 24, 2018

@mholt
Just updated the repository and registered the plugin. It's being analyzed.
I felt this plugin didn't fit very well on the current format of description + examples.
That's because instead of creating Caddyfile configuration, the user must create labels on Docker Swarm services.
Maybe it's my lack of creativity. 😄

@lucaslorentz

This comment has been minimized.

Contributor

lucaslorentz commented Feb 24, 2018

@mholt
Just downloaded a build from Caddy Download and it worked!

@mholt

This comment has been minimized.

Owner

mholt commented Feb 24, 2018

@lucaslorentz Awesome! That's great news!

I felt this plugin didn't fit very well on the current format of description + examples.
That's because instead of creating Caddyfile configuration, the user must create labels on Docker Swarm services.

Indeed; most plugins do have some Caddyfile configuration. But you should feel free to put any kind of example in there that makes sense, even if it is not about Caddy.

So glad to hear that it worked! Thanks for registering your plugin! I'll try to tweet about it tomorrow.

Does that close this issue then?

@lucaslorentz

This comment has been minimized.

Contributor

lucaslorentz commented Feb 24, 2018

This issue requests something a bit different than the plugin, but a lot of things changed in docker scene since 2015, when the issue was opened.

The plugin is aligned with similar products, like Traefik and Kubernetes Ingress controller and is tailored to explore the most of Caddy features.

I vote on closing the issue.

@ibnesayeed

This comment has been minimized.

Contributor

ibnesayeed commented Feb 24, 2018

That's because instead of creating Caddyfile configuration, the user must create labels on Docker Swarm services.
Maybe it's my lack of creativity. smile

In this case these configurations are well-suited as Docker labels to allow ad-hoc routing. If we can think of some configurable behavior of thee plugin itself then that can go in a Caddyfile.

@ibnesayeed

This comment has been minimized.

Contributor

ibnesayeed commented Feb 24, 2018

Does that close this issue then?

In a broader sense it does implement the core of this feature request. Further improvements can be made to the plugin. I have created a few tickets in the repo in that direction.

Now that we have a plugin for Docker Swarm, we also need to build one for Kubernetes. However, that can be tracked in a separate ticket.

@mholt

This comment has been minimized.

Owner

mholt commented Feb 24, 2018

And I do think something like this is best implemented as a plugin -- at least, for the foreseeable future. So, thanks!

For Kubernetes, see https://github.com/wehco/caddy-ingress-controller or maybe https://github.com/drud-archive/router. :)

@mholt mholt closed this Feb 24, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment