Latest (2 hr ago) caddy/caddy docker image locks down admin API, regardless of config

[collinbachi]

Expected:

I set enforce_origin to false, and I can reach the admin API at any origin. (https://caddyserver.com/docs/json/admin/#origins isn't explicit on the matter, but that's what I expect)

Actual:

I set enforce_origin to false, and I can't reach the admin API at any address other than the listening address.

Temporary solution:
Rolled back to caddy/caddy:2.0.0-rc.2-alpine and confirmed I could again reach the admin API remotely.

[francislavoie]

Can you give more details? What's your full Caddyfile or JSON config? How did you run Docker (docker run command or docker-compose.yml file)? What version were you using before you rolled back exactly? caddy:2.0.0-rc.3 or caddy/caddy:2.0.0-rc.3?

[collinbachi]

I think this is the relevant section of the JSON config:

"admin":{"listen":"0.0.0.0:2019"}

Before the rollback I was running the caddy/caddy:latest tag released 2 hours ago: https://hub.docker.com/layers/caddy/caddy/latest/images/sha256-81b8ed2e47aeed5ced4addf3993884bed4766b09d891a9fa671e2ec7c448689d?context=explore

I don't know what numbered version that corresponds to. I only noticed because the image updated itself at some point and suddenly my other containers couldn't reach the Caddy admin API.

I'll see if I can reproduce locally with minimal configuration before sharing the specifics of my deployment. I don't think it will require any special configuration, if I understand the issue correctly

[francislavoie]

FYI, from now on you should use the caddy docker image instead of caddy/caddy, we now have an official image!

Clear reproduction steps would be helpful, with curl -v, etc.

[postgetme]

same issue here.

one console:
echo hello > index.html
docker run -p 80:80 -p 2019:2019 -v $PWD/index.html:/usr/share/caddy/index.html -v caddy_data:/data caddy
{"level":"info","ts":1587030105.4372017,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1587030105.4415984,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1587030105.441695,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
{"level":"info","ts":1587030105.4417708,"logger":"tls","msg":"cleaned up storage units"}
2020/04/16 09:41:45 [INFO][cache:0xc0006062d0] Started certificate maintenance routine
{"level":"info","ts":1587030105.44193,"msg":"autosaved config","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1587030105.4419374,"msg":"serving initial configuration"}

anohter console:
curl localhost:2019/config/
curl: (56) Recv failure: Connection reset by peer

enter caddy container:
apk add curl
curl localhost:2019/config/
success get json config

[hairyhenderson]

I'm fairly certain that this is due to some changes in RC3:

Enhanced default security of admin endpoint

The commit in question seems to be caddyserver/caddy@a3bdc22

@postgetme: I don't think you're having the same issue as @collinbachi - @collinbachi has the "admin":{"listen":"0.0.0.0:2019"} config set, which is important when running in Docker (since the container's localhost is not the same as the host's localhost, unless you're sharing the host's network namespace).

@collinbachi If your config isn't working, I have a feeling that the issues is with Caddy RC3 itself, not in the Docker image.

@mholt I feel like the issue is in the allowedOrigins func somwhere, but I'm not clear on where (caddyserver/caddy@a3bdc22#diff-39b065607d3d0839bdc97f997d6a430cR145)

[hairyhenderson]

I set enforce_origin to false, and I can't reach the admin API at any address other than the listening address.

@collinbachi can you clarify what you mean by that? which address specifically are you trying to reach the API at, and which which exact command?

[hairyhenderson]

@collinbachi maybe post the output of a curl -v command to the endpoint?

[hairyhenderson]

@collinbachi and your full Caddy configuration would be useful too 🙂

[mholt]

I suspect the reason is indeed because of that linked commit, however, it's impossible to know for sure without more info, I think curl -v will give us that information.

By default, the admin endpoint only allows requests to the host to which the listener binds its socket; the default is localhost (as a special case, allowing localhost or ::1 or 127.0.0.1, etc) -- if you bind to something like foobar:2019, then the only allowed Host header is foobar. Or if you bind to 10.1.2.3:2019 then the only allowed Host header is 10.1.2.3. I guess if you bind to :2019 (all interfaces) or 0.0.0.0:2019 (all IPv4 interfaces), or [::1]:2019 (all IPv6 interfaces), then we need some special logic to accommodate that as well.

To clarify, this is all about the default permissions. As a workaround, you can set the "origins" property next to "listen" with a list of hostnames you will be accessing the admin API from. (The docs are slightly outdated, I will update them soon. You don't have to enable origin checking.)

I'll need to decide if we change the current behavior or leave it as-is. If I change it, the only change I can think of would be to disable the Host header checking if binding to a wildcard interface, which is (mildly) less secure. If I leave it as-is, then you still get better security and can simply set the "origins" field yourself with known, trusted Host values.

Does that make sense? Any thoughts?

(Of course, this is all assuming my hunch is correct; hopefully curl -v will help us know for sure.)

[hairyhenderson]

I guess if you bind to :2019 (all interfaces) or 0.0.0.0:2019 (all IPv4 interfaces), or [::1]:2019 (all IPv6 interfaces), then we need some special logic to accommodate that as well.

Yeah, that's what I had wondered, except shouldn't the enforce_origin: false option have disabled all of that checking?

[mholt]

Yeah, that's what I had wondered, except shouldn't the enforce_origin: false option have disabled all of that checking?

No, that only affects CORS enforcement. The default is false anyway.

[francislavoie]

@mholt we should spend more time looking to improve the unix socket situation for admin (especially on Linux, Windows does seem to be a bit of a dead end there unfortunately). I think using a unix socket for the admin endpoint instead of 0.0.0.0:2019 when running in Docker makes a lot of sense.

E.g. if you specify a unix socket, when Caddy is stopped the unix socket file sticks around. Running Caddy again, it complains that the socket file already exists - Caddy should remove the socket when stopped.

Also, as mentioned by someone else previously, the socket doesn't always have the right permissions on it. I'm pretty sure Caddy should set those to something sensible after starting.

[hairyhenderson]

Ah! ok, maybe we need an enforce_host option?

[hairyhenderson]

I think using a unix socket for the admin endpoint instead of 0.0.0.0:2019 when running in Docker makes a lot of sense.

It'd help with network security, to be sure, but sharing sockets can be a bit tricky with Docker. It's been a
long time since I've needed to do that, but I remember it being a pain... Plus it makes the API more awkward to access - users just playing around with curl would need to know to use --unix-socket and how to craft the URL as well.

[mholt]

If we do change things, one possibility is to disable Host checking if binding to a wildcard interface and then emitting a warning in the logs. If you're allowing all network interfaces to connect to your admin endpoint, then you better know what you're doing, I guess?

@francislavoie

we should spend more time looking to improve the unix socket situation for admin (especially on Linux, Windows does seem to be a bit of a dead end there unfortunately). I think using a unix socket for the admin endpoint instead of 0.0.0.0:2019 when running in Docker makes a lot of sense.

I believe the admin endpoint does let you bind to a Unix socket. It should work in rc3, if it doesn't, let me know. But in my testing (on a mac) it works fine.

[francislavoie]

I believe the admin endpoint does let you bind to a Unix socket. It should work in rc3, if it doesn't, let me know. But in my testing (on a mac) it works fine.

Yes, the issues I'm describing is specifically related to that. It works on first run, but subsequent runs don't without manually deleting the socket

[mholt]

That's weird... is that true of all unix sockets, and does that happenf or listen addresses for the HTTP server too? Might be worth an issue in the main repo to track that. (I guess I don't know much about how unix sockets work.)

[francislavoie]

Re unix sockets opened an issue: caddyserver/caddy#3269

[collinbachi]

I set enforce_origin to false, and I can't reach the admin API at any address other than the listening address.

After further testing, I take this back. I can reach the admin API at the hosts listed in "origins".

This resolves my issue, since I can whitelist the hostname I want to use to reach Caddy (which seems entirely reasonable to me, from a security perspective.) Thanks, and sorry for the misunderstanding!

[mholt]

No worries, I just pushed a change that disables that check if you listen on the wildcard interface. I think it's reasonable if you go to the trouble to listen on an open interface (we also emit a warning in the logs).

[coatezy]

I was wondering if this was resolved or is there still work to be done upstream? I’m seeing the same issue using the latest build. Using docker-compose I can connect via 0.0.0.0 once listen has been updated to 0.0.0.0:2019 and I can also connect via a given hostname if set in the list of origins, but if I deploy the same image and config to AWS ECS, the connection to the admin API simply fails to connect, although I can connect to port 80 and 443.

[mholt]

@coatezy I think you want this fix: caddyserver/caddy@f5ccb90 which hasn't been released yet, but should be next week.

[coatezy]

Thanks @mholt. You are right, it was late and had been hitting my head against the desk for a good hour or so. I woke up this morning and realised you had implemented a fix. I confirmed that my setup did work with the RC2 release as mentioned in comments above and have since got everything working with RC3. It is good to know that the fix is being released soon.

I now need to work out how to pump all logs to an HTTP endpoint so I can work out when certificates have been generated successfully or not, but thats another thing.

[mholt]

I now need to work out how to pump all logs to an HTTP endpoint so I can work out when certificates have been generated successfully or not, but thats another thing.

That'll probably get easier with a later version of Caddy in the 2.x tree.

[sykire]

My current workaround is to do something like this:

CaddyConfig.json

{
  "admin": {
    "listen": "0.0.0.0:2019"
  }
}

Dockerfile

FROM caddy:2-alpine

COPY CaddyConfig.json /etc/caddy/

EXPOSE 80
EXPOSE 443
EXPOSE 2019

CMD caddy run --config /etc/caddy/CaddyConfig.json --resume

Put then in the same folder then build my docker image.
I don't know if is the best approach but works for me.

[francislavoie]

Yep @DrecDroid that's a perfectly reasonable approach.

FYI, those EXPOSE lines don't actually do anything, they're essentially just comments to the reader. You still need to use -p when running (or the equivalent docker compose yml).

[sykire]

Yes, of course. Then I have to run as usual but now I have the CaddyConfig.json as I needed.

[beeekind]

Still a little confused on this one. I'm attempting to pull metrics from localhost:2019/metics from a separate docker container on the same docker network - so the origin address is dynamic. What settings should I use in my caddyfile to enable access to that particular endpoint? If not could I use a proxy directive to achieve the same effect?

Current Caddyfile:

{
  debug
  admin {
    origins *
  }
}

localhost {
  reverse_proxy golang:4200
}

localhost:9090 {
  reverse_proxy prometheus:9090
}

docker-compose

version: '3.8'

services:
  caddy:
    container_name: caddy
    image: 'caddy:latest'
    networks:
      - global
    ports:
      - '80:80'
      - '443:443'
      - '9090:9090'
    volumes:
      - caddy-data:/data 
      - caddy-config:/config
      - ./Caddyfile:/etc/caddy/Caddyfile 

  prometheus:
    container_name: prometheus 
    image: prom/prometheus
    networks: 
      - global 
    volumes:
      - prometheus-storage:/prometheus
      - ./prometheus.yml:/etc/prometheus/prometheus.yml   

volumes:
  caddy-data:
  caddy-config:
  prometheus-storage:

networks:
  global:
    external: true

and finally prometheus.yaml for good measure

global:
    scrape_interval: 15s
    scrape_timeout: 10s
    evaluation_interval: 15s

scrape_configs:
- job_name: caddy
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http 
  static_configs:
  - targets:
    - caddy:2019
  

[francislavoie]

@b3ntly you need to change the listen address to 0.0.0.0:2019.

For next time, please ask your usage questions on the Caddy community forums. We prefer to keep the GitHub issue board for bugs and feature requests. Don't forget to fill out the thread template so we can help you!