Reload not working with docker

[Xaelias]

Hi guys!

So I've been running Prometheus in a docker on my nas since yesterday, so I'm still trying things, and changing the config a lot.

Thing is, as of now, I don't seem to be able to reload the config, except if I restart the container.
It's not that big of a deal in itself, but it's irritating...

So I tried:
curl -XPOST localhost:9090/-/reload from the nas itself.
And I also tried to attach a shell to the container and do a:
kill -HUF 1 # 1 being the pid of prometheus, grabbed from ps

In both cases, the only thing I see in the logs is:

time="2017-03-29T00:36:41Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" source="main.go:248"

Nothing more, nothing less. The config (grabbed from curl localhost:9090/config or in the UI itself) don't change, and the changes don't take effect AFAIK.

Am I missing something?
Is it specific to running prometheus in a container?

Thanks :-)

[brancz]

Did you directly mount the file into the docker container instead of the directory? Because when mounting single files into a docker container, they cannot be updated from the outside.

[Xaelias]

I did mount the file, but when doing a docker cp I was able to see the updated file.
I'll double check next time I update it (might not have time before next week now) and test with the file and the directory.

[juliusv]

Yeah, you have to mount in the entire directory, otherwise Prometheus will not see any changes. 99% certain that's the issue here and we've seen it before, so closing. Please reopen in case there is a different reason.

[Xaelias]

Finally got around to testing this! Just wanted to confirm that indeed, after mounting the entire directory, the reload now works.

Thanks guys :-)

[apakhomov]

In my case mounting the whole directory is inconvenient. Why it's impossible to always use inotifywatch of config.file specified at startup? I divide different configs in subdirectories in /etc/prometheus and they are being mounted from different sources. Mounting the whole /etc/prometheus for the only prometheus.yml causes headake

[juliusv]

@apakhomov Note that this is not a Prometheus issue, but a Docker issue. When you mount in a single file into a container, the container will not see any updates to it, so there's nothing that Prometheus can do.