2.8.0 file_sd parsing issue during config reload

[Linux-oiD]

I'm using official Prometheus docker image:

Key	Value
Version	2.8.0
Revision	5936949
Branch	HEAD
Build user	root@4c4d5c29b71f
Build Date	20190312-07:46:58
GO version	go1.11.5

One of my target scrape config looks like this:

 - job_name: 'cluster'
    file_sd_configs:
    - files:
      - 'targets/cluster.yml'
      refresh_interval: 5m

I have simple sidecar container for manage configuration. It's watching for changes in configuration files and send POST request to prometheus /-/reload endpoint. Flag --web.enable-lifecycle is enabled.

I see next errors during config reload request:

2019-03-19 14:30:18,234 [__main__    ] INFO     Application started
2019-03-19 14:30:18,235 [__main__    ] INFO     Bucket name: prometheus-configs
2019-03-19 14:30:18,235 [__main__    ] INFO     Prometheus reload url: http://prometheus:9090/-/reload
2019-03-19 14:30:18,235 [__main__    ] INFO     Prometheus config path: /etc/prometheus-configs
. . .
2019-03-21 10:18:24,845 [__main__    ] INFO     Refreshing prometheus configs...
2019-03-21 10:18:25,276 [__main__    ] ERROR    Reload response code 405

logs of prometheus looks like this:

level=error ts=2019-03-21T10:18:25.264343304Z caller=file.go:214 component="discovery manager scrape" discovery=file msg="Error adding file watch" path=/etc/prometheus-configs/prometheus/targets err="no such file or directory"
level=error ts=2019-03-21T10:18:25.26446974Z caller=file.go:214 component="discovery manager scrape" discovery=file msg="Error adding file watch" path=/etc/prometheus-configs/prometheus/targets err="no such file or directory"

Prometheus continues working with old-one config.

But if I restart container - all works fine. Configuration is valid:

level=warn ts=2019-03-21T10:53:14.890888023Z caller=main.go:484 msg="Received SIGTERM, exiting gracefully..."
level=info ts=2019-03-21T10:53:14.890973816Z caller=main.go:509 msg="Stopping scrape discovery manager..."
level=info ts=2019-03-21T10:53:14.891002056Z caller=main.go:523 msg="Stopping notify discovery manager..."
level=info ts=2019-03-21T10:53:14.891029962Z caller=main.go:545 msg="Stopping scrape manager..."
level=info ts=2019-03-21T10:53:14.891071305Z caller=main.go:505 msg="Scrape discovery manager stopped"
level=info ts=2019-03-21T10:53:14.891152338Z caller=main.go:519 msg="Notify discovery manager stopped"
level=info ts=2019-03-21T10:53:14.891265695Z caller=manager.go:736 component="rule manager" msg="Stopping rule manager..."
level=info ts=2019-03-21T10:53:14.891422773Z caller=manager.go:742 component="rule manager" msg="Rule manager stopped"
level=info ts=2019-03-21T10:53:14.891558171Z caller=main.go:539 msg="Scrape manager stopped"
level=info ts=2019-03-21T10:53:14.948922444Z caller=notifier.go:521 component=notifier msg="Stopping notification manager..."
level=info ts=2019-03-21T10:53:14.948969387Z caller=main.go:708 msg="Notifier manager stopped"
level=info ts=2019-03-21T10:53:14.949620219Z caller=main.go:720 msg="See you next time!"
level=warn ts=2019-03-21T10:53:17.594347191Z caller=main.go:274 deprecation_notice="'storage.tsdb.retention' flag is deprecated use 'storage.tsdb.retention.time' instead."
level=info ts=2019-03-21T10:53:17.594540492Z caller=main.go:321 msg="Starting Prometheus" version="(version=2.8.0, branch=HEAD, revision=59369491cfdfe8dcb325723d6d28a837887a07b9)"
level=info ts=2019-03-21T10:53:17.594587249Z caller=main.go:322 build_context="(go=go1.11.5, user=root@4c4d5c29b71f, date=20190312-07:46:58)"
level=info ts=2019-03-21T10:53:17.594636622Z caller=main.go:323 host_details="(Linux 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 140f96048c3e (none))"
level=info ts=2019-03-21T10:53:17.594686501Z caller=main.go:324 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2019-03-21T10:53:17.594731906Z caller=main.go:325 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2019-03-21T10:53:17.600213629Z caller=main.go:640 msg="Starting TSDB ..."
level=info ts=2019-03-21T10:53:17.600309934Z caller=web.go:418 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2019-03-21T10:53:17.601055865Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1553004067094 maxt=1553061600000 ulid=01D6DC42BD4XE0QHEZSE2R75FX
level=info ts=2019-03-21T10:53:17.601219548Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1553061600000 maxt=1553126400000 ulid=01D6F31XASNX1PTFPTQXB3Q2R8
level=info ts=2019-03-21T10:53:17.601368398Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1553148000000 maxt=1553155200000 ulid=01D6FQN1N5QY30K1VXE8QNDBTS
level=info ts=2019-03-21T10:53:17.601485227Z caller=repair.go:48 component=tsdb msg="found healthy block" mint=1553126400000 maxt=1553148000000 ulid=01D6FQN287RT8145WWDXKY162P
level=warn ts=2019-03-21T10:53:22.187566368Z caller=head.go:450 component=tsdb msg="unknown series references" count=16726
level=info ts=2019-03-21T10:53:22.203264231Z caller=main.go:655 msg="TSDB started"
level=info ts=2019-03-21T10:53:22.20345203Z caller=main.go:724 msg="Loading configuration file" filename=/etc/prometheus-configs/prometheus/prometheus.yml
level=info ts=2019-03-21T10:53:22.230800407Z caller=main.go:751 msg="Completed loading of configuration file" filename=/etc/prometheus-configs/prometheus/prometheus.yml
level=info ts=2019-03-21T10:53:22.23088792Z caller=main.go:609 msg="Server is ready to receive web requests."

My config changes doesn't affect scrape target listed above, this is only one target with file_sd in configuration.

Seems like / in filepath is parsing incorrect during config reload flow.

[brian-brazil]

What's the PWD of the Prometheus?

[Linux-oiD]

@brian-brazil what do you mean? Prometheus running in docker container.
Here mounts and config parts from docker inspect:

            {
                "Type": "bind",
                "Source": "/etc/prometheus-configs",
                "Destination": "/etc/prometheus-configs",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "prometheus_data",
                "Source": "/var/lib/docker/volumes/prometheus_data/_data",
                "Destination": "/prometheus",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/etc/hosts",
                "Destination": "/etc/hosts",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "140f96048c3e",
            "Domainname": "",
            "User": "nobody",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "9090/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "--config.file=/etc/prometheus-configs/prometheus/prometheus.yml",
                "--storage.tsdb.path=/prometheus",
                "--storage.tsdb.retention=15d",
                "--web.console.libraries=/usr/share/prometheus/console_libraries",
                "--web.console.templates=/usr/share/prometheus/consoles",
                "--web.external-url=http://prometheus.infra:9090",
                "--web.enable-lifecycle",
                "--web.enable-admin-api"
            ],
            "Image": "prom/prometheus",
            "Volumes": {
                "/etc/hosts": {},
                "/etc/prometheus-configs": {},
                "/prometheus": {}
            },
            "WorkingDir": "/prometheus",
            "Entrypoint": [
                "/bin/prometheus"
            ],
            "OnBuild": null,
            "Labels": {
                "maintainer": "The Prometheus Authors <prometheus-developers@googlegroups.com>"
            }
        },

[brian-brazil]

I can't reproduce this. Are you sure that's your config file and that it didn't change?