where can I find prometheus logs?

[xixikaikai]

What did you do?
where can I find prometheus log?
What did you expect to see?
logs

• System information:
  Linux 3.10.0-514.2.2.el7.x86_64 x86_64
• Prometheus version:
  1.4.1

$ docker logs 86dfa5a
time="2017-01-24T07:50:47Z" level=info msg="Starting prometheus (version=1.4.1, branch=master, revision=2a89e8733f240d3cd57a6520b52c36ac4744ce12)" source="main.go:77"
time="2017-01-24T07:50:47Z" level=info msg="Build context (go=go1.7.3, user=root@e685d23d8809, date=20161128-09:59:22)" source="main.go:78"
time="2017-01-24T07:50:47Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" source="main.go:250"
time="2017-01-24T07:50:47Z" level=info msg="Loading series map and head chunks..." source="storage.go:354"
time="2017-01-24T07:50:47Z" level=info msg="0 series loaded." source="storage.go:359"
time="2017-01-24T07:50:47Z" level=info msg="Starting target manager..." source="targetmanager.go:63"
time="2017-01-24T07:50:47Z" level=info msg="Listening on :9090" source="web.go:248"
time="2017-01-24T07:55:47Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:549"
time="2017-01-24T07:55:47Z" level=info msg="Done checkpointing in-memory metrics and chunks in 59.669376ms." source="persistence.go:573"
time="2017-01-24T08:00:47Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:549"
time="2017-01-24T08:00:47Z" level=info msg="Done checkpointing in-memory metrics and chunks in 58.828337ms." source="persistence.go:573"
time="2017-01-24T08:05:47Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:549"
time="2017-01-24T08:05:47Z" level=info msg="Done checkpointing in-memory metrics and chunks in 58.198574ms." source="persistence.go:573"
time="2017-01-24T08:10:47Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:549"
time="2017-01-24T08:10:47Z" level=info msg="Done checkpointing in-memory metrics and chunks in 59.811229ms." source="persistence.go:573"
time="2017-01-24T08:15:47Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:549"
time="2017-01-24T08:15:47Z" level=info msg="Done checkpointing in-memory metrics and chunks in 57.654694ms." source="persistence.go:573"
time="2017-01-24T08:20:47Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:549"
time="2017-01-24T08:20:48Z" level=info msg="Done checkpointing in-memory metrics and chunks in 58.830557ms." source="persistence.go:573"
time="2017-01-24T08:25:48Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:549"
time="2017-01-24T08:25:48Z" level=info msg="Done checkpointing in-memory metrics and chunks in 57.968977ms." source="persistence.go:573"
time="2017-01-24T08:30:48Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:549"
time="2017-01-24T08:30:48Z" level=info msg="Done checkpointing in-memory metrics and chunks in 58.804957ms." source="persistence.go:573"
time="2017-01-24T08:35:48Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:549"
time="2017-01-24T08:35:48Z" level=info msg="Done checkpointing in-memory metrics and chunks in 59.490262ms." source="persistence.go:573"
time="2017-01-24T08:40:48Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:549"
time="2017-01-24T08:40:48Z" level=info msg="Done checkpointing in-memory metrics and chunks in 58.052496ms." source="persistence.go:573"
time="2017-01-24T08:45:48Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:549"
time="2017-01-24T08:45:48Z" level=info msg="Done checkpointing in-memory metrics and chunks in 62.90804ms." source="persistence.go:573"
time="2017-01-24T08:50:48Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:549"
time="2017-01-24T08:50:48Z" level=info msg="Done checkpointing in-memory metrics and chunks in 60.801515ms." source="persistence.go:573"

$ docker exec -it 86dfa5a /bin/bash
rpc error: code = 2 desc = oci runtime error: exec failed: exec: "/bin/bash": stat /bin/bash: no such file or directory

[xixikaikai]

I set up prometheus in docker container
and my config file is as follows:

{
"id": "/prometheus-dev-176",
"cmd": null,
"cpus": 0.25,
"mem": 1024,
"disk": 0,
"instances": 1,
"constraints": [
[
"hostname",
"CLUSTER",
"172.19.0.176"
]
],
"container": {
"type": "DOCKER",
"volumes": [
{
"containerPath": "/etc/prometheus",
"hostPath": "/home/ec2-user/kay/data/prometheus/etc/prometheus",
"mode": "RO"
},
{
"containerPath": "/prometheus",
"hostPath": "/home/ec2-user/kay/data/prometheus/data",
"mode": "RW"
}
],
"docker": {
"image": "XXXXXXXXXXX:5000/prometheus:1.X.1",
"network": "BRIDGE",
"portMappings": [
{
"containerPort": 9090,
"hostPort": 31090,
"servicePort": 10055,
"protocol": "tcp",
"labels": {}
}
],
"privileged": true,
"parameters": [],
"forcePullImage": false
}
},
"healthChecks": [
{
"path": "/",
"protocol": "TCP",
"portIndex": 0,
"gracePeriodSeconds": 600,
"intervalSeconds": 60,
"timeoutSeconds": 20,
"maxConsecutiveFailures": 5,
"ignoreHttp1xx": false
}
],
"labels": {
"HAPROXY_GROUP": "dev"
},
"portDefinitions": [
{
"port": 10055,
"protocol": "tcp",
"labels": {}
}
],
"args": [
"-storage.local.memory-chunks=262144",
"-storage.local.max-chunks-to-persist=131072",
"-config.file=/etc/prometheus/prometheus.yml"
]
}

[xixikaikai]

Now I need to centralise the logs from prometheus to ELK, but for now, I can only see logs from "docker logs" command, which can not be filter to logstash.
Can we map the logs of prometheus to host disk?

[andrewhowdencom]

I think Docker already writes these logs to disk. Check

/var/lib/docker/containers/*/*.log

(Note: In my case, I puppet docker through k8s, so it might be different)

[xixikaikai]

@andrewhowdencom Can we just map this log to another place when set up prometheus by config?

[andrewhowdencom]

You could get Prometheus to log to disk and mount in a host volume

[xixikaikai]

@andrewhowdencom Thanks for your advice. BTW, do you know the config item in prometheus which will map a host volume to provide it to log?

[andrewhowdencom]

The host volume stuff you'll have to do in Docker, but the Prometheus configuration is via the CLI.

$ prometheus -h
...
 == LOG ==
  
   -log.format "\"logger:stderr\""
      Set the log target and format. Example: 
      "logger:syslog?appname=bob&local=7" or "logger:stdout?json=true"

Edit: Alternatively, I'd think about just piping that output to disk somewhere in a parent process.

/path/to/prometheus 2> path/to/your/log/file.log

[xixikaikai]

@andrewhowdencom thanks very much. I will try it. Maybe syslog can be a good channel to logstash. If not, I will use disk volume mapping.

[burka]

You can always use another docker logging-driver to collect all docker logs and send them to ELK.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.