Skip to content

Improve the Query Log #1548

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
roidelapluie merged 2 commits into from
Feb 13, 2020
Merged

Improve the Query Log #1548

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
866a708
Improve the Query Log
Feb 12, 2020
362dc57
Address Brian's comments
Feb 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 25 additions & 35 deletions content/docs/guides/query-log.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,14 @@ sort_rank: 1
# Using the Prometheus query log

Prometheus has the ability to log all the queries run by the engine to a log
file. This guide demonstrates how to use that log file, which fields it does
contain and provides advanced tips about how to operate the log file.

The query log has been introduced in v2.16.0 and is disabled by default.
file, as of 2.16.0. This guide demonstrates how to use that log file, which
fields it contains, and provides advanced tips about how to operate the log
file.

## Enable the query log

The query log can be enabled and disabled at runtime. It can therefore be
activated when you want to investigate slownesses or high load on your
Prometheus instance.
The query log can be toggled at runtime. It can therefore be activated when you
want to investigate slownesses or high load on your Prometheus instance.

To enable or disable the query log, two steps are needed:

Expand All @@ -25,42 +23,34 @@ To enable or disable the query log, two steps are needed:
### Logging all the queries to a file

This example demonstrates how to log all the queries to
a file called `/var/log/prometheus/query.log`.

First, let's ensure that Prometheus has write access to a directory
`/var/log/prometheus`. We will assume that Prometheus is running under a user
called Prometheus:

```shell
# mkdir /var/log/prometheus
# chown prometheus: /var/log/prometheus
```
a file called `/prometheus/query.log`. We will assume that `/prometheus` is the
data directory and that Prometheus has write access to it.

Then, adapt the `prometheus.yml` configuration file:
First, adapt the `prometheus.yml` configuration file:

```yaml
global:
scrape_interval: 15s
evaluation_interval: 15s
query_log_file: /var/log/prometheus/query.log
query_log_file: /prometheus/query.log
scrape_configs:
- job_name: 'prometheus'
- targets: ['localhost:9090']
```

And, reload the Prometheus configuration:
Then, [reload](../prometheus/latest/management_api/#reload) the Prometheus configuration:


```shell
$ curl -X POST http://127.0.0.1:9090/-/reload
```

Or, if Prometheus is not launched with `--web.enable-lifecycle`:
Or, if Prometheus is not launched with `--web.enable-lifecycle`, and you're not
running on Windows, you can trigger the reload by sending a SIGHUP to the
Prometheus process.

```shell
$ killall -HUP prometheus
```

The file `/var/log/prometheus/query.log` should now exist and all the queries
The file `/prometheus/query.log` should now exist and all the queries
will be logged to that file.

To disable the query log, repeat the operation but remove `query_log_file` from
Expand Down Expand Up @@ -88,7 +78,7 @@ number of queries that could not be logged.
## Format of the query log

The query log is a JSON-formatted log. Here is an overview of the fields
present on any query:
present for a query:

```
{
Expand All @@ -112,17 +102,17 @@ present on any query:
}
```

- params: the query itself. The start and end timestamp, the step and the actual
query.
- stats: statistics. Currently, it contains internal engine timers.
- ts: timestamp of the log line.
- `params`: The query. The start and end timestamp, the step and the actual
query statement.
- `stats`: Statistics. Currently, it contains internal engine timers.
- `ts`: The timestamp when the query ended.

Additionally, depending on what triggered the request, you will have additional
fields in the JSON lines.

### API Queries and consoles

HTTP requests contain the client IP, the method and the path:
HTTP requests contain the client IP, the method, and the path:

```
{
Expand Down Expand Up @@ -161,14 +151,14 @@ of the file and the name of the group:
Prometheus will not rotate the query log itself. Instead, you can use external
tools to do so.

One of those tools is logrotate. Please refer to the documentation of your
operating system for the initial logrotate setup.
One of those tools is logrotate. It is enabled by default on most Linux
distributions.

Once it is set up, here is an example of file you can add as
Here is an example of file you can add as
`/etc/logrotate.d/prometheus`:

```
/var/log/prometheus/query.log {
/prometheus/query.log {
daily
rotate 7
compress
Expand Down