Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No metrics prometheus-nginxlog-exporter! #358

Open
ghost opened this issue Oct 24, 2023 · 3 comments
Open

No metrics prometheus-nginxlog-exporter! #358

ghost opened this issue Oct 24, 2023 · 3 comments
Labels
bug feedback-required

Comments

@ghost
Copy link

ghost commented Oct 24, 2023 •
edited by ghost
Loading

Hello, please help me figure out the reason for the lack of metrics

the exporter is installed in the container with the command

$ docker run -d --name nginxlog-exporter -p 4040:4040 -v /var/log/nginx:/mnt/nginxlogs -v /etc/prometheus/prometheus-nginxlog-exporter.hcl:/etc/prometheus-nginxlog-exporter.hcl quay.io/martinhelmich/prometheus-nginxlog-exporter -config-file /etc/prometheus-nginxlog-exporter.hcl

NGINX logs are configured like this:

  1. /etc/nginx/nginx.conf
log_format upstream_time '$remote_addr - $remote_user [$time_local] '
					 '"$request" $status $body_bytes_sent '
					 '"$http_referer" "$http_user_agent" '
					 'rt=$request_time uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';

access_log /var/log/nginx/access.log upstream_time;
  1. /etc/nginx/sites-available/itscript.ru.conf
access_log /var/log/nginx/access_itscript.ru.log upstream_time;

As a result, I get statistics like this:

# HELP allnginx_parse_errors_total Total number of log file lines that could not be parsed
# TYPE allnginx_parse_errors_total counter
allnginx_parse_errors_total{vhost="all"} 0
allnginx_parse_errors_total{vhost="itscript.ru"} 0
# HELP prometheus_nginxlog_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, goversion from which prometheus_nginxlog_exporter was built, and the goos and goarch for the build.
# TYPE prometheus_nginxlog_exporter_build_info gauge
prometheus_nginxlog_exporter_build_info{branch="v1.11.0",goarch="amd64",goos="linux",goversion="go1.18.10",revision="76552e7cadc1f76b4e28b5c872412db4a091aca0",tags="unknown",version="1.11.0"} 1

In the docker container log

$ docker logs a1991fa0a5ee
2023-10-24T09:21:49.635Z        info    prometheus-nginxlog-exporter/main.go:162        loading configuration file /etc/prometheus-nginxlog-exporter.hcl
2023-10-24T09:21:49.635Z        warn    config/struct_namespace.go:98   No globs for /var/log/nginx/access.log
2023-10-24T09:21:49.636Z        warn    config/struct_namespace.go:98   No globs for /var/log/nginx/access_itscript.ru.log
2023-10-24T09:21:49.636Z        info    prometheus-nginxlog-exporter/main.go:139        starting listener for namespace nginx
2023-10-24T09:21:49.636Z        info    prometheus-nginxlog-exporter/main.go:139        starting listener for namespace default
2023-10-24T09:21:49.636Z        info    prometheus-nginxlog-exporter/main.go:148        running HTTP server on address 0.0.0.0:4040, serving metrics at /metrics
2023-10-24T09:21:49.636Z        info    tail@v1.4.8/tail.go:212 Waiting for /var/log/nginx/access.log to appear...
2023-10-24T09:21:49.637Z        info    tail@v1.4.8/tail.go:212 Waiting for /var/log/nginx/access_itscript.ru.log to appear...
2023-10-24T09:25:05.735Z        info    prometheus-nginxlog-exporter/main.go:101        caught term terminated. exiting
2023-10-24T09:25:06.123Z        info    prometheus-nginxlog-exporter/main.go:162        loading configuration file /etc/prometheus-nginxlog-exporter.hcl
2023-10-24T09:25:06.124Z        warn    config/struct_namespace.go:98   No globs for /var/log/nginx/access.log
2023-10-24T09:25:06.124Z        warn    config/struct_namespace.go:98   No globs for /var/log/nginx/access_itscript.ru.log
2023-10-24T09:25:06.124Z        info    prometheus-nginxlog-exporter/main.go:139        starting listener for namespace all
2023-10-24T09:25:06.124Z        info    prometheus-nginxlog-exporter/main.go:139        starting listener for namespace itscript.ru
2023-10-24T09:25:06.124Z        info    prometheus-nginxlog-exporter/main.go:148        running HTTP server on address 0.0.0.0:4040, serving metrics at /metrics
2023-10-24T09:25:06.125Z        info    tail@v1.4.8/tail.go:212 Waiting for /var/log/nginx/access.log to appear...
2023-10-24T09:25:06.125Z        info    tail@v1.4.8/tail.go:212 Waiting for /var/log/nginx/access_itscript.ru.log to appear...

In the host access log
/var/log/nginx/access_itscript.ru.log

85.354.777.7 - - [24/Oct/2023:12:33:41 +0300] "GET /i.php HTTP/1.1" 404 197 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36" rt=0.000 uct="-" uht="-" urt="-"
85.354.777.7 - - [24/Oct/2023:12:33:41 +0300] "GET /i.php HTTP/1.1" 404 197 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36" rt=0.000 uct="-" uht="-" urt="-"
85.354.777.7 - - [24/Oct/2023:12:33:41 +0300] "GET /i.php HTTP/1.1" 404 197 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36" rt=0.000 uct="-" uht="-" urt="-"
85.354.777.7 - - [24/Oct/2023:12:33:41 +0300] "GET /i.php HTTP/1.1" 404 197 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36" rt=0.000 uct="-" uht="-" urt="-"
85.354.777.7 - - [24/Oct/2023:12:33:42 +0300] "GET /i.php HTTP/1.1" 404 197 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36" rt=0.000 uct="-" uht="-" urt="-"
85.354.777.7 - - [24/Oct/2023:12:33:42 +0300] "GET /i.php HTTP/1.1" 404 197 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36" rt=0.000 uct="-" uht="-" urt="-"
85.354.777.7 - - [24/Oct/2023:12:33:42 +0300] "GET /i.php HTTP/1.1" 404 197 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36" rt=0.000 uct="-" uht="-" urt="-"
85.354.777.7 - - [24/Oct/2023:12:33:42 +0300] "GET /i.php HTTP/1.1" 404 197 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36" rt=0.000 uct="-" uht="-" urt="-"
46.138.888.15 - - [24/Oct/2023:12:38:04 +0300] "GET / HTTP/1.1" 200 163 "-" "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Mobile Safari/537.36" rt=0.001 uct="0.000" uht="0.000" urt="0.000"

Exporter configuration file
/etc/prometheus/prometheus-nginxlog-exporter.hcl

listen {
  port = 4040
}

namespace "all" {
  source = {
    files = [
      "/var/log/nginx/access.log"
    ]
  }
  metrics_override = { prefix = "allnginx" }
  namespace_label = "vhost"
  format = "$remote_addr - $remote_user [$time_local] $request_method \"$request_uri\" $status"


  labels {
    app = "all"
  }
}

namespace "itscript.ru" {
  source = {
    files = [
      "/var/log/nginx/access_itscript.ru.log"
    ]
  }
  metrics_override = { prefix = "allnginx" }
  namespace_label = "vhost"
  format = "$remote_addr - $remote_user [$time_local] $request_method \"$request_uri\" $status"


  labels {
    app = "itscript.ru"
  }
}

Help me please!
@ghost ghost added the bug label Oct 24, 2023
@martin-helmich
Copy link
Owner

Looks like the log format configured in your exporter configuration doesn't match the actual log files; pay attention to the double quotes and the additional fields at the end of the log lines.

Hope that helps!

@ghost
Copy link
Author

ghost commented Oct 24, 2023 •
edited by ghost
Loading

Looks like the log format configured in your exporter configuration doesn't match the actual log files; pay attention to the double quotes and the additional fields at the end of the log lines.

Hope that helps!

Adjusted the log format in the prometheus-nginxlog-exporter.hcl config to match the format specified in /etc/nginx/nginx.conf

/etc/nginx/nginx.conf

log_format upstream_time '$remote_addr - $remote_user [$time_local] '
					 '"$request" $status $body_bytes_sent '
					 '"$http_referer" "$http_user_agent" '
					 'rt=$request_time uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';

/etc/prometheus/prometheus-nginxlog-exporter.hcl

format = "$remote_addr - $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" rt=$request_time uct=\"$upstream_connect_time\" uht=\"$upstream_header_time\" urt=\"$upstream_response_time\""

The log is written like this:

85.354.777.7 - - [24/Oct/2023:16:37:09 +0300] "GET / HTTP/1.1" 200 163 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36" rt=0.001 uct="0.000" uht="0.000" urt="0.000"

I restarted NGINX and the prometheus-nginxlog-exporter container, but it didn’t last long...

in the container log

2023-10-24T13:36:16.760Z info prometheus-nginxlog-exporter/main.go:101 caught term terminated. exiting
2023-10-24T13:36:17.144Z info prometheus-nginxlog-exporter/main.go:162 loading configuration file /etc/prometheus-nginxlog-exporter.hcl
2023-10-24T13:36:17.145Z warn config/struct_namespace.go:98 No globs for /var/log/nginx/access.log
2023-10-24T13:36:17.145Z warn config/struct_namespace.go:98 No globs for /var/log/nginx/access_itscript.ru.log
2023-10-24T13:36:17.145Z info prometheus-nginxlog-exporter/main.go:139 starting listener for namespace all
2023-10-24T13:36:17.145Z info prometheus-nginxlog-exporter/main.go:139 starting listener for namespace itscript.ru
2023-10-24T13:36:17.145Z info prometheus-nginxlog-exporter/main.go:148 running HTTP server on address 0.0.0.0:4040, serving metrics at /metrics
2023-10-24T13:36:17.146Z info tail@v1.4.8/tail.go:212 Waiting for /var/log/nginx/access.log to appear...
2023-10-24T13:36:17.146Z info tail@v1.4.8/tail.go:212 Waiting for /var/log/nginx/access_itscript.ru.log to appear.

@ghost
Copy link
Author

ghost commented Oct 24, 2023 •
edited by ghost
Loading

The problem was that when starting the container, monitoring was done in a directory inside the container
-v /var/log/nginx:/mnt/nginxlogs

which differs from the directory specified in the exporter configuration file
/etc/prometheus-nginxlog-exporter.hcl

source = {
	files = [
	  "/var/log/nginx/access.log"
	]
}

That is, the correct command to launch the container and mount the folder with logs should be

$ docker run -d --name nginxlog-exporter -p 4040:4040 -v /var/log/nginx:/var/log/nginx -v /etc/prometheus-nginxlog-exporter.hcl:/etc/prometheus-nginxlog-exporter.hcl quay.io/martinhelmich/prometheus-nginxlog-exporter -config-file /etc/prometheus-nginxlog-exporter.hcl

This is why there were entries in the container log that the exporter is waiting for files to appear

2023-10-24T13:36:17.146Z info tail@v1.4.8/tail.go:212 Waiting for /var/log/nginx/access.log to appear...
2023-10-24T13:36:17.146Z info tail@v1.4.8/tail.go:212 Waiting for /var/log/nginx/access_itscript.ru.log to appear.

Please add this to the documentation!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug feedback-required
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@martin-helmich