Avoid "GET /server-status?auto HTTP/1.1" in apache log files

[Vartkat]

Question summary

Avoid Apache log overload

OS / Environment

Ubuntu 16.04.5 LTS

Component Name

Apache and Web_Log

Expected results

I'm running Netdata along with Virtualmin.
Virtualmin is managing some virtual hosts, each hosts has its own log in /var/log/virtuamin/SomeHostNamelog.
The apache netadata module is filling up the /var/log/apache2/other_vhosts_access.log file with

SomeDomain.net:80 127.0.0.1 - - [13/Feb/2019:20:12:36 -0500] "GET /server-status?auto HTTP/1.1" 200 1459 "-" "-"

lines (one per second).

I read a previous question and the answer was suggesting disabling the apache module in /etc/netdata/python.d.conf file which I did. As soon as I do that the log file filling stops but also does the web_log charts.

So the question is, how to avoid these lines in logs while still having apache or web_log monitoring. Is there any way to tell Apache to ignore requests coming from netdata. Or the other way around, a way to tell netdata no to fill the log files ?

The problem is I encountered a intrusion attempt and was looking for details in the logs.
As the logs were filled with these lines, the time of the attempts was no more there (too far in time, log files were rotated and even deleted).

Thanks for your help

[ilyam8]

Hi @Vartkat

Apache and web_log aren't connected, web_log reads all new lines.

As soon as I do that the log file filling stops but also does the web_log charts.

what do you mean by stop? If there is no new log lines web_log reports zero values, so it is not stop, but empty charts.

If you want to exclude GET /server-status? from web_log (not count in web_log charts) use filter feature

netdata/collectors/python.d.plugin/web_log/web_log.conf

Lines 72 to 74 in 041725f

	# filter: # filter with regex
	# include: 'REGEX' # only those rows that matches the regex
	# exclude: 'REGEX' # all rows except those that matches the regex

filter:
  exclude: 'GET /server-status?'

And, ofc, you can decrease apache module update interval if you don't need per second monitoring.

[ilyam8]

Is there any way to tell Apache to ignore requests coming from netdata.

Yes. You need mod_setenvif.

The idea is to set internal env variable and then exclude all requests with that var. Example.

[Vartkat]

Thanks ilyam8 you point me in the right direction. I succeeded in avoiding server status to be logged by apache by making my /etc/apache2/conf-available/other-vhosts-access-log.conf

# Define an access log for VirtualHosts that don't define their own logfile
SetEnvIf Remote_Addr "127\.0\.0\.1" dontlog
CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined env=!dontlog

#CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

the log file stopped logging server-status every second, but the result is I got a functional apache chart but the web_log chart is flat. Weird part is I have no /etc/netdata/python.d/web_log.conf file, so I don't know if I have to build one from scratch or if there's something wrong with metadata. I updated it, no charge, no web_log.conf file.

[ilyam8]

@Vartkat

run python.d.plugin as netdata user in debug mode

sudo su -s /bin/bash netdata
./python.d.plugin web_log debug

[Vartkat]

$ sudo find / -name python.d.plugin

gives me

/usr/libexec/netdata/plugins.d/python.d.plugin
/usr/src/netdata.git/plugins.d/python.d.plugin
/usr/src/netdata.git/collectors/python.d.plugin
/usr/src/netdata.git/collectors/python.d.plugin/python.d.plugin

sudo su -s /bin/bash netdata /usr/libexec/netdata/plugins.d/python.d.plugin web_log debug
gives

2019-02-15 09:03:52: python.d INFO: plugin: main: Using python 2
2019-02-15 09:03:52: python.d DEBUG: plugin: main: loading '/etc/netdata/python.d.conf'
2019-02-15 09:03:52: python.d DEBUG: plugin: main: module load source: 'web_log' => [OK]
2019-02-15 09:03:52: python.d DEBUG: plugin: main: loading '/etc/netdata/python.d/web_log.conf'
2019-02-15 09:03:52: python.d ERROR: plugin: main: cannot load '/etc/netdata/python.d/web_log.conf' : [Errno 2] No such file or directory: '/etc/netdata/python.d/web_log.conf'. Will try stock version.
2019-02-15 09:03:52: python.d DEBUG: plugin: main: loading '/usr/lib/netdata/conf.d/python.d/web_log.conf'
2019-02-15 09:03:52: python.d DEBUG: plugin: main: job initialization: 'web_log nginx_log' => ['OK']
2019-02-15 09:03:52: python.d DEBUG: plugin: main: job initialization: 'web_log nginx_log2' => ['OK']
2019-02-15 09:03:52: python.d DEBUG: plugin: main: job initialization: 'web_log apache_log' => ['OK']
2019-02-15 09:03:52: python.d DEBUG: plugin: main: job initialization: 'web_log apache_log2' => ['OK']
2019-02-15 09:03:52: python.d DEBUG: plugin: main: job initialization: 'web_log apache_log3' => ['OK']
2019-02-15 09:03:52: python.d DEBUG: plugin: main: job initialization: 'web_log apache_vhosts_log' => ['OK']
2019-02-15 09:03:52: python.d DEBUG: plugin: main: job initialization: 'web_log gunicorn_log' => ['OK']
2019-02-15 09:03:52: python.d DEBUG: plugin: main: job initialization: 'web_log gunicorn_log2' => ['OK']
2019-02-15 09:03:52: python.d DEBUG: plugin: main: job initialization: 'web_log apache_cache' => ['OK']
2019-02-15 09:03:52: python.d DEBUG: plugin: main: job initialization: 'web_log apache2_cache' => ['OK']
2019-02-15 09:03:52: python.d DEBUG: plugin: main: job initialization: 'web_log httpd_cache' => ['OK']
2019-02-15 09:03:52: python.d DEBUG: plugin: main: job initialization: 'web_log squid_log1' => ['OK']
2019-02-15 09:03:52: python.d DEBUG: plugin: main: job initialization: 'web_log squid_log2' => ['OK']
2019-02-15 09:03:52: python.d DEBUG: plugin: main: module status: 'web_log' => [OK] (jobs: 13)
2019-02-15 09:03:52: python.d ERROR: web_log: nginx_log: /var/log/nginx/access.log not readable or not exist
2019-02-15 09:03:52: python.d INFO: web_log: nginx_log: check() => [FAILED]
2019-02-15 09:03:52: python.d ERROR: web_log: nginx_log2: /var/log/nginx/localhost.access_log not readable or not exist
2019-02-15 09:03:52: python.d INFO: web_log: nginx_log2: check() => [FAILED]
2019-02-15 09:03:52: python.d ERROR: web_log: apache_log: /var/log/apache2/access.log is empty
2019-02-15 09:03:52: python.d INFO: web_log: apache_log: check() => [FAILED]
2019-02-15 09:03:52: python.d ERROR: web_log: apache_log2: /var/log/apache2/access_log not readable or not exist
2019-02-15 09:03:52: python.d INFO: web_log: apache_log2: check() => [FAILED]
2019-02-15 09:03:52: python.d ERROR: web_log: apache_log3: /var/log/httpd/access_log not readable or not exist
2019-02-15 09:03:52: python.d INFO: web_log: apache_log3: check() => [FAILED]
2019-02-15 09:03:52: python.d INFO: web_log: apache_vhosts_log: Collected data: ['request', 'code', 'bytes_sent', 'address']
2019-02-15 09:03:52: python.d INFO: web_log: apache_vhosts_log: check() => [OK]
2019-02-15 09:03:52: python.d ERROR: web_log: gunicorn_log: /var/log/gunicorn/access.log not readable or not exist
2019-02-15 09:03:52: python.d INFO: web_log: gunicorn_log: check() => [FAILED]
2019-02-15 09:03:52: python.d ERROR: web_log: gunicorn_log2: /var/log/gunicorn/gunicorn-access.log not readable or not exist
2019-02-15 09:03:52: python.d INFO: web_log: gunicorn_log2: check() => [FAILED]
2019-02-15 09:03:52: python.d ERROR: web_log: apache_cache: /var/log/apache/cache.log not readable or not exist
2019-02-15 09:03:52: python.d INFO: web_log: apache_cache: check() => [FAILED]
2019-02-15 09:03:52: python.d ERROR: web_log: apache2_cache: /var/log/apache2/cache.log not readable or not exist
2019-02-15 09:03:52: python.d INFO: web_log: apache2_cache: check() => [FAILED]
2019-02-15 09:03:52: python.d ERROR: web_log: httpd_cache: /var/log/httpd/cache.log not readable or not exist
2019-02-15 09:03:52: python.d INFO: web_log: httpd_cache: check() => [FAILED]
2019-02-15 09:03:52: python.d ERROR: web_log: squid_log1: /var/log/squid3/access.log not readable or not exist
2019-02-15 09:03:52: python.d INFO: web_log: squid_log1: check() => [FAILED]
2019-02-15 09:03:52: python.d ERROR: web_log: squid_log2: /var/log/squid/access.log not readable or not exist
2019-02-15 09:03:52: python.d INFO: web_log: squid_log2: check() => [FAILED]
CHART netdata.runtime_web_log_apache_vhosts '' 'Execution time for web_log_apache_vhosts' 'ms' 'python.d' netdata.pythond_runtime line 145000 1
DIMENSION run_time 'run time' absolute 1 1

2019-02-15 09:03:52: python.d DEBUG: web_log: apache_vhosts_log: create() => [OK] (charts: 11)
2019-02-15 09:03:52: python.d DEBUG: web_log: apache_vhosts_log: started, update frequency: 1
CHART web_log_apache_vhosts.requests_per_ipproto '' 'Requests Per IP Protocol' 'requests/s' 'ip protocols' 'web_log.requests_per_ipproto' stacked 60007 1 '' 'python.d.plugin' 'web_log'
DIMENSION 'req_ipv4' 'ipv4' incremental 1 1 ''
DIMENSION 'req_ipv6' 'ipv6' incremental 1 1 ''

BEGIN web_log_apache_vhosts.requests_per_ipproto 0
SET 'req_ipv4' = 0
SET 'req_ipv6' = 0
END

CHART web_log_apache_vhosts.clients_all '' 'All Time Unique Client IPs' 'unique ips' 'clients' 'web_log.clients_all' stacked 60009 1 '' 'python.d.plugin' 'web_log'
DIMENSION 'unique_tot_ipv4' 'ipv4' absolute 1 1 ''
DIMENSION 'unique_tot_ipv6' 'ipv6' absolute 1 1 ''

BEGIN web_log_apache_vhosts.clients_all 0
SET 'unique_tot_ipv4' = 0
SET 'unique_tot_ipv6' = 0
END

fifth line
2019-02-15 09:03:52: python.d ERROR: plugin: main: cannot load '/etc/netdata/python.d/web_log.conf' : [Errno 2] No such file or directory: '/etc/netdata/python.d/web_log.conf'. Will try stock version.

confirms there's no web_log file.
Do I solve this by copy/paste the web_log file I can find on GitHub ?

[ilyam8]

@Vartkat

use edit-config script

[lgz :netdata]pwd
/opt/netdata/etc/netdata
[lgz :netdata]./edit-config

[Vartkat]

I did use edit-config (which for me is in /etc/netdata).
$ sudo su -s /bin/bash netdata ./edit-config python.d/web_log.conf
It told me that I had access rights issues.

Perhaps that's because I installed as root (I don't remember)
I corrected the issues and now web_log file is here and running.
Thank your very much for your help.

[ilyam8]

I corrected the issues

Nice! I am closing the issue, if you still have some questions feel free to ask

[candlerb]

Nice solution, in my case I went for a slight variation:

SetEnvIf Request_URI "^/server-status$" dontlog
CustomLog ${APACHE_LOG_DIR}/access.log combined env=!dontlog

[netdata-community-bot]

This issue has been mentioned on the Netdata Community Forums. There might be relevant details there:

https://community.netdata.cloud/t/apache-last-collected-secs-errors-constantly/910/25