NRPE: Unable to read output

[Eloar]

I've got 2 machines, one with PostgreSQL 10 running (DB), and one with other services including Nagios Core (lets call it just Nagios). I've installed NRPE on DB machine alongside some plugins. Commands configured in nrpe.cnf are like so:

# Misc

command[check_users]=/usr/lib64/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib64/nagios/plugins/check_load -r -w 1.75,1.50,1.00 -c 2.00,1.75,1.50
command[check_hdd]=/usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p /dev/vda
command[check_zombie_procs]=/usr/lib64/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib64/nagios/plugins/check_procs -w 150 -c 200

# PostgreSQL

command[check_postgres_locks]=/usr/lib64/nagios/plugins/check_postgres_locks -w 2 -c 3
command[check_postgres_bloat]=/usr/lib64/nagios/plugins/check_postgres_bloat -w='100 M' -c='200 M'
command[check_postgres_connection]=/usr/lib64/nagios/plugins/check_postgres_connection --db=postgres
command[check_postgres_backends]=/usr/lib64/nagios/plugins/check_postgres_backends -w=70 -c=100

From Nagios Machine I'm able to run any check from misc, but none from PostgreSQL section. Those fails with error:

NRPE: Unable to read output

I've installed nagios-plugin-nrpe on DB machine to eliminate network problem, and even on localhost I am unable to get valid response. Example:

$ /usr/lib64/nagios/plugins/check_nrpe -H localhost -c check_postgres_connection
NRPE: Unable to read output

locally it's fine:

$ /usr/lib64/nagios/plugins/check_postgres_connection --db=postgres
POSTGRES_CONNECTION OK: DB "postgres" version 10.3 | time=0.01s

and other check works just fine over NRPE:

$ /usr/lib64/nagios/plugins/check_nrpe -H 127.0.0.1 -c check_load
OK - load average per CPU: 0.01, 0.07, 0.07|load1=0.005;1.750;2.000;0; load5=0.075;1.500;1.750;0; load15=0.065;1.000;1.500;0;

I had trouble with connection timeout when 5666 port was not added to iptables as allowed, but that is not the case this time.

NRPE is configured to run as user nrpe and group nagios, where permissions on all files (symlinks included) in /usr/lib64/nagios/plugins are set to 755 and ownership root:nagios. I've installed check_postgres version 2.23.0.

Any ideas how to approach this problem? I've stumbled accross this plugin because of this tutorial.

[ubellavance]

Have you created the nrpe user in your Postgresql cluster? Also, did you configure pg_hba.conf?

[Eloar]

nrpe and nagios users and groups are created when nrpe is installed using yum. pg_hba.conf was modified. Without pg_hba.conf configuration check would not work locally. Check works fine, but only local on DB machine. It doesn't work when invoked over NRPE. Other checks works fine, either local and over network from Nagios Machine.

[ubellavance]

You are talking about the OS user and group. I'm talking about the postgresql user. Can you show me your pg_hba.conf file?

[ubellavance]

I am pretty confident I can help you, I did that before going on vacation.

[Eloar]

It should not matter. It gave me a clue, so I added to each command user parameter: -u postgres. Unfortunatelly result is the same

NRPE: Unable to read output

the most important part in pg_hba.conf is:

local    all    all    trust

[ubellavance]

I'm saying that from the top of my head, as I'm still on vacation, but I think you can try -h localhost.

[Eloar]

-h is option for help -H is for host, but no host is resolved to UNIX Socket and is equivalent to psql without host. I don't want to open access to postgres user for anything other than Unix socket

[ubellavance]

Makes sense, sorry. I'll try to VPN into the office in the next few days to give you details about my setup. Have you tried running the command directly from the shell as nrpe user? You may have to temporarily enable shell login by changing the shell for the nrpe user in /etc/passwd

[Eloar]

yes, just tried it again and it works. I mean direct execution of check_postgres action as nrpe user does work

# sudo -u nrpe -- /usr/lib64/nagios/plugins/check_postgres_connection --db=postgres
POSTGRES_CONNECTION OK: DB "postgres" version 10.3 | time=0.01s

[ubellavance]

Here is how I configured my nrpe actions:

command[check_postgres_action]=/usr/bin/check_postgres.pl -u nrpe --action=$ARG1$
command[check_postgres_action_db]=/usr/bin/check_postgres.pl -u nrpe --action=$ARG1$ --db=$ARG2$
command[check_postgres_action_warning_critical]=/usr/bin/check_postgres.pl -u nrpe --action=$ARG1$ -w=$ARG2$ -c=$ARG3$ $ARG4$ $ARG5$

And i call them this way from Nagios, for example:

/usr/lib/nagios/plugins/check_nrpe -H atqatld1 -c check_postgres_action -a backends

I have an nrpe user created on the postgres cluster and here is the relevant line of my pg_hba.conf file:

local   all     nrpe            peer

It is somewhat the equivalent of your trust but in your case I think that you allow any OS user to access any database in the PostgreSQL cluster, which is a bit risky.

Did you check for SELinux AVCs? What OS are you running? I had to create an SELinux module to make it work on RHEL 8.

[Eloar]

My pg_hba.conf line allows any local user to access any db user. It is bit risky but not that much. I'm afraid user nrpe on Postgres will need to have its access elevated.

I'm using CentOS 7, with SELinux enabled, but it is configured ok, as other checks work fine both local and over nrpe.

EDIT:
I've modified my environment configuration to similar to Yours.

1. added role nrpe
   CREATE ROLE nrpe WITH LOGIN;
2. modified pg_hba.conf
local all all peer
3. removed user option from nrpe.cfg

command[check_postgres_locks]=/usr/lib64/nagios/plugins/check_postgres_locks -w 2 -c 3
command[check_postgres_bloat]=/usr/lib64/nagios/plugins/check_postgres_bloat -w='100 M' -c='200 M'
command[check_postgres_connection]=/usr/lib64/nagios/plugins/check_postgres_connection --db=postgres
command[check_postgres_backends]=/usr/lib64/nagios/plugins/check_postgres_backends

4. tried locally

# sudo -u nrpe -- /usr/lib64/nagios/plugins/check_postgres_connection -u nrpe --db=postgres
POSTGRES_CONNECTION OK: DB "postgres" version 10.3 | time=0.04s

5. tried over nrpe (locally)

# /usr/lib64/nagios/plugins/check_nrpe -H 127.0.0.1 -c check_postgres_connection
NRPE: Unable to read output

Unfortunatelly problem is not resolved

[ubellavance]

I think you need the -u option. And I doubt SELinux's default configuration works for those checks. Do you log connections in your postgresql logs? Do you see connections attempts in the logs when executing via nrpe?

[Eloar]

I thought SELinux was a good clue, so I've checked files contexts in /usr/lib64/nagios/plugins. Plugins installed with yum had context system_u:object_r:nagios_unconfined_plugin_exec_t:s0 where check_postgres.pl and it's symlinks had context unconfined_u:object_r:lib_t:s0. Unfortunatelly changing it to nagios_unconfined_plugin_exec_t did not resolve issue.