foreman: plugin execution times out when DB password can't be found

[evgeni]

The foreman plugin tries to read the PostgreSQL password from /etc/foreman/database.yml and falls back to an empty string if that can't be found for some reason:

sos/sos/plugins/foreman.py

Lines 37 to 59 in 00a25de

	self.dbpasswd = ""
	try:
	for line in open("/etc/foreman/database.yml").read().splitlines():
	# skip empty lines and lines with comments
	if not line or line[0] == '#':
	continue
	if line.startswith("production:"):
	production_scope = True
	continue
	if production_scope and match(r"\s+host:\s+\S+", line):
	self.dbhost = line.split()[1]
	if production_scope and match(r"\s+password:\s+\S+", line):
	self.dbpasswd = line.split()[1]
	# if line starts with a text, it is a different scope
	if not line.startswith(" "):
	production_scope = False
	except IOError:
	# fallback when the cfg file is not accessible
	pass
	# strip wrapping ".." or '..' around password
	if (self.dbpasswd.startswith('"') and self.dbpasswd.endswith('"')) or \
	(self.dbpasswd.startswith('\'') and self.dbpasswd.endswith('\'')):
	self.dbpasswd = self.dbpasswd[1:-1]

However, when one tries to execute psql without a password, it will just sit there and wait for the user to enter one:

# PGPASSWORD= psql -h localhost -p 5432 -U foreman -d foreman -c 'select 1;'
Password for user foreman: 

This leads to the plugin timing out:

[plugin:foreman] command 'psql -h localhost -p 5432 -U foreman -d foreman  -c 'SELECT table_name, pg_size_pretty(total_bytes) AS total, pg_size_pretty(index_bytes) AS INDEX , pg_size_pretty(toast_bytes) AS toast, pg_size_pretty(table_bytes) AS TABLE FROM ( SELECT *, total_bytes-index_bytes-COALESCE(toast_bytes,0) AS table_bytes FROM (SELECT c.oid,nspname AS table_schema, relname AS TABLE_NAME, c.reltuples AS row_estimate, pg_total_relation_size(c.oid) AS total_bytes, pg_indexes_size(c.oid) AS index_bytes, pg_total_relation_size(reltoastrelid) AS toast_bytes FROM pg_class c LEFT JOIN pg_namespace n ON n.oid = c.relnamespace WHERE relkind = '"'"'r'"'"') a) a order by total_bytes DESC'' timed out after 300s
[plugin:foreman] command 'psql -h localhost -p 5432 -U foreman -d foreman  -c 'WITH prefix_counts AS (SELECT split_part(name,'"'"'::'"'"',1) FROM fact_names) SELECT COUNT(*), split_part AS "fact_name_prefix" FROM prefix_counts GROUP BY split_part ORDER BY count DESC LIMIT 100'' timed out after 600s
[plugin:foreman] command 'psql -h localhost -p 5432 -U foreman -d foreman  -c 'select * from dynflow_schema_info'' timed out after 600s

 Plugin foreman timed out

The most trivial fix for this is to pass --no-password to psql, which will "Never issue a password prompt. If the server requires password authentication and a password is not available from other sources such as a .pgpass file, the connection attempt will fail. This option can be useful in batch jobs and scripts where no user is present to enter a password." and that's exactly the case we have here.

I am linking the legacy-3.9 code above, as I am experiencing this on CentOS 7 with sos-3.9-5.el7.centos.2.noarch, but the code in master looks (for that part) identical and should suffer from the same issue.

And if you're wondering why we're facing this now, and not since a long time: The plugin works equally on Foreman servers and Foreman proxies (see packages definition in the plugin). There is no /etc/foreman/database.yml on the proxy, so the password can't be read, which was not a problem until recently as we didn't have a psql binary on the proxy. We have one now, but still no /etc/foreman/database.yml which leads to the before mentioned password prompt.

[pmoravec]

Nice finding; adding to my todo list to prepare a PR (until @evgeni volunteer himself, whatever works).

About the "foreman plugin executed also on foreman-proxy systems with no foreman": we should split the plugin to foreman and foreman-proxy, indeed. Maybe we shall do it altogether (the no password fallback makes sense even with separated foreman-proxy plugin).