[Metrics&Observability] Surface DB connections in /api/v2/metrics

[kdelee]

ISSUE TYPE

• Feature Idea

SUMMARY

Surface the number of database connections awx is making to roll more of the data administrators need to track into /api/v2/metrics

Users can track this now by creating some kind of exporter that directly queries PG, but this is in general hacky and not portable. We know if we exceed the max connections that things start to go bad, so it is a best practice to track this number.

Maybe we can follow pattern of

awx/awx/main/analytics/collectors.py

Lines 255 to 260 in 4dc956c

	def job_counts(since, **kwargs):
	counts = {}
	counts['total_jobs'] = models.UnifiedJob.objects.exclude(launch_type='sync').count()
	counts['status'] = dict(models.UnifiedJob.objects.exclude(launch_type='sync').values_list('status').annotate(Count('status')).order_by())
	counts['launch_type'] = dict(models.UnifiedJob.objects.exclude(launch_type='sync').values_list('launch_type').annotate(Count('launch_type')).order_by())
	return counts

And create a database_stats function that returns a dictionary with a key for current connections by awx user. I say a dictionary so we can later expand that if desired, like adding number of idle vs active connections, or the max connections. These could then get surfaced like we do for guages like ORG_COUNT.

awx/awx/main/analytics/metrics.py

Line 18 in 4dc956c

ORG_COUNT = Gauge('awx_organizations_total', 'Number of organizations', registry=REGISTRY)

[kdelee]

@jainnikhil30 expressed interest in doing this

[kdelee]

@Zokormazo in needs test now :D

[obaranov]

Validated from QE side. Added automated tests.