[Metricbeat] OpenStack Integration

[sorantis]

OpenStack is an open source orchestration platform for virtual machines. Think vSphere, but open source, or AWS EC2, S3 but on-prem. OpenStack consists of 6 core components that are responsible for managing compute, networking and storage resources. These components are:

• Nova - provisions virtual machines.
• Neutron - provisions virtual networks, routers.
• Cinder - provides block storage.
• Glance - provides image registry.
• Swift - provides object storage.
• Keystone - provides identity management (authentication, authorization, service management)

Horizon and Heat are out of scope of this integration and hence will not be considered in the issue.

Monitoring an OpenStack cluster implies monitoring its core services. While there are satellite components designed for this purpose, e.g. Ceilometer, currently the safest and optimal way of gathering information about cluster's health and performance is best done by talking to respective core services. Why? Not every OpenStack distribution bundles these satellite components, which makes it hard to rely on them.

The main entry point into any OpenStack cluster is the Keystone service that upon successful authentication provides access to other OpenStack services. In fact no interaction with the OpenStack core services happens without prior handshake with Keystone. The easiest way for connecting to Keystone would be a Golang SDK for managing OpenStack resources and services called gopherlang. This SDK should provide all the necessary functionality to collect OpenStack metrics (for concrete examples see the prometheus-exporter).

Once authenticated via Keystone any client can access the APIs of each core service and pull available information about service health, usage, etc.

User Story

As an OpenStack operator I want to be able to monitor all OpenStack services so that I could see my cluster's resource usage, health and performance in a simple way.

Metrics to collect

Below is a list of some of the key metrics that should cover basic OpenStack monitoring use cases.

Nova	Neutron	Cinder	Glance	Keystone	Swift
# Total VMs	# Networks	# Volumes	# Images	# Domains	# Objects in a container
# Running VMs	# Routers	# Snapshots		# Users	Container size
# Available vCPUs	# Subnets	# Total Pool Capacity (Gb)		# Groups
# Used vCPUs	# Ports	# Used Pool Capacity (Gb)		# Projects
# Available Memory	# Floating IPs	Limits: Max/Used Volume Gbs		# Regions
# Used Memory	# Total IPs	Volume Status
# Available local storage	# Used IPs	Cinder Agent State
# Used local storage	Neutron Agent State
Limits: Max/Used vCPUs, Memory
Server Status
Availability Zones
# Security Groups
Nova Agent State

Running OpenStack
Running OpenStack on a local machine is fairly easy with DevStack.

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[axrayn]

As an enterprise customer, we're definitely interested in the improvements being proposed in this PR.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ChrsMark]

cc: @ravikesarwani

[petericebear]

+1 for this functionality

[morganchenjp]

+1, really expecting this functionality.

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!