Skip to content
Permalink
Browse files

Create "central" agent

This changeset is phase one of changing the pollster API to support
polling individual instance resources. In this change, the single
compute agent is divided into two separate daemons.

The compute agent, for polling instance data, is still meant to
run on the compute server. The new "central" agent, for polling
resources not tied to a compute node, is meant to run on a management
server (probably the same place the collector runs). The configuration
of the pollsters is updated so that they are loaded by the
appropriate agent.

New base classes are introduced for each of the types of pollsters.
For now, the APIs remain the same.

The code implementing the agent and plugins has been moved around
to reflect the new logical relationships, and the documentation
is updated (including new installation instructions).

Change-Id: Ica6e947b2e457f7db6672147af1369a24066037d
Signed-off-by: Doug Hellmann <doug.hellmann@dreamhost.com>
  • Loading branch information...
Doug Hellmann
Doug Hellmann committed Jul 30, 2012
1 parent 2eebd4a commit 0e8f2359d9d1b5fc02fbc0fe92e788eb757feaa5
@@ -0,0 +1,39 @@
#!/usr/bin/env python
# -*- encoding: utf-8 -*-
#
# Copyright © 2012 eNovance <licensing@enovance.com>
#
# Author: Julien Danjou <julien@danjou.info>
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.

import eventlet
eventlet.monkey_patch()
import sys

from ceilometer.service import prepare_service
from ceilometer.openstack.common import cfg
from nova import service


if __name__ == '__main__':

prepare_service(sys.argv)
server = \
service.Service.create(
binary='ceilometer-agent-central',
topic='ceilometer.agent.central',
manager='ceilometer.central.manager.AgentManager',
periodic_interval=cfg.CONF.periodic_interval)
service.serve(server)
service.wait()
@@ -36,9 +36,10 @@ if __name__ == '__main__':

prepare_service(sys.argv)
server = \
service.Service.create(binary='ceilometer-agent',
topic='ceilometer.agent',
manager='ceilometer.agent.manager.AgentManager',
periodic_interval=cfg.CONF.periodic_interval)
service.Service.create(
binary='ceilometer-agent-compute',
topic='ceilometer.agent.compute',
manager='ceilometer.compute.manager.AgentManager',
periodic_interval=cfg.CONF.periodic_interval)
service.serve(server)
service.wait()
File renamed without changes.
@@ -0,0 +1,71 @@
# -*- encoding: utf-8 -*-
#
# Copyright © 2012 eNovance <licensing@enovance.com>
#
# Author: Julien Danjou <julien@danjou.info>
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.

import pkg_resources

from nova import manager

from ceilometer.openstack.common import log
from ceilometer import publish


LOG = log.getLogger(__name__)

PLUGIN_NAMESPACE = 'ceilometer.poll.central'


class AgentManager(manager.Manager):

def init_host(self):
self._load_plugins()
return

def _load_plugins(self):
self.pollsters = []
for ep in pkg_resources.iter_entry_points(PLUGIN_NAMESPACE):
try:
plugin_class = ep.load()
plugin = plugin_class()
# FIXME(dhellmann): Currently assumes all plugins are
# enabled when they are discovered and
# importable. Need to add check against global
# configuration flag and check that asks the plugin if
# it should be enabled.
self.pollsters.append((ep.name, plugin))
LOG.info('loaded pollster %s:%s',
PLUGIN_NAMESPACE, ep.name)
except Exception as err:
LOG.warning('Failed to load pollster %s:%s',
ep.name, err)
LOG.exception(err)
if not self.pollsters:
LOG.warning('Failed to load any pollsters for %s',
PLUGIN_NAMESPACE)
return

def periodic_tasks(self, context, raise_on_error=False):
"""Tasks to be run at a periodic interval."""
for name, pollster in self.pollsters:
try:
LOG.info('polling %s', name)
for c in pollster.get_counters(self, context):
LOG.info('COUNTER: %s', c)
publish.publish_counter(context, c)
except Exception as err:
LOG.warning('Continuing after error from %s: %s', name, err)
LOG.exception(err)
@@ -0,0 +1,25 @@
# -*- encoding: utf-8 -*-
#
# Copyright © 2012 New Dream Network, LLC (DreamHost)
#
# Author: Doug Hellmann <doug.hellmann@dreamhost.com>
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Base class for plugins used by the central agent.
"""

from ceilometer import plugin


class CentralPollster(plugin.PollsterBase):
"""Base class for plugins that support the polling API."""
@@ -21,7 +21,7 @@
from nova import flags

from ceilometer import counter
from ceilometer import plugin
from ceilometer.compute import plugin
from ceilometer.compute import instance as compute_instance
from ceilometer.openstack.common import importutils
from ceilometer.openstack.common import log
@@ -61,7 +61,7 @@ def make_counter_from_instance(instance, name, type, volume):
)


class DiskIOPollster(plugin.PollsterBase):
class DiskIOPollster(plugin.ComputePollster):

LOG = log.getLogger(__name__ + '.diskio')

@@ -111,7 +111,7 @@ def get_counters(self, manager, context):
)


class CPUPollster(plugin.PollsterBase):
class CPUPollster(plugin.ComputePollster):

LOG = log.getLogger(__name__ + '.cpu')

File renamed without changes.
@@ -0,0 +1,35 @@
# -*- encoding: utf-8 -*-
#
# Copyright © 2012 New Dream Network, LLC (DreamHost)
#
# Author: Doug Hellmann <doug.hellmann@dreamhost.com>
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Base class for plugins used by the compute agent.
"""

import abc

from ceilometer import plugin


class ComputePollster(plugin.PollsterBase):
"""Base class for plugins that support the polling API on the
compute node."""

__metaclass__ = abc.ABCMeta

@abc.abstractmethod
def get_counters(self, manager, context):
"""Return a sequence of Counter instances from polling the
resources."""
No changes.
@@ -21,16 +21,16 @@
from ceilometer.openstack.common import log

from ceilometer import counter
from ceilometer import plugin
from ceilometer.central import plugin


class FloatingIPPollster(plugin.PollsterBase):
class FloatingIPPollster(plugin.CentralPollster):

LOG = log.getLogger(__name__ + '.floatingip')

def get_counters(self, manager, context):
try:
ips = manager.db.floating_ip_get_all_by_host(context, manager.host)
ips = manager.db.floating_ip_get_all(context)
except exception.FloatingIpNotFoundForHost:
pass
else:
@@ -44,7 +44,7 @@ def get_counters(self, manager, context):
user_id=None,
project_id=ip.project_id,
resource_id=ip.id,
timestamp=None,
timestamp=None, # FIXME(dhellmann): This needs to be now()
duration=None,
resource_metadata={
'address': ip.address,
@@ -15,24 +15,29 @@ High Level Description
double: database; architecture
double: API; architecture

There are 4 basic components to the system:
There are 5 basic components to the system:

1. An :term:`agent` runs on each compute node and polls for resource
utilization statistics. There may be other types of agents in the
future, but for now we will focus on creating the compute agent.
1. An :term:`compute agent` runs on each compute node and polls for
resource utilization statistics. There may be other types of agents
in the future, but for now we will focus on creating the compute
agent.

2. The :term:`collector` runs on one or more central management
2. An :term:`central agent` runs on a central management server to
poll for resource utilization statistics for resources not tied
to instances or compute nodes.

3. The :term:`collector` runs on one or more central management
servers to monitor the message queues (for notifications and for
metering data coming from the agent). Notification messages are
processed and turned into metering messages and sent back out onto
the message bus using the appropriate topic. Metering messages are
written to the data store without modification.

3. The :term:`data store` is a database capable of handling concurrent
4. The :term:`data store` is a database capable of handling concurrent
writes (from one or more collector instances) and reads (from the
API server).

4. The :term:`API server` runs on one or more central management
5. The :term:`API server` runs on one or more central management
servers to provide access to the data from the data store. See
EfficientMetering#API for details.

@@ -116,20 +121,17 @@ Metering data comes from two sources: through notifications built into
the existing OpenStack components and by polling the infrastructure
(such as via libvirt). Polling for compute resources is handled by an
agent running on the compute node (where communication with the
hypervisor is more efficient).

.. note::

We only poll compute resources for now, but when other types of
polling are implemented the pollsters are likely to run somewhere
other than the compute node.

The agent daemon is configured to run one or more *pollster*
plugins using the ``ceilometer.poll.compute`` namespace. The agent
periodically asks each pollster for instances of ``Counter``
objects. The agent framework converts the Counters to metering
messages, which it then signs and transmits on the metering message
bus.
hypervisor is more efficient). The compute agent daemon is configured
to run one or more *pollster* plugins using the
``ceilometer.poll.compute`` namespace. Polling for resources not tied
to the compute node is handled by the central agent. The central
agent daemon is configured to run one or more *pollster* plugins using
the ``ceilometer.poll.central`` namespace.

The agents periodically asks each pollster for instances of
``Counter`` objects. The agent framework converts the Counters to
metering messages, which it then signs and transmits on the metering
message bus.

The pollster plugins do not communicate with the message bus directly,
unless it is necessary to do so in order to collect the information

0 comments on commit 0e8f235

Please sign in to comment.
You can’t perform that action at this time.