Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Introducing a dnsmasq PXE filter driver
A PXE filter driver is introduced that works by configuring and controlling the dnsmasq service. Closes-Bug: 1693813 Related-Bug: 1665666 Change-Id: I63fe91ee4f9ac3021bcfd9a4a378af56af800fac
- Loading branch information
Showing
9 changed files
with
588 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,123 @@ | ||
.. _dnsmasq_pxe_filter: | ||
|
||
**dnsmasq** PXE filter | ||
====================== | ||
|
||
Often an inspection PXE DHCP stack is implemented by the **dnsmasq** service. | ||
This PXE filter implementation relies on directly configuring the **dnsmasq** | ||
DHCP service to provide a caching PXE-traffic filter of node MAC addresses. | ||
|
||
How it works | ||
------------ | ||
|
||
Using a configuration *file per MAC address* allows one to implement a | ||
filtering mechanism based on the ``ignore`` directive:: | ||
|
||
$ cat /etc/dnsmasq.d/de-ad-be-ef-de-ad | ||
de:ad:be:ef:de:ad,ignore | ||
$ | ||
|
||
The filename is used to keep track of all MAC addresses in the cache, avoiding | ||
file parsing. The content of the file determines the MAC address access policy. | ||
|
||
Thanks to the ``inotify`` facility, **dnsmasq** is notified instantly once a | ||
new file is *created* or an existing file is *modified* in the | ||
DHCP hosts directory. Thus, to white-list a MAC address, one has to | ||
remove the ``ignore`` directive:: | ||
|
||
$ cat /etc/dnsmasq.d/de-ad-be-ef-de-ad | ||
de:ad:be:ef:de:ad | ||
$ | ||
|
||
The hosts directory content establishes a *cached* MAC addresses filter that is | ||
kept synchronized with the **ironic** port list. | ||
|
||
.. note:: | ||
|
||
The **dnsmasq** inotify facility implementation doesn't react on a file being | ||
removed or truncated. | ||
|
||
Configuration | ||
------------- | ||
|
||
To enable the **dnsmasq** PXE filter, update the PXE filter driver name:: | ||
|
||
[pxe_filter] | ||
driver = dnsmasq | ||
|
||
The DHCP hosts directory can be specified to override the default | ||
``/var/lib/ironic-inspector/dhcp-hostsdir``:: | ||
|
||
[dnsmasq_pxe_filter] | ||
dhcp_hostsdir = /etc/ironic-inspector/dhcp-hostsdir | ||
|
||
The filter design relies on the hosts directory being in exclusive | ||
**inspector** control. The hosts directory should be considered a *private | ||
cache* directory of **inspector** that **dnsmasq** polls configuration updates | ||
from, through the ``inotify`` facility. The directory has to be writable by | ||
**inspector** and readable by **dnsmasq**. | ||
|
||
One can also override the default start and stop commands to control the | ||
**dnsmasq** service:: | ||
|
||
[dnsmasq_pxe_filter] | ||
dnsmasq_start_command = dnsmasq --conf-file /etc/ironic-inspector/dnsmasq.conf | ||
dnsmasq_stop_command = kill $(cat /var/run/dnsmasq.pid) | ||
|
||
.. note:: | ||
|
||
It is also possible to set an empty/none string or to use shell expansion in | ||
place of the commands. An empty start command means the **dnsmasq** service | ||
won't be started upon the filter initialization, an empty stop command means | ||
the service won't be stopped upon an (error) exit. | ||
|
||
|
||
.. note:: | ||
|
||
These commands are executed through the ``rootwrap`` facility, so overriding | ||
may require a filter file to be created in the ``rootwrap.d`` directory. A | ||
sample configuration for **devstack** use might be: | ||
|
||
.. code-block:: console | ||
sudo cat > "$IRONIC_INSPECTOR_CONF_DIR/rootwrap.d/ironic-inspector-dnsmasq-systemctl.filters" <<EOF | ||
[Filters] | ||
# ironic_inspector/pxe_filter/dnsmasq.py | ||
systemctl: CommandFilter, systemctl, root, restart, devstack@ironic-inspector-dnsmasq | ||
systemctl: CommandFilter, systemctl, root, stop, devstack@ironic-inspector-dnsmasq | ||
EOF | ||
Supported dnsmasq versions | ||
-------------------------- | ||
|
||
This filter driver has been checked by **inspector** CI with **dnsmasq** | ||
versions `>=2.76`. The ``inotify`` facility was introduced_ to **dnsmasq** in | ||
the version `2.73`. | ||
|
||
.. _introduced: http://www.thekelleys.org.uk/dnsmasq/CHANGELOG | ||
|
||
Caveats | ||
------- | ||
|
||
The initial synchronization will put some load on the **dnsmasq** service | ||
starting based on the amount of ports **ironic** keeps. This can take up to a | ||
minute of full CPU load for huge amounts of MACs (tens of thousands). | ||
Subsequent filter synchronizations will only cause the **dnsmasq** to parse | ||
the modified files. Typically those are the bare metal nodes being added or | ||
phased out from the compute service, meaning dozens of file updates per sync | ||
call. | ||
|
||
The **inspector** takes over the control of the DHCP hosts directory to | ||
implement its filter cache. Files are generated dynamically so should not be | ||
edited by hand. To minimize the interference between the deployment and | ||
introspection, **inspector** has to start the **dnsmasq** service only after | ||
the initial synchronization. Conversely, the **dnsmasq** service is stopped | ||
upon (unexpected) **inspector** exit. | ||
|
||
To avoid accumulating stale DHCP host files over time, the driver cleans up | ||
the DHCP hosts directory during the ``init_filter`` call. | ||
|
||
Although the filter driver tries its best to always stop the **dnsmasq** | ||
service, it is recommended that the operator configures the **dnsmasq** | ||
service in such a way that it terminates upon **inspector** (unexpected) exit | ||
to prevent a stale blacklist from being used by the **dnsmasq** service. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,176 @@ | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or | ||
# implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
# NOTE(milan) the filter design relies on the hostdir[1] being in exclusive | ||
# inspector control. The hostdir should be considered a private cache directory | ||
# of inspector that dnsmasq has read access to and polls updates from, through | ||
# the inotify facility. | ||
# | ||
# [1] see the --dhcp-hostsdir option description in | ||
# http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html | ||
|
||
|
||
import os | ||
|
||
from oslo_concurrency import processutils | ||
from oslo_config import cfg | ||
from oslo_log import log | ||
from oslo_utils import timeutils | ||
|
||
from ironic_inspector.common import ironic as ir_utils | ||
from ironic_inspector import node_cache | ||
from ironic_inspector.pxe_filter import base as pxe_filter | ||
|
||
CONF = cfg.CONF | ||
LOG = log.getLogger(__name__) | ||
|
||
_ROOTWRAP_COMMAND = 'sudo ironic-inspector-rootwrap {rootwrap_config!s}' | ||
_MACBL_LEN = len('ff:ff:ff:ff:ff:ff,ignore\n') | ||
|
||
|
||
class DnsmasqFilter(pxe_filter.BaseFilter): | ||
"""The dnsmasq PXE filter driver. | ||
A pxe filter driver implementation that controls access to dnsmasq | ||
through amending its configuration. | ||
""" | ||
|
||
def reset(self): | ||
"""Stop dnsmasq and upcall reset.""" | ||
_execute(CONF.dnsmasq_pxe_filter.dnsmasq_stop_command, | ||
ignore_errors=True) | ||
super(DnsmasqFilter, self).reset() | ||
|
||
def _sync(self, ironic): | ||
"""Sync the inspector, ironic and dnsmasq state. Locked. | ||
:raises: IOError, OSError. | ||
:returns: None. | ||
""" | ||
LOG.debug('Syncing the driver') | ||
timestamp_start = timeutils.utcnow() | ||
active_macs = node_cache.active_macs() | ||
ironic_macs = set(port.address for port in | ||
ironic.port.list(limit=0, fields=['address'])) | ||
blacklist_macs = _get_blacklist() | ||
# NOTE(milan) whitelist MACs of ports not kept in ironic anymore | ||
# also whitelist active MACs that are still blacklisted in the | ||
# dnsmasq configuration but have just been asked to be introspected | ||
for mac in ((blacklist_macs - ironic_macs) | | ||
(blacklist_macs & active_macs)): | ||
_whitelist_mac(mac) | ||
# blacklist new ports that aren't being inspected | ||
for mac in ironic_macs - (blacklist_macs | active_macs): | ||
_blacklist_mac(mac) | ||
timestamp_end = timeutils.utcnow() | ||
LOG.debug('The dnsmasq PXE filter was synchronized (took %s)', | ||
timestamp_end - timestamp_start) | ||
|
||
@pxe_filter.locked_driver_event(pxe_filter.Events.sync) | ||
def sync(self, ironic): | ||
"""Sync dnsmasq configuration with current Ironic&Inspector state. | ||
Polls all ironic ports. Those being inspected, the active ones, are | ||
whitelisted while the rest are blacklisted in the dnsmasq | ||
configuration. | ||
:param ironic: an ironic client instance. | ||
:raises: OSError, IOError. | ||
:returns: None. | ||
""" | ||
self._sync(ironic) | ||
|
||
@pxe_filter.locked_driver_event(pxe_filter.Events.initialize) | ||
def init_filter(self): | ||
"""Performs an initial sync with ironic and starts dnsmasq. | ||
The initial _sync() call reduces the chances dnsmasq might lose | ||
some inotify blacklist events by prefetching the blacklist before | ||
the dnsmasq is started. | ||
:raises: OSError, IOError. | ||
:returns: None. | ||
""" | ||
_purge_dhcp_hostsdir() | ||
ironic = ir_utils.get_client() | ||
self._sync(ironic) | ||
_execute(CONF.dnsmasq_pxe_filter.dnsmasq_start_command) | ||
LOG.info('The dnsmasq PXE filter was initialized') | ||
|
||
|
||
def _purge_dhcp_hostsdir(): | ||
"""Remove all the DHCP hosts files. | ||
:raises: FileNotFoundError in case the dhcp_hostsdir is invalid. | ||
IOError in case of non-writable file or a record not being a file. | ||
:returns: None. | ||
""" | ||
dhcp_hostsdir = CONF.dnsmasq_pxe_filter.dhcp_hostsdir | ||
LOG.debug('Purging %s', dhcp_hostsdir) | ||
for mac in os.listdir(dhcp_hostsdir): | ||
path = os.path.join(dhcp_hostsdir, mac) | ||
# NOTE(milan) relying on a failure here aborting the init_filter() call | ||
os.remove(path) | ||
LOG.debug('Removed %s', path) | ||
|
||
|
||
def _get_blacklist(): | ||
"""Get addresses currently blacklisted in dnsmasq. | ||
:raises: FileNotFoundError in case the dhcp_hostsdir is invalid. | ||
:returns: a set of MACs currently blacklisted in dnsmasq. | ||
""" | ||
hostsdir = CONF.dnsmasq_pxe_filter.dhcp_hostsdir | ||
# whitelisted MACs lack the ,ignore directive | ||
return set(address for address in os.listdir(hostsdir) | ||
if os.stat(os.path.join(hostsdir, address)).st_size == | ||
_MACBL_LEN) | ||
|
||
|
||
def _blacklist_mac(mac): | ||
"""Creates a dhcp_hostsdir ignore record for the MAC. | ||
:raises: FileNotFoundError in case the dhcp_hostsdir is invalid, | ||
IOError in case the dhcp host MAC file isn't writable. | ||
:returns: None. | ||
""" | ||
path = os.path.join(CONF.dnsmasq_pxe_filter.dhcp_hostsdir, mac) | ||
# NOTE(milan) line-buffering enforced to ensure dnsmasq record update | ||
# through inotify, which reacts on f.close() | ||
with open(path, 'w', 1) as f: | ||
f.write('%s,ignore\n' % mac) | ||
LOG.debug('Blacklisted %s', mac) | ||
|
||
|
||
def _whitelist_mac(mac): | ||
"""Un-ignores the dhcp_hostsdir record for the MAC. | ||
:raises: FileNotFoundError in case the dhcp_hostsdir is invalid, | ||
IOError in case the dhcp host MAC file isn't writable. | ||
:returns: None. | ||
""" | ||
path = os.path.join(CONF.dnsmasq_pxe_filter.dhcp_hostsdir, mac) | ||
with open(path, 'w', 1) as f: | ||
# remove the ,ignore directive | ||
f.write('%s\n' % mac) | ||
LOG.debug('Whitelisted %s', mac) | ||
|
||
|
||
def _execute(cmd=None, ignore_errors=False): | ||
# e.g: '/bin/kill $(cat /var/run/dnsmasq.pid)' | ||
if not cmd: | ||
return | ||
|
||
helper = _ROOTWRAP_COMMAND.format(rootwrap_config=CONF.rootwrap_config) | ||
processutils.execute(cmd, run_as_root=True, root_helper=helper, shell=True, | ||
check_exit_code=not ignore_errors) |
Oops, something went wrong.