From 32f91aacdbf5ed33cc7184afafa612b71cc46025 Mon Sep 17 00:00:00 2001 From: timellis Date: Fri, 1 Aug 2014 18:01:10 +0000 Subject: [PATCH] remove outdated doc/ and point to wiki --- README.md | 4 +- doc/Makefile | 70 -------------- doc/cmdline.rst | 84 ---------------- doc/conf.py | 179 ---------------------------------- doc/getting_started.rst | 82 ---------------- doc/index.rst | 17 ---- doc/services.rst | 56 ----------- doc/tutorial.rst | 124 ------------------------ doc/using.rst | 207 ---------------------------------------- 9 files changed, 2 insertions(+), 821 deletions(-) delete mode 100644 doc/Makefile delete mode 100644 doc/cmdline.rst delete mode 100644 doc/conf.py delete mode 100644 doc/getting_started.rst delete mode 100644 doc/index.rst delete mode 100644 doc/services.rst delete mode 100644 doc/tutorial.rst delete mode 100644 doc/using.rst diff --git a/README.md b/README.md index 807e245..788bce2 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ Clusto is a cluster management tool. It helps you keep track of your inventory, where it is, how it's connected, and provides an abstracted interface for interacting with the elements of the infrastructure. -Some docs are available under docs/ and there are more code examples in the -unit tests in src/clusto/test/. +There are code examples in the unit tests in src/clusto/test/. +To learn more, start with the wiki, here: https://github.com/clusto/clusto/wiki diff --git a/doc/Makefile b/doc/Makefile deleted file mode 100644 index e996997..0000000 --- a/doc/Makefile +++ /dev/null @@ -1,70 +0,0 @@ -# Makefile for Sphinx documentation -# - -# You can set these variables from the command line. -SPHINXOPTS = -SPHINXBUILD = sphinx-build -PAPER = - -# Internal variables. -PAPEROPT_a4 = -D latex_paper_size=a4 -PAPEROPT_letter = -D latex_paper_size=letter -ALLSPHINXOPTS = -d .build/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . - -.PHONY: help clean html web pickle htmlhelp latex changes linkcheck - -help: - @echo "Please use \`make ' where is one of" - @echo " html to make standalone HTML files" - @echo " pickle to make pickle files (usable by e.g. sphinx-web)" - @echo " htmlhelp to make HTML files and a HTML help project" - @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" - @echo " changes to make an overview over all changed/added/deprecated items" - @echo " linkcheck to check all external links for integrity" - -clean: - -rm -rf .build/* - -html: - mkdir -p .build/html .build/doctrees - $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) .build/html - @echo - @echo "Build finished. The HTML pages are in .build/html." - -pickle: - mkdir -p .build/pickle .build/doctrees - $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) .build/pickle - @echo - @echo "Build finished; now you can process the pickle files or run" - @echo " sphinx-web .build/pickle" - @echo "to start the sphinx-web server." - -web: pickle - -htmlhelp: - mkdir -p .build/htmlhelp .build/doctrees - $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) .build/htmlhelp - @echo - @echo "Build finished; now you can run HTML Help Workshop with the" \ - ".hhp project file in .build/htmlhelp." - -latex: - mkdir -p .build/latex .build/doctrees - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) .build/latex - @echo - @echo "Build finished; the LaTeX files are in .build/latex." - @echo "Run \`make all-pdf' or \`make all-ps' in that directory to" \ - "run these through (pdf)latex." - -changes: - mkdir -p .build/changes .build/doctrees - $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) .build/changes - @echo - @echo "The overview file is in .build/changes." - -linkcheck: - mkdir -p .build/linkcheck .build/doctrees - $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) .build/linkcheck - @echo - @echo "Link check complete; look for any errors in the above output " \ - "or in .build/linkcheck/output.txt." diff --git a/doc/cmdline.rst b/doc/cmdline.rst deleted file mode 100644 index 4f81e98..0000000 --- a/doc/cmdline.rst +++ /dev/null @@ -1,84 +0,0 @@ -###################### - Command line usage -###################### - -:Release: |version| -:Date: |today| - -There are a number of interactive clusto commands you can use for inspecting and manipulating the database. A lot of them are broken because we haven't used them in a while. Feel free to fix them when you're done reading this doc. - -clusto info -~~~~~~~~~~~ -Gives a quick overview of the important attributes of an object identified by name, IP, or MAC address:: - - $ clusto info s0233 - Name: s0233 - Type: server - IP: 10.2.129.191 - Parents: sjc1-017, production, rofl-gamma, wtf-gamma, zomg-gamma-nokill - - Serial: P1238090236 - Memory: 16 GB - Disk: 1500 GB (3) - CPU Cores: 8 - - nic-eth(1) mac = 00:a0:d1:e8:57:70 - nic-eth(2) mac = 00:a0:d1:e8:57:71 - -clusto attr -~~~~~~~~~~~ -Manipulates the attributes of an object:: - - $ clusto attr add -k puppet -s class -v site::ldap-client s0245 - $ clusto attr show -m -k puppet -s class s0245 - KEY SUBKEY NUM VALUE - puppet class None site::ldap-client - -clusto pool -~~~~~~~~~~~ -Manipulates the contents of pool objects:: - - $ clusto pool create testing - $ clusto pool insert s0245 - $ clusto pool show testing - s0245 - $ clusto pool remove testing s0245 - $ clusto pool delete testing - -clusto shell -~~~~~~~~~~~~ -Probably the most powerful command for working with clusto, this command spawns an ipython shell with the clusto database connection initialized and a few clusto "builtins" imported into the local scope. This is a great way to test out new ideas, run complicated one-shot queries against clusto, or just to get more comfortable with the clusto interface. If you want to create objects, you'll have to do:: - - from clusto.drivers import * - -clusto tree -~~~~~~~~~~~ - -Outputs a tree-like listing of objects contained within other objects:: - - $ clusto tree sjc1-003 - name: sjc1-003 - name: sjc1-003-pwr1 - name: sjc1-003-sw1 - name: sjc1-003-ts1 - name: s0001 - name: s0002 - ... - $ clusto tree sjc1-003 name system - name: sjc1-003 system: None - name: sjc1-003-pwr1 system: None - name: sjc1-003-sw1 system: None - name: sjc1-003-ts1 system: None - name: s0001 system: [u'P1238110062\n', 8] - name: s0002 system: [u'P1238110064\n', 8] - name: s0004 system: [u'P1238110066\n', 8] - -Useful aliases -~~~~~~~~~~~~~~ - -I've found the following shorthand aliases for clusto commands to be extremely handy:: - - $ alias cinfo="clusto info" - $ alias clp="clusto list-pool" - - diff --git a/doc/conf.py b/doc/conf.py deleted file mode 100644 index 931b708..0000000 --- a/doc/conf.py +++ /dev/null @@ -1,179 +0,0 @@ -# -*- coding: utf-8 -*- -# -# clusto documentation build configuration file, created by -# sphinx-quickstart on Wed Feb 24 23:53:25 2010. -# -# This file is execfile()d with the current directory set to its containing dir. -# -# The contents of this file are pickled, so don't put values in the namespace -# that aren't pickleable (module imports are okay, they're removed automatically). -# -# All configuration values have a default value; values that are commented out -# serve to show the default value. - -import sys, os - -# If your extensions are in another directory, add it here. If the directory -# is relative to the documentation root, use os.path.abspath to make it -# absolute, like shown here. -#sys.path.append(os.path.abspath('../')) - -# General configuration -# --------------------- - -# Add any Sphinx extension module names here, as strings. They can be extensions -# coming with Sphinx (named 'sphinx.ext.*') or your custom ones. -extensions = ['sphinx.ext.autodoc'] - -# Add any paths that contain templates here, relative to this directory. -templates_path = ['.templates'] - -# The suffix of source filenames. -source_suffix = '.rst' - -# The master toctree document. -master_doc = 'index' - -# General substitutions. -project = 'clusto' -copyright = '2011, Clusto Org' - -# The default replacements for |version| and |release|, also used in various -# other places throughout the built documents. -# -# The short X.Y version. -#version = '0.5' -# The full version, including alpha/beta/rc tags. -#release = '0.5.27' - -# There are two options for replacing |today|: either, you set today to some -# non-false value, then it is used: -#today = '' -# Else, today_fmt is used as the format for a strftime call. -today_fmt = '%B %d, %Y' - -# List of documents that shouldn't be included in the build. -#unused_docs = [] - -# List of directories, relative to source directories, that shouldn't be searched -# for source files. -#exclude_dirs = [] - -# The reST default role (used for this markup: `text`) to use for all documents. -#default_role = None - -# If true, '()' will be appended to :func: etc. cross-reference text. -#add_function_parentheses = True - -# If true, the current module name will be prepended to all description -# unit titles (such as .. function::). -#add_module_names = True - -# If true, sectionauthor and moduleauthor directives will be shown in the -# output. They are ignored by default. -#show_authors = False - -# The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'sphinx' - - -# Options for HTML output -# ----------------------- - -# The style sheet to use for HTML and HTML Help pages. A file of that name -# must exist either in Sphinx' static/ path, or in one of the custom paths -# given in html_static_path. -html_style = 'default.css' - -# The name for this set of Sphinx documents. If None, it defaults to -# " v documentation". -#html_title = None - -# A shorter title for the navigation bar. Default is the same as html_title. -#html_short_title = None - -# The name of an image file (within the static path) to place at the top of -# the sidebar. -#html_logo = None - -# The name of an image file (within the static path) to use as favicon of the -# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 -# pixels large. -#html_favicon = None - -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['.static'] - -# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, -# using the given strftime format. -html_last_updated_fmt = '%b %d, %Y' - -# If true, SmartyPants will be used to convert quotes and dashes to -# typographically correct entities. -#html_use_smartypants = True - -# Custom sidebar templates, maps document names to template names. -#html_sidebars = {} - -# Additional templates that should be rendered to pages, maps page names to -# template names. -#html_additional_pages = {} - -# If false, no module index is generated. -#html_use_modindex = True - -# If false, no index is generated. -#html_use_index = True - -# If true, the index is split into individual pages for each letter. -#html_split_index = False - -# If true, the reST sources are included in the HTML build as _sources/. -html_copy_source = False - -# If true, an OpenSearch description file will be output, and all pages will -# contain a tag referring to it. The value of this option must be the -# base URL from which the finished HTML is served. -#html_use_opensearch = '' - -# If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml"). -#html_file_suffix = '' - -# Output file base name for HTML help builder. -htmlhelp_basename = 'clustodoc' - - -# Options for LaTeX output -# ------------------------ - -# The paper size ('letter' or 'a4'). -#latex_paper_size = 'letter' - -# The font size ('10pt', '11pt' or '12pt'). -#latex_font_size = '10pt' - -# Grouping the document tree into LaTeX files. List of tuples -# (source start file, target name, title, author, document class [howto/manual]). -#latex_documents = [ -# ('index', 'clusto.tex', 'clusto Documentation', -# 'Digg, Inc.', 'manual'), -#] - -# The name of an image file (relative to this directory) to place at the top of -# the title page. -#latex_logo = None - -# For "manual" documents, if this is true, then toplevel headings are parts, -# not chapters. -#latex_use_parts = False - -# Additional stuff for the LaTeX preamble. -#latex_preamble = '' - -# Documents to append as an appendix to all manuals. -#latex_appendices = [] - -# If false, no module index is generated. -#latex_use_modindex = True diff --git a/doc/getting_started.rst b/doc/getting_started.rst deleted file mode 100644 index aa485c9..0000000 --- a/doc/getting_started.rst +++ /dev/null @@ -1,82 +0,0 @@ -################################## - Getting Started -################################## - -:Release: |version| -:Date: |today| - -Installation ------------- - -From debian package -~~~~~~~~~~~~~~~~~~~ -Add the following to your /etc/apt/sources.list:: - - deb http://mirrors.digg.com/digg digg-public-lenny main contrib non-free - -Update the index and install clusto:: - - # aptitude update - # aptitude install clusto - -Building an rpm package -~~~~~~~~~~~~~~~~~~~~~~~ -You may need to install rpmdevtools beforehand, or run the rpmbuild as root:: - - $ cd rpm/ - $ ./make_tarball.sh - $ rpmbuild -ta clusto-$VERSION.tar.gz - -From source -~~~~~~~~~~~ -:: - - $ git clone git://github.com/digg/clusto.git - $ cd clusto - $ python setup.py build - -As root:: - - # python setup.py install - # mkdir /etc/clusto /var/lib/clusto - # cp conf/* /etc/clusto/ - # cp contrib/* /var/lib/clusto/ - -You may need to install additional python libraries for certain features of clusto to function properly. - -- SQLAlchemy: http://www.sqlalchemy.org/ -- setuptools: http://peak.telecommunity.com/ -- scapy: http://www.secdev.org/projects/scapy/doc/ -- ipython: http://ipython.scipy.org/moin/ -- IPy: http://c0re.23.nu/c0de/IPy/ -- MySQLdb: http://sourceforge.net/projects/mysql-python/ -- libvirt: http://libvirt.org/ -- simplejson: http://code.google.com/p/simplejson/ - -Configuration -------------- - -Clusto is built on top of SQLAlchemy and therefore should work with any backend database available to that library. For the same of simplicity, the following documentation assumes you are using a MySQL backend. - -Create a MySQL database -~~~~~~~~~~~~~~~~~~~~~~~ -:: - - # aptitude install mysql-server - # mysql -u root - mysql> CREATE DATABASE clusto; - mysql> GRANT ALL PRIVILEGES ON clusto.* TO 'clustouser'@'localhost' IDENTIFIED BY 'clustopass'; - mysql> FLUSH PRIVILEGES; - -/etc/clusto/clusto.conf -~~~~~~~~~~~~~~~~~~~~~~~ -:: - - [clusto] - dsn = mysql://clustouser:clustopass@127.0.0.1/clusto - -Creating tables -~~~~~~~~~~~~~~~ -Using clusto-shell:: - - init_clusto() diff --git a/doc/index.rst b/doc/index.rst deleted file mode 100644 index b06baad..0000000 --- a/doc/index.rst +++ /dev/null @@ -1,17 +0,0 @@ -.. clusto documentation master file, created by sphinx-quickstart on Wed Feb 24 23:53:25 2010. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -Welcome to clusto's documentation! -================================== - -Contents: - -.. toctree:: - :maxdepth: 2 - - getting_started.rst - tutorial.rst - using.rst - services.rst - cmdline.rst diff --git a/doc/services.rst b/doc/services.rst deleted file mode 100644 index 73623cf..0000000 --- a/doc/services.rst +++ /dev/null @@ -1,56 +0,0 @@ -################################## - Services -################################## - -:Release: |version| -:Date: |today| - -DHCP server -~~~~~~~~~~~ -As clusto acts as a single source of truth about your network, it only makes sense that you would use it for assigning network configuration to servers. Clusto's DHCP server isn't really dynamic though, it only hands out IPs that are already bound to server objects in the database and ignores DHCP requests from hosts that don't exist in the database or don't have IPs bound, though in both cases it will log a warning with details. - -The design of clusto-dhcpd is a bit different from the other daemons in that it communicates with the clusto database entirely over HTTP via the clusto-httpd daemon. This method was used to avoid the transfer of large result sets from the database to the DHCP server when certain queries took place (eg. IPManager.get_ip_manager), only to have the results filtered later on. Admittedly, this is due to inefficient querying by the clusto library and will probably be fixed in a later release. - -Configuration options ---------------------- -*api_url* -The base URL of the clusto-httpd server. Must NOT include a trailing slash - -*update_ipmi* -Boolean that sets whether or not requests will be checked for a client_id that resembles an IPMI card and distinguish it from a normal server MAC. If you don't have IPMI cards that use the same physical interfaces as your eth0, you probably don't need this. - -*extra_options* -Specify custom options that can be set via clusto attributes, in addition to those already supported by scapy. This is a dict where the key is an integer option ID and the value is a name to map to that option, to be used later as a subkey. - -Usage ------ -Upon receiving a DHCP request packet, clusto-dhcpd will look for any server object in the database with a Attribute(key='port-nic-eth', subkey='mac', value='00:11:22:33:44:55', number=1). If no object is found, the request is ignored. Otherwise, the daemon queries Attribute(key='dhcp', subkey='enabled', value=1, merge_container_attrs=True). This attribute is merged so that you can enable DHCP at a pool or datacenter level, depending on your needs. If the enabled attribute is found, the daemon builds a response based upon any remaining key='dhcp' attributes, including those merged from the parents. More specific attributes take precedence. - -Example -------- -:: - - p = Pool('pxeboot-servers') - p.set_attr(key='dhcp', subkey='enabled', value=1) - p.set_attr(key='dhcp', subkey='tftp_server', value='192.168.1.1') - p.set_attr(key='dhcp', subkey='tftp_filename', value='/pxelinux.0') - p.set_attr(key='dhcp', subkey='domain', value='example.org') - - s = BasicServer('ex0000') - s.set_port_attr('nic-eth', 1, 'mac', '00:11:22:33:44:55') - s.bind_ip_to_osport('192.168.1.50', 'eth0') - -SNMP trap listener -~~~~~~~~~~~~~~~~~~ -One of the services that ships with clusto called clusto-snmptrapd, will listen for SNMP traps from Cisco switches implementing the CISCO-MAC-NOTIFICATION-MIB which can be enabled with the following configuration on supported IOS releases:: - - snmp-server enable traps mac-notification change - snmp-server host version 2c mac-notification - mac address-table notification change - ! - interface - snmp trap mac-notification change added - -When clusto-snmptrapd receives one of these traps, it gathers the IP of the switch, the port number, the VLAN that learned the MAC, and the MAC itself. It then queries the clusto database for the switch IP and checks for an attribute of key='snmp', subkey='discovery', value=1... If this attribute is set on the switch or any of it's parents, the daemon checks to see if there's an object connected to the switch port in clusto. If not, then we assume this is a new server and create a new PenguinServer object with a name generated from a SimpleEntityNameManager called 'servernames'. The daemon then gets the switch's parent rack, gets the rack factory for that rack, then calls add_server(server, switchport) on the rack factory instance. - -It's a bit of a complicated process, but the end result is that new servers get added to clusto automatically as soon as they appear on the network without any human intervention aside from creating the rack instance. diff --git a/doc/tutorial.rst b/doc/tutorial.rst deleted file mode 100644 index 2b2d0f2..0000000 --- a/doc/tutorial.rst +++ /dev/null @@ -1,124 +0,0 @@ -###################### - Tutorial -###################### - -:Release: |version| -:Date: |today| - -Setting up a development environment ------------------------------------- -:: - -$ git clone git://github.com/digg/clusto.git -$ cd clusto -$ mkdir env -$ virtualenv env -$ source env/bin/activate -$ python setup.py install - -At this point, you have a virtualenv populated with the latest clusto code. Every time you want to use the new clusto code, you'll need to do this:: - -$ cd clusto -$ source env/bin/activate -$ export CLUSTODSN="sqlite:////path/to/sqlite.db" - -I usually edit the activate script and add the export command somewhere near the top. - -Writing clusto scripts ----------------------- - -Initialization -~~~~~~~~~~~~~~ -:: - - from clusto.scripthelpers import init_script - from clusto.drivers import * - import clusto - - init_script() - -Transactions -~~~~~~~~~~~~ - -Clusto stores versioning information for all objects. By default, all modifications to the clusto database are atomic. If you find yourself making a large number of related modifications at the same time, you may enclose them in a transaction so that the changeset is treated as a single version:: - - try: - clusto.begin_transaction() - # make a bunch of changes to objects - clusto.commit() - except: - clusto.rollback_transaction() - -Server creation workflow -~~~~~~~~~~~~~~~~~~~~~~~~ -:: - - names = clusto.get_by_name('servernames') - server = names.allocate(PenguinServer) - - # Put this server in a specific RU - rack = clusto.get_by_name('sjc1-000') - rack.insert(server, 1) - - # Connect eth0 to a switch port and bind a MAC address and IP - switch = clusto.get_by_name('sjc1-000-sw1') - server.connect_ports('nic-eth', 0, switch, 1) - server.set_port_attr('nic-eth', 0, 'mac', '00:11:22:33:44:55') - server.bind_ip_to_port('1.2.3.4', 'nic-eth', 0) - - # Connect power to an outlet - power = clusto.get_by_name('sjc1-000-pwr1') - server.connect_ports('pwr-nema-5', 0, power, 'aa1') - -Add a server to a pool -~~~~~~~~~~~~~~~~~~~~~~ -:: - - pool = clusto.get_by_name('development') - pool.insert(server) - -Remove a server from a pool -~~~~~~~~~~~~~~~~~~~~~~~~~~~ -:: - - pool.remove(server) - -Set an attribute on an object -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -:: - - pool = clusto.get_by_name('n00b_read') - pool.set_attr(key='mysql', subkey='port', value=3309) - -Querying clusto ---------------- - -Get a list of servers in a pool -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -:: - - servers = clusto.get_by_name('production').contents(clusto_types=['server']) - repr(servers) - [BasicServer(name=s0014, type=server, driver=basicserver), - BasicServer(name=s0015, type=server, driver=basicserver), - BasicServer(name=s0016, type=server, driver=basicserver), - BasicServer(name=s0017, type=server, driver=basicserver), - BasicServer(name=s0018, type=server, driver=basicserver), - BasicServer(name=s0019, type=server, driver=basicserver), - ... - -Find servers with the given IP address -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -:: - - servers = IPManager.get_device('10.2.128.103') - repr(servers) - [BasicServer(name=mgmt, type=server, driver=basicserver)] - -Find all IPs bound to a given server -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -:: - - iplist = clusto.get_by_name('s0000').get_ips() - repr(iplist) - ['10.2.128.107'] diff --git a/doc/using.rst b/doc/using.rst deleted file mode 100644 index fa0c5f7..0000000 --- a/doc/using.rst +++ /dev/null @@ -1,207 +0,0 @@ -################################## - Using clusto -################################## - -:Release: |version| -:Date: |today| - -In order to use clusto, you'll have to make it aware of your server environment. Every environment is built differently and may require different approaches to the discovery of servers and usage of clusto's features. - -IPManagers for your subnets -~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Clusto requires that every IP address you bind to a server must be associated with an IPManager instance for that subnet. This allows you to do dynamic allocation of addresses and associate related information (eg. netmask, broadcast) with every address. These features are especially useful when using clusto's DHCP server. - -:: - - from clusto.drivers import IPManager - ipman = IPManager('subnet-dmz', baseip='192.168.20.0', netmask='255.255.252.0') - -SimpleEntityNameManager -~~~~~~~~~~~~~~~~~~~~~~~ -In order to automatically discover and create new server objects, Clusto needs to be able to generate new names for objects. SimpleEntityNameManagers provide an allocate(Driver) method that creates a new instance and generates a name for it. - -:: - - from clusto.drivers import SimpleEntityNameManager, PenguinServer - - names = SimpleEntityNameManager('servernames', basename='s', digits=4) - server = names.allocate(PenguinServer) - -This will create a new server instance with a name like "s0000" - -Creating a datacenter -~~~~~~~~~~~~~~~~~~~~~ -Clusto comes ready to use with Basic drivers for a variety of entities and devices most systems engineers are familiar with. These Basic drivers can be used directly if only basic functionality is desired, but are intended to be subclassed and overridden with different variables and method. It is considered a best practice to create subclasses in case you need to customize a driver later on. -This is an example of a custom datacenter driver representing a colocation facility. (src/clusto/drivers/locations/datacenters/equinixdatacenter.py):: - - from basicdatacenter import BasicDatacenter - - class EquinixDatacenter(BasicDatacenter): - ''' - Equinix datacenter driver - ''' - - _driver_name = 'equinixdatacenter' - -Add the following to src/clusto/drivers/locations/datacenters/__init__.py:: - - from equinixdatacenter import * - -This subclass is an example of a simple wrapper that does not provide additional functionality beyond distinguishing this type of datacenter from another one. - -At clusto's core, it's a tool for gluing together other services, protocols, and logic in a data-centric way. An example of such usage would be a method that opens a ticket with the colo provider for a remote hands action. Add the following to equinixdatacenter.py:: - - from smtplib import SMTP - def remote_hands(self, message, priority='low', mail_server='mail.example.com', mail_from='clusto@example.com', mail_to='remotehands@datacenter.net'): - msg = 'From: %s\r\nTo: %s\r\nSubject: Remote hands request: %s priority\r\n\r\n%s' % (mail_from, mail_to, priority, request) - server = SMTP(mail_server) - server.sendmail(mail_from, mail_to, request) - server.quit() - -Using clusto-shell we can instantiate our new datacenter and test the remote_hands method:: - - from clusto.drivers import EquinixDatacenter - - datacenter = EquinixDatacenter('eqix-sv3') - datacenter.remote_hands('Turn on power for new racks in my cage', priority='high') - -Adding racks to the datacenter -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Rack factories are fairly specific to each organization's environment... Common rack layouts are uncommon. For these types of highly-customized features, you'll need to modify the files in /var/lib/clusto to suit your needs. Clusto ships with some examples of the files used by Digg. - -Take a look at /var/lib/clusto/rackfactory.py to get an idea of how to programatically define rack layouts. - -Continuing our example from above, let's assume the remote hands at the datacenter plugged in your new rack and closed the ticket. Now you want to be able to manipulate the rack in clusto and start adding servers to it. Let's go back to clusto-shell:: - - from clusto.drivers import APCRack - - rack = APCRack('sv3-001') - rack.set_attr(key='racklayout', value='201001') - datacenter = get_by_name('eqix-sv3') - datacenter.insert(rack) - -That's all you need to do to create a new rack instance and insert it into a datacenter. The set_attr for rack layout will be explained in the following section. - -Rack factory -~~~~~~~~~~~~ -If you have more than one rack using the same layout of devices and connections between them, clusto can ease a lot of the data entry involved with creating and populating new racks with custom RackFactory classes:: - - from rackfactory import get_factory - - factory = get_factory('sv3-001') - factory.connect_ports() - -The get_factory call will get the 'sv3-001' rack instance from clusto and lookup an attribute of key='rackfactory' to determine which factory class should be used to fill in the rest of the information. The __init__ method of Digg201001RackFactory also creates network, console, and power switch instances with names based on the name of the rack. - -The connect_ports method ensures that this rack is in the datacenter, that the network, console, and power instances are in this rack, and that their ports are all connected as intended. This gives us the basic structure of everything that will be identical across all racks with this layout. - -Virtual machines -~~~~~~~~~~~~~~~~ -Clusto supports managing virtual machines and their host environments through subclasses of VMManager. At the time of writing the only working implementation is XenVMManager. - -Assuming you already have some server objects and you've installed Xen with libvirtd on those servers, with an LVM volume group named "vg0", the clusto side of things goes a bit like this:: - - from clusto.drivers import XenVMManager, XenVirtualServer - - hosts = [ - clusto.get_by_name('xenhost1') - clusto.get_by_name('xenhost2') - clusto.get_by_name('xenhost3') - ] - - # Every hypervisor MUST have the following four attributes set - for x in hosts: - x.set_attr(key='xen', subkey='volume-group', value='vg0') # LVM VG name - x.set_attr(key='system', subkey='memory', value=16384) # RAM in MB - x.set_attr(key='system', subkey='disk', value=2000) # Disk in GB - x.set_attr(key='system', subkey='cpucount', value=8) # Logical CPUs - - manager = XenVMManager('xenmanager') - [manager.insert(x) for x in hosts] - -Now that you have your host machines configured, allocate a new VM:: - - vm = XenVirtualServer('xenvm1') - vm.bind_ip_to_osport('192.168.1.51', 'eth0') - vm.set_port_attr('nic-eth', 1, 'mac', '02:52:0a:00:00:01') - vm.set_attr(key='system', subkey='memory', value=1024) - vm.set_attr(key='system', subkey='disk', value=20) - vm.set_attr(key='system', subkey='cpucount', value=1) - manager.allocate(vm) - - vm.vm_create() # Create the LVM logical volumes, define the domain - vm.vm_start() # Start the defined domain - vm.vm_console() # SSH to the host and open the VM's console - vm.vm_stop() # Shutdown the VM - -If you already have IPManager and SimpleEntityNameManager instances setup, the command line tools should work as well:: - - $ clusto vm create --disk 20 --memory 1024 - Created v1000 - $ clusto vm start v1000 - -EC2 virtual machines -~~~~~~~~~~~~~~~~~~~~ -ALPHA! - -EC2 support is in it's very early stages but works like this:: - - from clusto.drivers import EC2VirtualServer, EC2VMManager - - ec2manager = EC2VMManager("myec2", aws_access_key_id="...", aws_secret_access_key="...") - - vserver = EC2VirtualServer("v01") - vserver.add_attr(key='aws', subkey='ec2_instance_type',value='m1.small') - vserver.add_attr(key='aws', subkey='ec2_region',value='eu-west-1') - vserver.add_attr(key='aws', subkey='ec2_ami',value='ami-cf4d67bb') - - - ec2manager.allocate(vserver) - - vserver.get_state() # checks the instance state with EC2 - - vserver.get_ips() # gets the public and private IPs from EC2 - - vserver.set_attr(key='aws', subkey='ec2_allow_termination', value=False) - - ec2manager.deallocate(vserver) - >>> EC2VMManagerException: Not Allowed to terminate v01. - - vserver.set_attr(key='aws', subkey='ec2_allow_termination', value=True) - - ec2manager.deallocate(vserver) # terminates the instance at EC2 - -The EC2VMManager has stubs for budgeting which aren't fleshed out yet (to restrict $/hr spent). There is also a helper function:: - - ec2manager.get_all_ec2_instance_resources() - -That returns all the instance information for all regions as seen by EC2. This might not be the greatest idea when you start to get a lot of instances. - - -Pool types -~~~~~~~~~~ -Clusto currently supports three different types of "pools." Pools are used to group entities together and apply attributes to the groupings. The supported pool types are: - -Pool - This is your basic pool type. An entity can only be in a given pool once. -ExclusivePool - A given entity can only be in an ExclusivePool if it is in NO other pools. This is useful for tracking things like unallocated servers; you would not want a server marked unallocated if it is already a member of your "production" pool. -UniquePool - A given entity can only be in a UniquePool if it is not in any other UniquePool(s). This pool type is intended to be subclassed. For example you may want to create EnvironmentPool to represent "development", "staging", and "production" environments; UniquePool is useful because you would not want a given entity to be in both "production" and "development". - -:: - - from clusto.drivers import UniquePool, PenguinServer - - class EnvironmentPool(UniquePool): - _driver_name = "environment_pool" - - development = EnvironmentPool("development") - staging = EnvironmentPool("staging") - production = EnvironmentPool("production") - server = PenguinServer("s0001") - development.insert(server) - production.insert(server) - -PoolException: PenguinServer(name=s0001, type=server, driver=penguinserver) is already in UniquePool(s) [UniquePool(name=development, type=pool, driver=unique_pool)]. -