Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Browse files

Work in progress migration to use clihelper and reduce the code footp…

…rint.

- Change Logging configuration to use dictConfig format
- Update company name in LICENSE
- Update requirements and test config in README.md
- Change requirements in setup.py
- Update the console_script in setup.py
- Remove __all__ in __init__.py
- docstring cleanup in application.py
- remove cli.py
- Add a new controller class extending clihelper.Controller
- Update the logger and doc strings in clients/pgsql.py
- Cleanup process.py, extending multiprocessing.Process for the TinmanProcess
- Remove unused functions from utils.py, should be able to remove it fully
- Whitespace cleanups and removal of unused imports
  • Loading branch information...
commit 5afaeb6c96c87a113dda0daf6156ee27e845a276 1 parent ffb8ea2
Gavin M. Roy authored
View
4 LICENSE
@@ -1,4 +1,4 @@
-Copyright (c) 2011, Insider Guides, Inc
+Copyright (c) 2011-2012, MeetMe
All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
@@ -9,7 +9,7 @@ are permitted provided that the following conditions are met:
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
- * Neither the name of the Insider Guides, Inc. nor the names of its
+ * Neither the name of the MeetMe nor the names of its
contributors may be used to endorse or promote products derived from this
software without specific prior written permission.
View
24 README.md
@@ -16,6 +16,7 @@ decorators and utilities.
setting logging levels for individual packages.
## Requirements
+- clihelper
- ipaddr
- pyyaml
@@ -100,21 +101,14 @@ Configure the tornado.httpserver.HTTPServer with the following options:
- xheaders: Enable X-Header support in tornado.httpserver.HTTPServer
#### Logging Options
-Enable standard python logging with the following options:
+Logging uses the dictConfig format as specified at
-- directory: Optional log file output directory
-- filename: Optional filename, not needed for syslog
-- format: Logging output format
-- level: One of debug, error, warning, info
-- handler: Optional handler
-- syslog: If handler == syslog, parameters for syslog
- - address: Syslog address
- - facility: Syslog facility
+ http://docs.python.org/library/logging.config.html#dictionary-schema-details
#### Route List
The route list is specified using the top-level Routes keyword. Routes consist
of a list of individual route items that may consist of mulitiple items in a route
-tuple
+tuple.
##### Traditional Route Tuples
The traditional route tuple, as expected by Tornado is a two or three item tuple
@@ -147,6 +141,10 @@ The following is an example tinman application configuration:
%YAML 1.2
---
+ user: www-data
+ group: www-data
+ pidfile: /var/run/tinman/tinman.pid
+
Application:
base_path: /home/foo/mywebsite
debug: True
@@ -164,8 +162,10 @@ The following is an example tinman application configuration:
pika: pika
myapp: myapp.handlers
formatters:
- verbose: "%(levelname) -10s %(asctime)s %(funcName) -25s: %(message)s"
- syslog: "%(levelname)s <PID %(process)d:%(processName)s> %(message)s"
+ verbose:
+ format: "%(levelname) -10s %(asctime)s %(funcName) -25s: %(message)s"
+ syslog:
+ format: "%(levelname)s <PID %(process)d:%(processName)s> %(message)s"
handlers:
console:
class: logging.StreamHandler
View
127 etc/example.yaml
@@ -1,89 +1,98 @@
%YAML 1.2
---
Application:
- debug: True
- xsrf_cookies: False
- whitelist:
- - 10.0.0.0/8
- - 192.168.1.0/24
- - 1.2.3.4/32
+ debug: true
+ stats_port: 9090
+ xsrf_cookies: false
+ wake_interval: 60
+ whitelist:
+ - 10.0.0.0/8
+ - 192.168.1.0/24
+ - 1.2.3.4/32
+
+Daemon:
+ pidfile: /tmp/myapp.pid
HTTPServer:
- no_keep_alive: False
- ports: [8000]
- xheaders: True
+ no_keep_alive: false
+ ports: [8000]
+ xheaders: true
Logging:
- loggers:
- tinman:
- level: INFO
- propagate: True
- filters:
- tinman: tinman
- myapp: myapp.handlers
+ version: 1
+
formatters:
verbose:
- format: "%(levelname) -10s %(asctime)s %(name) -30s %(funcName) -25s: %(message)s"
+ format: '%(levelname) -10s %(asctime)s %(process)-6d %(processName) -20s %(name) -20s %(funcName) -25s: %(message)s'
+ datefmt: '%Y-%m-%d %H:%M:%S'
syslog:
- format: "%(levelname)s <PID %(process)d:%(processName)s> %(name).%(funcName)s: %(message)s"
+ format: ' %(levelname)s <PID %(process)d:%(processName)s> %(name)s.%(funcName)s(): %(message)s'
+
handlers:
console:
class: logging.StreamHandler
- formatter: verbose
debug_only: True
- filters:
- - myapp
- - tinman
- level: DEBUG
+ formatter: verbose
+
+ error:
+ filename: /Users/gmr/Source/Tinman/logs/error.log
+ class: logging.handlers.RotatingFileHandler
+ maxBytes: 104857600
+ backupCount: 6
+ formatter: verbose
+
file:
- filename: /var/log/tinman.log
- class: logging.RotatingFileHandler
- mode: a
+ filename: /Users/gmr/Source/Tinman/logs/tinman.log
+ class: logging.handlers.RotatingFileHandler
maxBytes: 104857600
backupCount: 6
- encoding: UTF-8
- delay: False
formatter: verbose
- filters:
- - myapp
- - tinman
+
syslog:
class: logging.handlers.SysLogHandler
facility: local6
address: /var/run/syslog
formatter: syslog
+
+ loggers:
+
+ clihelper:
+ handlers: [console]
+ level: DEBUG
+ propagate: True
+ formatter: verbose
+
+ tinman:
+ handlers: [console, file]
+ propagate: True
+ formatter: verbose
+ level: DEBUG
+
+ tornado:
+ handlers: [console, file]
+ propagate: True
+ formatter: verbose
level: INFO
- filters:
- - myapp
- - tinman
-# As of 0.2.1 having this automatically creates a RabbitMQ connection for your tinman app
-# It is exposed as application.tinman.rabbitmq
+ root:
+ handlers: [error]
+ formatter: verbose
+ level: ERROR
+
+ disable_existing_loggers: True
+
+
RabbitMQ:
- host: localhost
- port: 5672
- username: guest
- password: guest
- virtual_host: /
-
-# Route / Request Handler mapping
-# New format includes better support for complex regex where previous would break YAML's sequence format
-Routes:
+ host: localhost
+ port: 5672
+ username: guest
+ password: guest
+ virtual_host: /
- # Homepage
+Routes:
+ - [/, test.example.Home]
+ - [re, '/(c[a-f0-9]f[a-f0-9]{1,3}-[a-f0-9]{8}).gif', test.example.Pixel]
-
- - /
- - test.example.Home
-
- # Auto-generated pixel using the regex supported syntax, put re before the URI regex in the list
- -
- # /c1f1-7c5d9e0f.gif
- - re
- - /(c[a-f0-9]f[a-f0-9]{1,3}-[a-f0-9]{8}).gif
- - test.example.Pixel
-
- # Any URL not matched up to this point
- -
- .*
- tornado.web.RedirectHandler
- - {"url": "http://www.github.com"}
+ - url: http://www.github.com
View
15 setup.py
@@ -21,16 +21,19 @@
url='http://github.com/gmr/tinman',
license='BSD',
packages=['tinman', 'tinman.clients'],
- requires=['ipaddr',
- 'logging_config',
+ requires=['clihelper',
+ 'ipaddr',
'python_daemon',
'pyyaml'],
extras_require= {'RabbitMQ': 'pika',
'PostgreSQL': 'psycopg2',
'LDAP': 'ldap',
'Redis': 'brukva'},
- data_files=[('/usr/local/share/tinamn/init.d', ['etc/init.d/tinman']),
- ('/usr/local/share/tinamn/', ['etc/example.yaml', 'README.md']),
- ('/usr/local/share/tinamn/sysconfig', ['etc/sysconfig/tinman'])],
- entry_points=dict(console_scripts=['tinman=tinman.cli:main']),
+ data_files=[('/usr/local/share/tinamn/init.d',
+ ['etc/init.d/tinman']),
+ ('/usr/local/share/tinamn/',
+ ['etc/example.yaml', 'README.md']),
+ ('/usr/local/share/tinamn/sysconfig',
+ ['etc/sysconfig/tinman'])],
+ entry_points=dict(console_scripts=['tinman=tinman.controller:main']),
zip_safe=True)
View
0  test/__init__.py
No changes.
View
16 test/example.py
@@ -1,16 +0,0 @@
-from tinman.utils import log_method_call
-from tornado.web import RequestHandler
-
-
-class Home(RequestHandler):
-
- @log_method_call
- def get(self):
- self.write("Hello, World!")
-
-
-class Catchall(RequestHandler):
-
- @log_method_call
- def get(self, parameters):
- self.write("URI: %s" % parameters)
View
112 test/whitelist_test.py
@@ -1,112 +0,0 @@
-from mock import Mock
-
-import sys
-sys.path.insert(0, '..')
-
-import tinman.whitelist as whitelist
-
-# Monkey patch in our HTTPError to be an AssertionError
-whitelist.HTTPError = AssertionError
-
-
-# Mock up the values
-class RequestMock(object):
-
- def __init__(self, remote_ip):
-
- # Mock the application object
- self.application = Mock()
- self.application.settings = dict()
-
- # Mock up the request object
- self.request = Mock()
- self.request.remote_ip = remote_ip
-
- @whitelist.whitelisted
- def whitelisted_method(self):
- return True
-
- @whitelist.whitelisted("11.12.13.0/24")
- def whitelisted_specific(self):
- return True
-
-
-def test_empty_whitelist():
-
- # Mock a request
- mock = RequestMock('1.2.3.4')
-
- # This test should raise a ValueError exception in the whitelist
- try:
- rval = mock.whitelisted_method()
- except ValueError:
- rval = False
-
- # If we didn't get a value error, fail
- if rval:
- assert False, "ValueError not raised for empty whitelist"
-
-
-def test_whitelisted_ip():
-
- # Mock a request
- mock = RequestMock('1.2.3.4')
- mock.application.settings['whitelist'] = ['1.2.3.0/24']
-
- try:
- rval = mock.whitelisted_method()
- except whitelist.HTTPError:
- rval = False
-
- # mock.whitelisted_method should return true for our mock data
- if not rval:
- assert False, "Whitelisted IP address didn't pass"
-
-
-def test_non_whitelisted_ip():
-
- # Mock a request
- mock = RequestMock('2.2.3.4')
- mock.application.settings['whitelist'] = ['1.2.3.0/24']
-
- # This test should raise a ValueError exception in the whitelist
- try:
- rval = mock.whitelisted_method()
- except whitelist.HTTPError:
- rval = False
-
- # If we didn't get a value error, fail
- if rval:
- assert False, "whitelist did not raise HTTPError"
-
-
-def test_specified_whitelisted_ip():
-
- # Mock a request
- mock = RequestMock('11.12.13.14')
- mock.application.settings['whitelist'] = ['1.2.3.0/24']
-
- try:
- rval = mock.whitelisted_specific()
- except whitelist.HTTPError:
- rval = False
-
- # mock.whitelisted_method should return true for our mock data
- if not rval:
- assert False, "Whitelisted IP address didn't pass"
-
-
-def test_specified_non_whitelisted_ip():
-
- # Mock a request
- mock = RequestMock('2.2.3.4')
-
- # This test should raise a ValueError exception in the whitelist
- try:
- rval = mock.whitelisted_specific()
- except whitelist.HTTPError:
- rval = False
-
- # If we didn't get a value error, fail
- if rval:
- assert False, "whitelist did not raise HTTPError"
View
14 tinman/__init__.py
@@ -1,17 +1,9 @@
-#!/usr/bin/env python
"""
Core Tinman imports
+
"""
__author__ = 'Gavin M. Roy'
__email__ = '<gmr@myyearbook.com>'
__since__ = '2011-03-14'
-__version__ = "0.4.1"
-
-__all__ = ['tinman.application',
- 'tinman.cache',
- 'tinman.cli',
- 'tinman.clients',
- 'tinman.loader',
- 'tinman.process',
- 'tinman.utils',
- 'tinman.whitelist']
+__version__ = '0.5.0'
+__desc__ = 'Tornado application wrapper and toolset'
View
6 tinman/application.py
@@ -23,7 +23,7 @@ def _replace_value(original, key, value):
:type key: str
:param value: The string value to replace it with
:type value: str
- :returns: str
+ :rtype: str
"""
return original.replace(key, value)
@@ -156,7 +156,7 @@ def _prepare_route(self, attributes):
:param attributes: Route attributes
:type attributes: list or tuple
- :returns: list of prepared route
+ :rtype: list of prepared route
"""
# Validate it's a list or set
if type(attributes) not in (list, tuple):
@@ -206,7 +206,7 @@ def _prepare_routes(self, routes):
:param routes: Routes to prepare
:type routes: list
- :returns: list
+ :rtype: list
:raises: ValueError
"""
View
241 tinman/cli.py
@@ -1,241 +0,0 @@
-"""
-Wrap the command line interaction in an object
-
-"""
-__author__ = 'Gavin M. Roy'
-__email__ = 'gmr@myyearbook.com'
-__since__ = '2011-12-31'
-
-# Tinman imports
-from . import process
-from . import utils
-from . import __version__
-
-import logging
-import logging_config
-import optparse
-import os
-import sys
-import time
-
-
-class TinmanCLI(object):
- """Main application controller class"""
- def __init__(self):
- """Create a new instance of the TinmanCLI class"""
- self._children = list()
- self._config = dict()
- self._options = None
- self._logger = logging.getLogger(__name__)
-
- def _check_required_configuration_parameters(self):
- """Validates that the required configuration parameters are set.
-
- :raises: AttributeError
-
- """
- # Required sections
- if 'Application' not in self._config:
- raise AttributeError("Missing Application section in configuration")
-
- if 'HTTPServer' not in self._config:
- raise AttributeError("Missing HTTPServer section in configuration")
-
- if 'Logging' not in self._config:
- raise AttributeError("Missing Logging section in configuration")
-
- if not isinstance(self._config['Routes'], list):
- raise AttributeError("Error in Routes section in configuration")
-
- def _daemonize(self):
- """Daemonize the python process if we need to, otherwise set the app in
- debug mode.
-
- """
- utils.daemonize(pidfile=self._config.get("pidfile", None),
- user=self._config.get("user", None),
- group=self._config.get("group", None))
-
- def _fixup_configuration(self):
- """Rewrite the SSL certreqs option if it exists, do this once instead
- # of in each process like we do for imports and other things
-
- """
- if 'ssl_options' in self._config['HTTPServer']:
- self._fixup_ssl_config()
-
- def _fixup_ssl_config(self):
- """Check the config to see if SSL configuration options have been passed
- and replace none, option, and required with the correct values in
- the certreqs attribute if it is specified.
-
- """
- if 'cert_reqs' in self._config['HTTPServer']['ssl_options']:
-
- # Build a mapping dictionary
- import ssl
- reqs = {'none': ssl.CERT_NONE,
- 'optional': ssl.CERT_OPTIONAL,
- 'required': ssl.CERT_REQUIRED}
-
- # Get the value
- cert_reqs = \
- reqs[self._config['HTTPServer']['ssl_options']['cert_reqs']]
-
- # Remap the value
- self._config['HTTPServer']['ssl_options']['cert_reqs'] = cert_reqs
-
- def _load_configuration(self):
- """Load the configuration for the given options.
-
- """
- # No configuration file?
- if not self._options.config:
- self._config = self._load_test_config()
- return
-
- # Load the configuration file
- self._config = utils.load_yaml_file(self._options.config)
-
- # Fixup the any of the configuration as needed
- self._fixup_configuration()
-
- def _load_test_config(self):
- """Load the test config from the test module returning a dictionary.
-
- :returns: dict
-
- """
- sys.stdout.write('\nConfiguration not specified, running Tinman Test '
- 'Application\n')
- from . import test
- return test.CONFIG
-
- def _process_options(self):
- """Process the cli options returning the options and arguments"""
- usage = "usage: %prog -c <configfile> [options]"
- version_string = "%%prog v%s" % __version__
- description = "Tinman's Tornado application runner"
-
- # Create our parser and setup our command line options
- parser = optparse.OptionParser(usage=usage,
- version=version_string,
- description=description)
-
- parser.add_option("-c", "--config",
- action="store",
- default=False,
- dest="config",
- help="Specify the configuration file for use")
- parser.add_option("-f", "--foreground",
- action="store_true",
- dest="foreground",
- default=False,
- help="Run interactively in console")
-
- # Parse our options and arguments
- return parser.parse_args()
-
- def _remove_pidfile(self):
- """Remove the PID file from the filesystem.
-
- """
- if 'pidfile' in self._config and \
- os.path.exists(self._config['pidfile']):
- try:
- os.unlink(self._config['pidfile'])
- except OSError as e:
- self._logger.error("Could not remove pidfile: %s", e)
-
- def _setup_logging(self, config, debug):
- """Construct the logging config object and
-
- """
- self._logging = logging_config.Logging(config, debug)
- self._logging.setup()
- self._logging.remove_root_logger()
- self._logging.remove_existing_loggers()
-
- def _terminate_children(self):
- """Send term signals to all of the children and wait for them to
- shutdown.
-
- """
- for child in self._children:
- self._logger.debug("Sending terminate signal to %s", child.name)
- child.terminate()
-
- # Loop while the children are shutting down
- self._logger.debug("Waiting for children to shutdown")
- while True in [child.is_alive() for child in self._children]:
- time.sleep(0.5)
-
- def _tinman_process(self):
- """Create the core tinman process object, start it and return the handle
-
- :returns: tinman.process.TinmanProcess
-
- """
- tinman = process.TinmanProcess(self._config)
- tinman.start(self._config)
- return tinman
-
- def _wait_while_running(self):
- """Just loop and sleep while we are actively running."""
- utils.running = True
- while utils.running:
- try:
- time.sleep(1)
- except KeyboardInterrupt:
- self._logger.info("CTRL-C pressed, shutting down.")
- break
-
- def run(self):
- """Run the Tinman Process"""
-
- # Process our command line options
- self._options, args = self._process_options()
-
- # Load the configuration
- self._load_configuration()
-
- # Setup the logging for this process
- self._setup_logging(self._config['Logging'], self._options.foreground)
-
- # Check our required options
- self._check_required_configuration_parameters()
-
- # If we have a base path set prepend it to our Python import path
- if 'base_path' in self._config:
- sys.path.insert(0, self._config['base_path'])
-
- # Set debug if needed
- if self._options.foreground:
- self._config['Application']['debug'] = True
-
- # Setup our signal handlers
- utils.setup_signals()
-
- # Daemonize if we need to
- if not self._options.foreground:
- self._daemonize()
-
- # Create the core object
- self._tinman_process()
-
- # Block while running
- self._wait_while_running()
-
- # Tell all the children to shutdown
- self._terminate_children()
-
- # Remove our pidfile
- self._remove_pidfile()
-
- # Log that we're shutdown cleanly
- self._logger.info("tinman has shutdown")
-
-def main():
-
- tinman = TinmanCLI()
- tinman.run()
View
18 tinman/clients/pgsql.py
@@ -17,7 +17,7 @@
from psycopg2 import extras
_CONNECTIONS = None
-_LOGGER = logging.getLogger(__name__)
+logger = logging.getLogger(__name__)
def _add_cached_connection(connection_hash, connection):
@@ -37,9 +37,9 @@ def _add_cached_connection(connection_hash, connection):
# Assign the connection to the pool
_CONNECTIONS[connection_hash] = {'connections': 1,
'connection': connection}
- _LOGGER.info('Appended connection %s to module pool', connection_hash)
+ logger.info('Appended connection %s to module pool', connection_hash)
else:
- _LOGGER.error('Attempt to append already cached connection: %s',
+ logger.error('Attempt to append already cached connection: %s',
connection_hash)
@@ -48,7 +48,7 @@ def _generate_connection_hash(dsn, async=False):
:param str dsn: DSN for connection
:param bool async: Connection is setup in async mode
- :returns: str
+ :rtype: str
"""
# Create our hashlib object
@@ -66,7 +66,7 @@ def _get_cached_connection(connection_hash):
connection with the same exact connection parameters and use it if so.
:param str connection_hash: Hash generated by _generate_connection_hash
- :returns: psycopg2._psycopg.connection or None
+ :rtype: psycopg2._psycopg.connection or None
"""
global _CONNECTIONS
@@ -147,20 +147,20 @@ def __init__(self, host='localhost', port=5432, dbname=None, user='www',
# If we got a result, just log our success in doing so
if self._pgsql:
- _LOGGER.debug("Re-using cached connection: %s",
+ logger.debug("Re-using cached connection: %s",
self._connection_hash)
# Create a new PostgreSQL connection and cache it
else:
# Connect to a PostgreSQL daemon
- _LOGGER.info("Connecting to %s:%i:%s as %s",
+ logger.info("Connecting to %s:%i:%s as %s",
host, port, dbname, user)
self._pgsql = pg_connect(dsn)
# Commit after every command
self._pgsql.set_isolation_level(
extensions.ISOLATION_LEVEL_AUTOCOMMIT)
- _LOGGER.info('Connected to PostgreSQL')
+ logger.info('Connected to PostgreSQL')
# Add the connection to our module level pool
_add_cached_connection(self._connection_hash, self._pgsql)
@@ -181,7 +181,7 @@ def connection(self):
def cursor(self):
"""Returns the cursor instance
- :returns: psycopg2._psycopg.cursor
+ :rtype: psycopg2._psycopg.cursor
"""
return self._cursor
View
167 tinman/controller.py
@@ -0,0 +1,167 @@
+"""
+The Tinman Controller class, uses clihelper for most of the main functionality
+with regard to configuration, logging and daemoniaztion. Spawns a
+tornado.HTTPServer and Application per port using multiprocessing.
+
+"""
+__author__ = 'Gavin M. Roy'
+__email__ = 'gmr@myyearbook.com'
+__since__ = '2011-12-31'
+
+
+import clihelper
+import logging
+import multiprocessing
+import sys
+import time
+from tornado import version as tornado_version
+
+
+# Tinman Imports
+from tinman import __desc__
+from tinman import __version__
+from tinman import application
+from tinman import process
+
+# Additional required configuration keys
+_REQUIRED_CONFIG_KEYS = ['HTTPServer', 'Routes']
+_SHUTDOWN_SLEEP_INTERVAL = 0.25
+
+logger = logging.getLogger(__name__)
+
+
+class TinmanController(clihelper.Controller):
+ """Main application controller class. Responsible for spawning all of the
+ HTTPServer / Applications.
+
+ """
+ def __init__(self, options, arguments):
+ """Create a new instance of the TinmanController class
+
+ :param optparse.Values options: CLI Options
+ :param list arguments: Additional CLI arguments
+
+ """
+ super(TinmanController, self).__init__(options, arguments)
+
+ # A list of child processes
+ self._children = list()
+
+ # The queue to use for receiving stats updates from child processes
+ self._stats_queue = self._get_stats_queue()
+
+ def _create_process(self, port):
+ """Create an Application and HTTPServer for the given port.
+
+ :param int port: The port to listen on
+ :rtype: multiprocessing.Process
+
+ """
+ logger.info('Creating process for TCP port %i', port)
+ return process.TinmanProcess(name="ServerProcess.%i" % port,
+ args=(self._config,
+ port,
+ self._stats_queue,
+ self._debug))
+
+ def _get_http_server_config(self):
+ """Return the HTTPServer configuration
+
+ :rtype: dict
+
+ """
+ return self._config['HTTPServer']
+
+ def _get_stats_queue(self):
+ """Return an instance of multiprocessing.Queue for passing stats back
+ to this process.
+
+ :rtype: multiprocessing.Queue
+
+ """
+ return multiprocessing.Queue()
+
+ def _insert_base_path(self):
+ """Inserts a base path into the sys.path list if one is specified in
+ the configuration.
+
+ """
+ base_path = self._get_application_config().get('base_path')
+ if base_path:
+ logger.debug('Appending %s to the sys.path list')
+ sys.path.insert(0, base_path)
+
+ def _process(self):
+ """Called when the controlling loop wakes. Use to gather stats
+ information to present to the stats HTTP server.
+
+ """
+ logger.debug('Waking up parent process')
+
+ def _reload_configuration(self):
+ """Reload the configuration via clihelper.Controller and then perform
+ the fixups needed.
+
+ """
+ super(TinmanController, self)._reload_configuration()
+
+ # Notify children
+
+ def _setup(self):
+ """Additional setup steps."""
+ logger.info('Tinman v%s starting up with Tornado v%s',
+ __version__, tornado_version)
+
+ # Prepend the base path if it is set
+ self._insert_base_path()
+
+ # Startup the child processes
+ self._start_children()
+
+ def _shutdown(self):
+ """Called when the application is shutting down, notify the child
+ processes and loop until they are shutdown.
+
+ """
+ self._set_state(self._STATE_SHUTTING_DOWN)
+
+ logger.info('Terminating child processes')
+ for child in self._children:
+ child.terminate()
+
+ # Loop while children are alive
+ logger.info('Waiting for all child processes to die')
+ while all([child.is_alive() for child in self._children]):
+ time.sleep(_SHUTDOWN_SLEEP_INTERVAL)
+
+ logger.debug('All child processes have stopped')
+
+ # Note that the shutdown process is complete
+ self._shutdown_complete()
+
+ def _start_children(self):
+ """Start the child processes"""
+ http_server_config = self._get_http_server_config()
+
+ # Iterate over the ports in the config
+ for port in http_server_config['ports']:
+ child = self._create_process(port)
+ child.start()
+ self._children.append(child)
+
+
+def add_required_config_keys():
+ """Add each of the items in the _REQUIRED_CONFIG_KEYS to the
+ clihelper._CONFIG_KEYS for validation of the configuration file. If one of
+ the items is not present in the config file, an exception will be thrown
+ and the application will be shutdown.
+
+ """
+ [clihelper.add_config_key(key) for key in _REQUIRED_CONFIG_KEYS]
+
+def main():
+ """Invoked by the script installed by setuptools."""
+ clihelper.setup('tinman', __desc__, __version__)
+ add_required_config_keys()
+ clihelper.run(TinmanController)
+
View
368 tinman/process.py
@@ -1,175 +1,254 @@
"""
-Tinman command line interface
+process.py
+
"""
+
__author__ = 'Gavin M. Roy'
__email__ = 'gmr@myyearbook.com'
-__since__ = '2011-06-06'
-
-# Tinman imports
-from . import application
-from . import __version__
+__since__ = '2012-04-29'
-# General imports
+from tinman import application
+import clihelper
+from tornado import httpserver
+from tornado import ioloop
import logging
-import logging_config
import multiprocessing
import signal
import socket
-
-# Tornado imports
-from tornado import httpserver
-from tornado import ioloop
+import ssl
from tornado import version as tornado_version
+logger = logging.getLogger(__name__)
+
-class TinmanProcess(object):
- """
- Manages the tinman master process when run from the cli
- """
- def __init__(self, config):
- """Create a TinmanProcess object
+class TinmanProcess(multiprocessing.Process):
+ """The process holding the HTTPServer and Application"""
- :param dict config: The configuration dictionary
+ def __init__(self, group=None, target=None, name=None, args=(), kwargs={}):
+ """Create a new instance of TinmanProcess
"""
- # Set a default IOloop to None
- self._ioloop = None
+ super(TinmanProcess, self).__init__(group, target, name, args, kwargs)
- # Set the configuration
- self._config = config
+ # Passed in values
+ self._config = args[0]
+ self._port = args[1]
+ self._stats_queue = args[2]
+ self._debug = args[3]
- # Get our logger
- self._logger = logging.getLogger(__name__)
+ # Internal attributes holding instance information
+ self._app = None
+ self._server = None
+ self._request_counters = dict()
- # Setup logging
- self._setup_logging(config['Logging'],
- config['Application'].get('debug', False))
+ # Fixup the configuration parameters
+ self._fixup_configuration(self._config)
- def _build_connections(self):
- """Build and attach our supported connections to our IOLoop and
- application object.
+ def _application(self, config):
+ """Create and return a new instance of tornado.web.Application
+
+ :param dict config: The application configuration
"""
- # Build a PostgreSQL connection if it exists
- if self._config.get('Postgres'):
- self._build_postgres_connection()
+ logging.debug('Creating a new Application with %r', config)
+ return application.TinmanApplication(self._get_routes(), **config)
- # Build a RabbitMQ connection if it exists
- if self._config.get('RabbitMQ'):
- self._build_rabbitmq_connection()
+ def _fixup_configuration(self, config):
+ """Rewrite the SSL certreqs option if it exists, do this once instead
+ # of in each process like we do for imports and other things
- # Build a Redis connection if it exists
- if self._config.get('Redis'):
- self._build_redis_connection()
+ :param dict config: the configuration dictionary
- def _build_postgres_connection(self):
- """Create a connection to PostgreSQL if we have it configured in our
- configuration file.
+ """
+ if 'ssl_options' in config['HTTPServer']:
+ self._fixup_ssl_config(config['HTTPServer']['ssl_options'])
+
+ # Set the debug to True if running in the foreground
+ if self._debug and not config['Application'].get('debug'):
+ config['Application']['debug'] = True
+
+ def _fixup_ssl_config(self, config):
+ """Check the config to see if SSL configuration options have been passed
+ and replace none, option, and required with the correct values in
+ the certreqs attribute if it is specified.
+
+ :param dict config: the HTTPServer > ssl_options configuration dict
"""
- # Import Redis only if we need it
- from clients import pgsql
+ if 'cert_reqs' in config:
- # Get a Redis specific config node
- config = self._config['Postgres']
+ # Build a mapping dictionary
+ requirements = {'none': ssl.CERT_NONE,
+ 'optional': ssl.CERT_OPTIONAL,
+ 'required': ssl.CERT_REQUIRED}
+ # Remap the value
+ config['cert_reqs'] = requirements[config['cert_reqs']]
- # Create our Redis instance, it will auto-connect and setup
- conn = pgsql.PgSQL(config.get('host'),
- config.get('port'),
- config.get('dbname'),
- config.get('user'),
- config.get('password'))
+ def _get_application_config(self):
+ """Return the Application configuration
- # Add it to our tinman attribute at the application scope
- self._app.tinman.add('pgsql', conn)
+ :rtype: dict
- def _build_rabbitmq_connection(self):
- """Create a connection to RabbitMQ if we have it configured in our
- configuration file.
+ """
+ return self._config['Application']
+
+ def _get_handlers(self):
+ """Return the dictionary of URI to Handler mappings, providing
+ instances of the handlers instead of the classes.
+
+ :rtype: dict
"""
- # Import RabbitMQ only if we need it
- from clients import rabbitmq
+ routes = self._get_routes_config()
- # Get a RabbitMQ specific config node
- config = self._config['RabbitMQ']
- # Create our RabbitMQ instance, it will auto-connect and setup based on
- # this
- rabbitmq = rabbitmq.RabbitMQ(config.get('host'),
- config.get('port'),
- config.get('virtual_host'),
- config.get('username'),
- config.get('password'))
- # Add it to our tinman attribute at the application scope
- self._app.tinman.add('rabbitmq', rabbitmq)
+ def _get_http_server_config(self):
+ """Return the HTTPServer configuration
- def _build_redis_connection(self):
- """Create a connection to Redis if we have it configured in our
- configuration file.
+ :rtype: dict
"""
- # Import Redis only if we need it
- from clients import redis
+ return self._config['HTTPServer']
- # Get a Redis specific config node
- config = self._config['Redis']
+ def _get_postgres_config(self):
+ """Return the PostgreSQL configuration if it exists
- # Create our Redis instance, it will auto-connect and setup
- redis = redis.Redis(config.get('host'),
- config.get('port'),
- config.get('db'),
- config.get('password'))
+ :rtype: dict
+
+ """
+ return self._config.get('PostgreSQL')
+
+ def _get_rabbitmq_config(self):
+ """Return the RabbitMQ configuration if it exists
+
+ :rtype: dict
+
+ """
+ return self._config.get('RabbitMQ')
+
+ def _get_redis_config(self):
+ """Return the RabbitMQ configuration if it exists
+
+ :rtype: dict
+
+ """
+ return self._config.get('RabbitMQ')
+
+ def _get_routes(self):
+ """Return the route list from the configuration.
+
+ :rtype: list
+
+ """
+ return self._config['Routes']
- # Add it to our tinman attribute at the application scope
- self._app.tinman.add('redis', redis)
+ def _http_server(self, config):
+ """Setup the HTTPServer
- def _shutdown_signal_handler(self, signum, frame):
- """Called on SIGTERM to shutdown the sub-process"""
- if self._ioloop:
- self._ioloop.stop()
+ :rtype: tornado.httpserver.HTTPServer
- @property
- def _http_server_arguments(self):
+ """
+ logger.debug('Returning a HTTPServer with %r', config)
+ return self._start_httpserver(self._port,
+ self._http_server_arguments(config))
+
+ def _http_server_arguments(self, config):
"""Return a dictionary of HTTPServer arguments using the default values
as specified in the HTTPServer class docstrings if no values are
specified.
- :returns: dict
+ :param dict config: The HTTPServer specific section of the config
+ :rtype: dict
+
+ """
+ return {'no_keep_alive': config.get('no_keep_alive', False),
+ 'ssl_options': config.get('ssl_options'),
+ 'xheaders': config.get('xheaders', False)}
+
+ def _setup_services(self):
+ """Create an instance for each of the configured auto-configured
+ services.
+
+ """
+
+
+ def _setup_signal_handlers(self):
+ """Called when a child process is spawned to register the signal
+ handlers
+
+ """
+ logger.debug('Registering signal handlers')
+ signal.signal(signal.SIGTERM, self.on_sigterm)
+
+ def _setup_postgresql(self):
+ """If a PostgreSQL instance is configured, create a new PostgreSQL
+ connection and cursor.
+
+ """
+ config = self._get_postgres_config()
+ if not config:
+ return None
+
+ logger.debug('Constructing PostgreSQL Connection')
+ from tinman.clients import pgsql
+
+ return pgsql.PgSQL(config.get('host'),
+ config.get('port'),
+ config.get('dbname'),
+ config.get('user'),
+ config.get('password'))
+
+ def _setup_rabbitmq_connection(self):
+ """Create a connection to RabbitMQ if we have it configured in our
+ configuration file.
"""
- # No reason to not always pass these
- args = {'no_keep_alive': self._config['HTTPServer'].get('no_keep_alive',
- False),
- 'xheaders': self._config['HTTPServer'].get('xheaders',
- False)}
+ config = self._get_redis_config()
+ if not config:
+ return None
- # Only pass ssl_options if we have it set in the config
- if 'ssl_options' in self._config['HTTPServer']:
- args['ssl_options'] = self._config['HTTPServer'].get('ssl_options',
- dict())
- # Return the arguments
- return args
+ # Import RabbitMQ only if we need it
+ from clients import rabbitmq
+
+ # Create the connected RabbitMQ instance
+ return rabbitmq.RabbitMQ(config.get('host'),
+ config.get('port'),
+ config.get('virtual_host'),
+ config.get('username'),
+ config.get('password'))
- def _setup_logging(self, config, debug):
- """Construct the logging config object and
+ def _setup_redis_connection(self):
+ """Create a connection to Redis if we have it configured in our
+ configuration file.
"""
- self._logging = logging_config.Logging(config, debug)
- self._logging.setup()
+ config = self._get_redis_config()
+ if not config:
+ return None
+
+ # Import Redis only if we need it
+ from clients import redis
+
+ # Create our Redis instance, it will auto-connect and setup
+ return redis.Redis(config.get('host'),
+ config.get('port'),
+ config.get('db'),
+ config.get('password'))
def _start_httpserver(self, port, args):
"""Start the HTTPServer
:param int port: The port to run the HTTPServer on
:param dict args: Dictionary of arguments for HTTPServer
+ :rtype: tornado.httpserver.HTTPServer
"""
# Start the HTTP Server
- self._logger.info("Starting Tornado v%s HTTPServer on port %i",
- tornado_version, port)
- http_server = httpserver.HTTPServer(self._app, **args)
+ logger.info("Starting Tornado v%s HTTPServer on port %i",
+ tornado_version, port)
+ http_server = httpserver.HTTPServer(self._application,
+ **args)
try:
http_server.listen(port)
except socket.error as error:
@@ -181,59 +260,42 @@ def _start_httpserver(self, port, args):
# Patch in the HTTP Port for Logging
self._app.http_port = port
- def _subprocess_start(self, config, port):
- """Start the process specific application and HTTP server for the given
- config and port
+ return http_server
- :param dict config: The configuration dictionary parsed by Tinman
- :param int port: The port to listen on
+ def on_sigterm(self, signal, frame):
+ logger.debug('Child process received SIGTERM')
+
+ # Stop the IOLoop
+ self._ioloop.stop()
+
+ def run(self):
+ """Called when the process has started
+
+ :param int port: The HTTP Server port
"""
- # Setup our signal handler
- signal.signal(signal.SIGTERM, self._shutdown_signal_handler)
+ logger.debug('Initializing process')
- # Start our application
- self._app = application.TinmanApplication(config.get('Routes', None),
- **config.get('Application',
- dict()))
+ # Now in a child process so setup logging for this process
+ clihelper.setup_logging(self._debug)
- # Try and build any connection types we automatically support
- self._build_connections()
+ # Register the signal handlers
+ self._setup_signal_handlers()
- # Build a dictionary of valid HTTP Server arguments
- args = self._http_server_arguments
+ # Create the application instance
+ self._app = self._application(self._get_application_config())
- # Start the HTTP Server
- self._start_httpserver(port, args)
+ # Setup the auto-created IO services
+ self._setup_services()
+
+ # Create the HTTPServer
+ self._server = self._http_server(self._get_http_server_config())
- # Get a handle to the instance of IOLoop
+ # Hold on to the IOLoop in case it's needed for responding to signals
self._ioloop = ioloop.IOLoop.instance()
- # Start the IOLoop
+ # Start the IOLoop, blocking until it is stopped
try:
self._ioloop.start()
except KeyboardInterrupt:
- self._logger.info('KeyboardInterrupt received, shutting down.')
-
- def start(self, config):
- """Start the TinmanProcess for the given config. This will in turn
- spawn a new process for each port of the HTTP server and then move on.
-
- :param dict config: The configuration dictionary parsed by Tinman
-
- """
- # Loop through and kick off our processes
- self._children = []
- for port in config['HTTPServer']['ports']:
- self._logger.info("Starting Tinman v%s process for port %i",
- __version__, port)
-
- # Kick off the child process
- child = multiprocessing.Process(target=self._subprocess_start,
- name="tinman-%i" % port,
- args=(config, port))
- self._children.append(child)
- child.start()
-
- # Log our completion
- self._logger.debug("%i child(ren) spawned", len(self._children))
+ pass
View
204 tinman/utils.py
@@ -1,193 +1,37 @@
-# -*- coding: utf-8 -*-
"""
-Functions used mainly in startup and shutdown of tornado applications
+@TODO see if we can move these functions to a more appropriate spot
+
"""
-import logging
import os
-import os.path
-import signal
import sys
-import yaml
-
-# Windows doesn't support these
-try:
- import grp
-except ImportError:
- grp = None
-try:
- import pwd
-except ImportError:
- pwd = None
-
-from functools import wraps
from socket import gethostname
-# Callback handlers
-rehash_handler = None
-shutdown_handler = None
-# Application state for shutdown
-running = False
-# Get a _LOGGER for the module
-_LOGGER = logging.getLogger(__name__)
+def application_name():
+ """Returns the currently running application name
+ :rtype: str
-def application_name():
- """
- Returns the currently running application name
"""
return os.path.split(sys.argv[0])[1]
def hostname():
- """
- Returns the hostname for the machine we're running on
- """
- return gethostname().split(".")[0]
-
-
-def daemonize(pidfile=None, user=None, group=None):
- """
- Fork the Python app into the background and close the appropriate
- "files" to detach from console. Based off of code by Jürgen Hermann and
- http://code.activestate.com/recipes/66012/
+ """Returns the hostname for the machine we're running on
- Parameters:
+ :rtype: str
- * pidfile: Pass in a file to write the pid, defaults to
- /tmp/current_process_name-pid_number.pid
- * user: User to run as, defaults to current user
- * group: Group to run as, defaults to current group
"""
- # Flush stdout and stderr
- sys.stdout.flush()
- sys.stderr.flush()
-
- # Set our default uid, gid
- uid, gid = -1, -1
-
- # Get the user id if we have a user set
- if pwd and user:
- uid = pwd.getpwnam(user).pw_uid
-
- # Get the group id if we have a group set
- if grp and group:
- gid = grp.getgrnam(group).gr_gid
-
- # Fork off from the process that called us
- pid = os.fork()
- if pid > 0:
- sys.exit(0)
-
- # Second fork to put into daemon mode
- pid = os.fork()
- if pid > 0:
- # exit from second parent, print eventual PID before
- sys.stdout.write('%s: started - PID # %d\n' % (application_name(),
- pid))
-
- # Setup a pidfile if we weren't passed one
- pidfile = pidfile or \
- os.path.normpath('/tmp/%s-%i.pid' % (application_name(),
- pid))
-
- # Write a pidfile out
- with open(pidfile, 'w') as f:
- f.write('%i\n' % pid)
-
- # If we have uid or gid change the uid/gid for the file
- if uid > -1 or gid > -1:
- os.fchown(f.fileno(), uid, gid)
-
- # Exit the parent process
- sys.exit(0)
-
- # Detach from parent environment
- os.chdir(os.path.normpath('/'))
- os.umask(0)
- os.setsid()
-
- # Redirect stdout, stderr, stdin
- si = file('/dev/null', 'r')
- so = file('/dev/null', 'a+')
- se = file('/dev/null', 'a+', 0)
- os.dup2(si.fileno(), sys.stdin.fileno())
- os.dup2(so.fileno(), sys.stdout.fileno())
- os.dup2(se.fileno(), sys.stderr.fileno())
-
- # Set the running user
- if user and group:
- _LOGGER.info("Changing the running user:group to %s:%s", user, group)
- elif user:
- _LOGGER.info("Changing the running user to %s", user)
- elif group:
- _LOGGER.info("Changing the group to %s", group)
-
- # If we have a uid and it's not for the running user
- if uid > -1 and uid != os.geteuid():
- try:
- os.seteuid(uid)
- _LOGGER.debug("User changed to %s(%i)", user, uid)
- except OSError as e:
- _LOGGER.error("Could not set the user: %s", str(e))
-
- # if we have a gid and it's not for the current group
- if gid > -1 and gid != os.getegid():
- try:
- os.setgid(gid)
- _LOGGER.debug("Process group changed to %s(%i)", group, gid)
- except OSError as e:
- _LOGGER.error("Could not set the group: %s", str(e))
-
- return True
-
-
-def shutdown():
- """
- Cleanly shutdown the application
- """
- global running
-
- # Tell all our children to stop
- if shutdown_handler:
- shutdown_handler()
-
- # Set the running state
- running = False
-
-
-def setup_signals():
- """
- Setup the signals we want to be notified on
- """
- signal.signal(signal.SIGTERM, _shutdown_signal_handler)
- try:
- signal.signal(signal.SIGHUP, _rehash_signal_handler)
- except AttributeError:
- pass
-
-def _shutdown_signal_handler(signum, frame):
- """
- Called on SIGTERM to shutdown the application
- """
- _LOGGER.info("SIGTERM received, shutting down")
- shutdown()
-
-
-def _rehash_signal_handler(signum, frame):
- """
- Would be cool to handle this and effect changes in the config
- """
- _LOGGER.info("SIGHUP received, rehashing config")
- if rehash_handler:
- rehash_handler()
+ return gethostname().split(".")[0]
def import_namespaced_class(path):
- """
- Pass in a string in the format of foo.Bar, foo.bar.Baz, foo.bar.baz.Qux
+ """Pass in a string in the format of foo.Bar, foo.bar.Baz, foo.bar.baz.Qux
and it will return a handle to the class
+
+ :rtype: class
+
"""
# Split up our string containing the import and class
parts = path.split('.')
@@ -202,27 +46,3 @@ def import_namespaced_class(path):
# Return the class handle
return class_handle
-
-
-def load_yaml_file(config_file):
- """ Load the YAML configuration file from disk or error out
- if not found or parsable.
-
- :param str config_file: Full path to the filename
- :returns: dict
-
- """
- try:
- with file(config_file, 'r') as handle:
- config = yaml.load(handle)
-
- except IOError:
- sys.stderr.write('Configuration file not found "%s"\n' % config_file)
- sys.exit(1)
-
- except yaml.scanner.ScannerError as err:
- sys.stderr.write('Invalid configuration file "%s":\n%s\n' %\
- (config_file, err))
- sys.exit(1)
-
- return config
View
4 tinman/whitelist.py
@@ -1,16 +1,16 @@
"""
Tinman Whitelist Module
+
"""
__author__ = "Gavin M. Roy"
__email__ = "gmr@myyearbook.com"
__date__ = "2011-03-13"
-__version__ = 0.1
-from functools import wraps
from ipaddr import IPv4Network, IPv4Address
from tornado.web import HTTPError
from types import FunctionType
+
def whitelisted(argument=None):
"""
Decorates a method requiring that the requesting IP address is whitelisted
Please sign in to comment.
Something went wrong with that request. Please try again.