Permalink
Browse files

update nomad with possibility to try multiple urls

  • Loading branch information...
1 parent f10a0d1 commit e294cb5463ad333c166af2fe232f4e90fb42a386 @piranha committed Sep 3, 2013
Showing with 154 additions and 60 deletions.
  1. +77 −21 README.rst
  2. +2 −2 nomad/__init__.py
  3. +18 −6 nomad/repo.py
  4. +23 −14 nomad/utils.py
  5. +1 −1 setup.py
  6. +33 −16 tests/urls.t
View
98 README.rst
@@ -17,15 +17,15 @@ migrate and can run pre- and post-processing routines written in any language
.. begin-writeup
-Concept
+Layout
-------
-Nomad's migration store is a directory with ``nomad.ini`` and a other
-directories inside. Each directory in it containing ``migration.ini`` is a
-single migration and name of this child directory is an unique identifier of a
+Nomad's migration store is a directory with ``nomad.ini`` and directories with
+migrations inside. Each such directory must contain ``migration.ini`` to be
+recognized as a migration and this directory name is an unique identifier of a
migration.
-It looks like this::
+Your directory tree thus will look like this::
migrations/
nomad.ini
@@ -38,37 +38,93 @@ It looks like this::
2-up.sql
3-post.py
-And a ``nomad.ini`` could look like this::
+Nomad uses table called ``nomad`` to track what was applied already. It's just a
+list of applied migrations and dates when they were applied.
+
+Usage
+-----
+
+To start working, create ``nomad.ini``, and initialize your database (I assume
+it already exists)::
+
+ $ nomad init
+
+Then you can start creating your first migration::
+
+ $ nomad create 0-initial
+
+Put your ALTERs and CREATEs in ``0-initial/up.sql`` and apply a migration::
+
+ $ nomad apply -a # or nomad apply 0-initial
+
+Nomad should report which migrations it applied successfully, but you can check
+status of that with ``nomad ls -a`` (or ``nomad ls`` to see only unapplied
+migrations).
+
+I guess it's time to create new migration:
+
+ $ nomad create 1-next -d 0-initial
+
+``-d 0-initial`` means you want your ``1-next`` to depend on ``0-initial``. This
+means nomad will never apply ``1-next`` without applying ``0-initial``
+first. You usually want to depend on migrations which created tables you're
+going to alter, or just to make it easier - on the latest available migration.
+
+Configuration
+-------------
+
+Nomad reads configuration from ``nomad.ini``, here is an example::
[nomad]
engine = sqla
url = pgsql://user:password@host:port/db
-Possible options for ``engine``:
+Possible configuration options:
+
+- ``engine`` (required) - SQL engine to use, possible options:
+ - ``sqla`` - use SQLAlchemy as an adapter, supports everything SQLAlchemy supports
+ - ``dbapi`` - use regular DB API, supports ``sqlite``, ``mysql`` and ``pgsql``
+- ``url`` (required) - URL to database, takes multiple options, see format below
+- ``path`` - path to migrations (default: directory with ``nomad.ini``)
+
+Each migration has its own ``migration.ini`` file, which at the moment has
+single configuration option, ``nomad.dependencies``, defining which migration
+(or migrations) this one depends.
+
+Note that ini-files are parsed with extended interpolation (use it like
+``${var}`` or ``${section.var}``), two predefined variables are provided:
+
+- ``confpath`` - path to ``nomad.ini``
+- ``confdir`` - path to directory, containing ``nomad.ini``
+- ``dir`` (migration only) - path to directory of migration
+
+URL format
+~~~~~~~~~~
-- ``sqla`` - use SQLAlchemy as an adapter, supports everything SQLAlchemy supports
-- ``dbapi`` - use regular DB API, supports ``sqlite``, ``mysql`` and ``pgsql``
+Nomad can read connection url to database in a few various ways. ``nomad.url``
+configuration option is a space separated list of descriptions of how nomad can
+obtain database connection url.
-``url`` can be defined in a few various ways:
+The easiest one is simply an url (like in config example). The others are:
-- ``url = <your-url-to-db>`` - just a static connection url
-- ``url-file = <path-to-file>`` - a path to file containing connection url
-- ``url-python = <python.mod>:<variable.name>`` - a Python path to a module,
+- ``file:<path-to-file>`` - a path to file containing connection url
+- ``py:<python.mod>:<variable.name>`` - a Python path to a module,
containing a variable with connection url
-- ``url-command = <cmd-to-execute>`` - command line to execute to get connection
+- ``cmd:<cmd-to-execute>`` - command line to execute to get connection
url
-- ``url-json = <path-to-file>:key.0.key`` - path to file with JSON and then path
+- ``json:<path-to-file>:key.0.key`` - path to file with JSON and then path
to a connection url within JSON object
-- ``url-ini = <path-to-file>:<section.key>`` - path to INI file (parsed by
+- ``ini:<path-to-file>:<section.key>`` - path to INI file (parsed by
configparser with extended interpolation) and then path to a connection url
within this file
-Note that ``nomad.ini`` is parsed with extended interpolation (use it like
-``${var}`` or ``${section.var}``), and provides two predefined variables:
-
-- ``confpath`` - path to ``nomad.ini``
-- ``confdir`` - path to directory, containing ``nomad.ini``
+An example::
+ [nomad]
+ url =
+ ini:${confdir}/../settings.ini:db.url
+ json:${confdir}/../settings.json:db.url
+ sqlite:///${confdir}/../local.db
Main properties
---------------
View
4 nomad/__init__.py
@@ -30,8 +30,8 @@ def inner(*args, **kwargs):
try:
repo = Repository(kwargs['config'], kwargs['define'])
except NomadIniNotFound, e:
- sys.stderr.write('Create %r to use nomad, example:\n%s\n' %
- (e.message, EXAMPLE_INI))
+ sys.stderr.write("Create '%s' to use nomad, example:\n%s\n" %
+ (e, EXAMPLE_INI))
abort('config file not found')
except (IOError, NomadError), e:
abort(e)
View
24 nomad/repo.py
@@ -5,7 +5,7 @@
from functools import wraps
from nomad.utils import (cachedproperty, geturl, NomadError, NomadIniNotFound,
- clean_sql)
+ clean_sql, abort)
def tx(getrepo):
@@ -30,11 +30,12 @@ class Repository(object):
}
def __init__(self, confpath, overrides=None):
- self.conf = ConfigParser(interpolation=ExtendedInterpolation(),
- defaults={
+ self.conf = ConfigParser(
+ interpolation=ExtendedInterpolation(),
+ defaults={
'confpath': op.abspath(confpath),
'confdir': op.dirname(op.abspath(confpath)),
- })
+ })
self.conf.read_dict(self.DEFAULTS)
if not self.conf.read([confpath]):
raise NomadIniNotFound(confpath)
@@ -43,6 +44,7 @@ def __init__(self, confpath, overrides=None):
section, key = k.split('.')
self.conf.set(section, key, v)
+ self.confpath = confpath
self.path = self.conf.get('nomad', 'path',
fallback=op.dirname(confpath) or '.')
@@ -56,7 +58,11 @@ def __init__(self, confpath, overrides=None):
enginemod = __import__(enginepath, {}, {}, [''])
except ImportError, e:
raise NomadError('cannot use engine %s: %s' % (enginepath, e))
- self.engine = getattr(enginemod, 'engine')(geturl(self))
+ try:
+ url = geturl(self.conf['nomad']['url'])
+ except KeyError:
+ abort('database url in %s is not found' % self)
+ self.engine = getattr(enginemod, 'engine')(url)
def __repr__(self):
return '<%s: %s>' % (type(self).__name__, self.path)
@@ -101,7 +107,13 @@ def __init__(self, repo, name, force=False, applied=False):
self.name = name
if not op.exists(op.join(repo.path, name)) and not force:
raise NomadError('migration not found')
- self.conf = ConfigParser(interpolation=ExtendedInterpolation())
+ self.conf = ConfigParser(
+ interpolation=ExtendedInterpolation(),
+ defaults={
+ 'confpath': op.abspath(self.repo.confpath),
+ 'confdir': op.dirname(op.abspath(self.repo.confpath)),
+ 'dir': op.abspath(op.join(repo.path, name))
+ })
self.conf.read([op.join(repo.path, name, 'migration.ini')])
self._deps = [x.strip() for x in self.conf.get('nomad', 'dependencies',
fallback='').split(',')
View
37 nomad/utils.py
@@ -46,6 +46,12 @@ def abort(msg, code=1):
sys.exit(code)
+def shsplit(s):
+ if not isinstance(s, str):
+ s = s.encode('utf-8')
+ return shlex.split(s)
+
+
def humankey(fn):
'''Turn a string into a list of substrings and numbers.
@@ -127,24 +133,27 @@ def get_ini(path):
URLTYPES = {
- 'url': lambda url: url,
- 'url-python': get_python,
- 'url-file': get_file,
- 'url-command': get_command,
- 'url-json': get_json,
- 'url-ini': get_ini,
+ 'python': get_python,
+ 'py': get_python,
+ 'file': get_file,
+ 'command': get_command,
+ 'cmd': get_command,
+ 'json': get_json,
+ 'ini': get_ini,
}
-def geturl(repo):
- conf = repo.conf['nomad']
- for k, fn in URLTYPES.iteritems():
- if k in conf:
+def geturl(urlspec):
+ for url in shsplit(urlspec):
+ bits = url.split(':', 1)
+ if len(bits) > 1 and bits[0] in URLTYPES:
try:
- return fn(conf[k])
- except (IOError, OSError, KeyError), e:
- abort(e)
- abort('database url in %s is not found' % repo)
+ return URLTYPES[bits[0]](bits[1])
+ except (IOError, OSError, KeyError):
+ pass
+ else:
+ return url
+ abort("could not read database url from spec '%s'" % urlspec)
if __name__ == '__main__':
View
2 setup.py
@@ -25,7 +25,7 @@ def read(fname):
description = 'simple sql migration tool to save you from becoming mad',
long_description = read('README.rst'),
license = 'BSD',
- version = '0.5',
+ version = '1.0',
author = 'Alexander Solovyov',
author_email = 'alexander@solovyov.net',
url = 'http://github.com/piranha/nomad/',
View
49 tests/urls.t
@@ -33,7 +33,7 @@ URL from Python object from ``sys.path``::
$ cat > nomad.ini <<EOF
> [nomad]
> engine = sqla
- > url-python = somemod:dburl
+ > url = python:somemod:dburl
> EOF
$ PYTHONPATH=.:$PYTHONPATH $NOMAD info
<Repository: .>:
@@ -46,7 +46,7 @@ URL from Python object using path::
$ cat > nomad.ini <<EOF
> [nomad]
> engine = sqla
- > url-python = \${confdir}/somemod.py:dburl
+ > url = python:\${confdir}/somemod.py:dburl
> EOF
$ $NOMAD info
<Repository: .>:
@@ -61,7 +61,7 @@ URL from Python package::
$ cat > nomad.ini <<EOF
> [nomad]
> engine = sqla
- > url-python = \${confdir}/package:dburl
+ > url = python:\${confdir}/package:dburl
> EOF
$ $NOMAD info
<Repository: .>:
@@ -75,7 +75,7 @@ URL from a file::
$ cat > nomad.ini <<EOF
> [nomad]
> engine = sqla
- > url-file = url
+ > url = file:url
> EOF
$ $NOMAD info
<Repository: .>:
@@ -85,47 +85,64 @@ URL from a file::
URL from a command::
- $ echo 'sqlite:///test-file.db' > url
+ $ echo 'sqlite:///test-cmd.db' > url
$ cat > nomad.ini <<EOF
> [nomad]
> engine = sqla
- > url-command = cat url
+ > url = command:"cat url"
> EOF
$ $NOMAD info
<Repository: .>:
- <SAEngine: sqlite:///test-file.db>
+ <SAEngine: sqlite:///test-cmd.db>
Uninitialized repository
URL from JSON file::
- $ echo '{"db": [{"url": "sqlite:///test-file.db"}]}' > url.json
- $ cat > nomad.init <<EOF
+ $ echo '{"db": [{"url": "sqlite:///test-json.db"}]}' > url.json
+ $ cat > nomad.ini <<EOF
> [nomad]
> engine = sqla
- > url-json = url.json:db.0.url
+ > url = json:url.json:db.0.url
> EOF
$ $NOMAD info
<Repository: .>:
- <SAEngine: sqlite:///test-file.db>
+ <SAEngine: sqlite:///test-json.db>
Uninitialized repository
URL from INI file::
- $ echo '[db]\nurl = sqlite:///test-file.db' > url.ini
- $ cat > nomad.init <<EOF
+ $ echo '[db]\nurl = sqlite:///test-ini.db' > url.ini
+ $ cat > nomad.ini <<EOF
> [nomad]
> engine = sqla
- > url-ini = url.ini:db.url
+ > url = ini:url.ini:db.url
> EOF
$ $NOMAD info
<Repository: .>:
- <SAEngine: sqlite:///test-file.db>
+ <SAEngine: sqlite:///test-ini.db>
Uninitialized repository
-
Nothing defined::
$ echo '[nomad]\nengine=sqla' > nomad.ini
$ $NOMAD info
\x1b[31mError: database url in <Repository: .> is not found\x1b[0m (esc)
[1]
+
+MultiURL::
+
+ $ rm url.ini
+ $ cat > nomad.ini <<EOF
+ > [nomad]
+ > engine = sqla
+ > url = ini:url.ini:db.url sqlite:///test.db
+ > EOF
+ $ $NOMAD info
+ <Repository: .>:
+ <SAEngine: sqlite:///test.db>
+ Uninitialized repository
+ $ echo '[db]\nurl = sqlite:///test-multi.db' > url.ini
+ $ $NOMAD info
+ <Repository: .>:
+ <SAEngine: sqlite:///test-multi.db>
+ Uninitialized repository

0 comments on commit e294cb5

Please sign in to comment.