make tests build databases on the fly (geopython#195)

docs/testing.rst

@@ -15,29 +15,29 @@ Compliance benchmarking is done via the OGC `Compliance & Interoperability Testi
 Tester
 ------
 
-The pycsw tests framework (in ``tests``) is a collection of testsuites to perform automated regession testing of the codebase.
+The pycsw tests framework (in ``tests``) is a collection of testsuites to perform automated regession testing of the codebase.  Test are run against all pushes to the GitHub repository via `Travis CI`_.
 
-Running
-^^^^^^^
+Running Locally
+^^^^^^^^^^^^^^^
 
-The tests framework can be run from ``tests``:
+The tests framework can be run from ``tests`` using Pavement (see ``pavement.py``) tasks for convenience:
 
 .. code-block:: bash
 
   $ cd /path/to/pycsw
-  $ cd tests
-  # run all tests against http://localhost:8000/
-  $ python ./run_tests.py -u http://localhost:8000/
-  # run only specific testsuites against http://localhost:8000/
-  $ python ./run_tests.py -u http://localhost:8000/ -s apiso,fgdc
-  # lots of output
+  # run all tests (starts up http://localhost:8000)
+  $ paver test
+  # run tests only against specific testsuites 
+  $ paver test -s apiso,fgdc
+  # run all tests, including harvesting (this is turned off by default given the volatility of remote services/data testing)
+  $ paver test -r
 
-The tests runs HTTP GET and POST requests.  The expected output for each test can be found in ``expected``.  Results are categorized as ``passed``, ``failed``, or ``initialized``.  A summary of results is output at the end of the run.
+The tests perform HTTP GET and POST requests against ``http://localhost:8000``.  The expected output for each test can be found in ``expected``.  Results are categorized as ``passed``, ``failed``, or ``initialized``.  A summary of results is output at the end of the run.
 
 Failed Tests
 ^^^^^^^^^^^^
 
-If a given test has failed, the output is saved in ``results``.  The resulting failure can be analyzed by running ``diff expected/name_of_test.xml results/name_of_test.xml`` to find variances.
+If a given test has failed, the output is saved in ``results``.  The resulting failure can be analyzed by running ``diff tests/expected/name_of_test.xml tests/results/name_of_test.xml`` to find variances.  The Pavement task returns a status code which indicates the number of tests which have failed (i.e. ``echo $?``).
 
 Test Suites
 ^^^^^^^^^^^
@@ -47,6 +47,7 @@ The tests framework is run against a series of 'suites' (in ``tests/suites``), e
 * ``tests/suites/suite/default.cfg``: the configuration for the suite
 * ``tests/suites/suite/post``: directory of XML documents for HTTP POST requests
 * ``tests/suites/suite/get/requests.txt``: directory and text file of KVP for HTTP GET requests
+* ``tests/suites/suite/data``: directory of sample XML data required for the test suite.  Database and test data are setup/loaded automatically as part of testing
 
 When the tests are invoked, the following operations are run:
 
@@ -63,15 +64,22 @@ To add tests to an existing suite:
 
 * for HTTP POST tests, add XML documents to ``tests/suites/suite/post``
 * for HTTP GET tests, add tests (one per line) to ``tests/suites/suite/get/requests.txt``
-* run ``python ./run_tests.py -u <url>``
+* run ``paver test``
 
 To add a new test suite:
 
 * create a new directory under ``tests/suites`` (e.g. ``foo``)
-* create a new configuration in ``tests/suites/foo/default.cfg``.  Ensure that all file paths are relative to ``path/to/pycsw``
+* create a new configuration in ``tests/suites/foo/default.cfg``
+
+  * Ensure that all file paths are relative to ``path/to/pycsw``
+  * Ensure that ``repository.database`` points to an SQLite3 database called ``tests/suites/foo/data/records.db``.  The database *must* be called ``records.db`` and the directory ``tests/suites/foo/data`` *must* exist
+
 * populate HTTP POST requests in ``tests/suites/foo/post``
 * populate HTTP GET requests in ``tests/suites/foo/get/requests.txt``
-* run ``python ./run_tests.py -u <url>``
+* if the testsuite requires test data, create ``tests/suites/foo/data`` are store XML file there
+* run ``paver test`` (or ``paver test -s foo`` to test only the new test suite)
+
+The new test suite database will be created automatically and used as part of tests.
 
 Web Testing
 ^^^^^^^^^^^
@@ -85,3 +93,4 @@ You can also use the pycsw tests via your web browser to perform sample requests
 Then navigate to ``http://host/path/to/pycsw/tests/index.html``.
 
 .. _`Compliance & Interoperability Testing & Evaluation Initiative`: http://cite.opengeospatial.org/
+.. _`Travis CI`: http://travis-ci.org/geopython/pycsw

pavement.py

@@ -143,6 +143,27 @@ def package_tar_gz(options):
         tar.add(package_name)
         tar.close()
 
+@task
+def setup_testdata():
+    """Create test databases and load test data"""
+
+    test_database_parameters = {
+        # suite: has_testdata
+        'apiso': True,
+        'cite': True,
+        'harvesting': False,
+        'manager': False
+    }
+
+    for database, has_testdata in test_database_parameters.iteritems():
+        info('Setting up test database %s' % database)
+        cfg = path('tests/suites/%s/default.cfg' % database)
+        sh('pycsw-admin.py -c setup_db -f %s' % cfg)
+        if has_testdata:
+            datapath = 'tests/suites/%s/data' % database
+            info('Loading test data from %s' % datapath)
+            sh('pycsw-admin.py -c load_records -f %s -p %s' % (cfg, datapath))
+
 
 @task
 @cmdopts([
@@ -169,6 +190,7 @@ def test(options):
         # run against default server
         call_task('stop')
         call_task('reset')
+        call_task('setup_testdata')
         call_task('start')
         url = 'http://localhost:8000'
 
@@ -272,8 +294,7 @@ def stop():
 ])
 def reset(options):
     """Return codebase to pristine state"""
-    sh('git checkout tests/suites/manager/data/records.db')
-    sh('git checkout tests/suites/harvesting/data/records.db')
+    sh('rm -f tests/suites/*/data/records.db')
 
     force = options.get('force')
     if force:

tests/suites/apiso/default.cfg

@@ -73,7 +73,7 @@ contact_role=pointOfContact
 
 [repository]
 # sqlite
-database=sqlite:///tests/suites/apiso/data/data.db
+database=sqlite:///tests/suites/apiso/data/records.db
 # postgres
 #database=postgres://username:password@localhost/pycsw
 table=records