Skip to content
This repository has been archived by the owner on Nov 9, 2023. It is now read-only.

Commit

Permalink
update script usage testing docs
Browse files Browse the repository at this point in the history
now that we don't have output files/dirs included under qiime_test_data/
  • Loading branch information
jairideout committed Dec 12, 2013
1 parent 485c649 commit de7b2a6
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 18 deletions.
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Original file line Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ git checkout -b my-topic-branch


7. Make your changes, add them (with ``git add``), and commit them (with ``git commit``). Don't forget to update associated scripts and tests as necessary. You should make incremental commits, rather than one massive commit at the end. Write descriptive commit messages to accompany each commit. 7. Make your changes, add them (with ``git add``), and commit them (with ``git commit``). Don't forget to update associated scripts and tests as necessary. You should make incremental commits, rather than one massive commit at the end. Write descriptive commit messages to accompany each commit.


8. When you think you're ready to submit your code, again ensure that you have the latest version of all files in case some changed while you were working on your edits. You can do this by merging master into your topic branch:: 8. When you think you're ready to submit your code, again ensure that you have the latest version of all files in case some changed while you were working on your edits. You can do this by merging master into your topic branch:
``` ```
git checkout my-topic-branch git checkout my-topic-branch
git pull upstream master git pull upstream master
Expand Down
23 changes: 6 additions & 17 deletions doc/developer/script_testing.rst
Original file line number Original file line Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Automated script testing


QIIME is well-tested: we make automated software testing a high priority, as you can see by reviewing the ``Qiime/tests/`` directory `here <https://github.com/qiime/qiime/tree/master/tests>`_. After installing QIIME, users can run the automated test suite by changing to the ``Qiime/tests`` directory and running:: QIIME is well-tested: we make automated software testing a high priority, as you can see by reviewing the ``Qiime/tests/`` directory `here <https://github.com/qiime/qiime/tree/master/tests>`_. After installing QIIME, users can run the automated test suite by changing to the ``Qiime/tests`` directory and running::


all_tests.py ./all_tests.py


These tests are additionally run on a nightly basis, and developers are alerted to test failures. These tests are additionally run on a nightly basis, and developers are alerted to test failures.


Expand Down Expand Up @@ -47,11 +47,10 @@ The recommended way of testing QIIME is to run QIIME's unit tests and script usa


How the script usage tests work How the script usage tests work
=============================== ===============================
You'll see many sub-directories in ``qiime_test_data`` with names corresponding to the names of QIIME scripts. Each of these directories contains example input and output for the corresponding QIIME script. For example, the ``make_otu_table`` directory contains the following test data for ``make_otu_table.py``:: You'll see many sub-directories in ``qiime_test_data`` with names corresponding to the names of QIIME scripts. Each of these directories contains example input files for the corresponding QIIME script. For example, the ``make_otu_table`` directory contains the following test data for ``make_otu_table.py``::


ls make_otu_table/ ls make_otu_table/
chimeric_seqs.txt otu_table.biom otu_table_non_chimeric.biom tax_assignments.txt chimeric_seqs.txt otu_map.txt pynast_failures.fna tax_assignments.txt
otu_map.txt otu_table_no_pynast_failures.biom pynast_failures.fna


If you call ``make_otu_table.py -h``, you'll see the following usage examples:: If you call ``make_otu_table.py -h``, you'll see the following usage examples::


Expand All @@ -68,16 +67,16 @@ If you call ``make_otu_table.py -h``, you'll see the following usage examples::
Make OTU table, excluding a list of OTU ids: Make an OTU table, excluding the sequences listed in chimeric_seqs.txt Make OTU table, excluding a list of OTU ids: Make an OTU table, excluding the sequences listed in chimeric_seqs.txt
make_otu_table.py -i otu_map.txt -o otu_table_non_chimeric.biom -e chimeric_seqs.txt make_otu_table.py -i otu_map.txt -o otu_table_non_chimeric.biom -e chimeric_seqs.txt


What you'll notice is that the usage example input and output files correspond to the files in ``qiime/qiime_test_data/make_otu_table``. The script interface testing works by copying all of the files in ``make_otu_table`` to a temporary directory, changing into that directory, running each of the usage examples, and confirming that the script exited successfully (i.e., with an exit status of ``0``). What you'll notice is that the usage example input files correspond to the files in ``qiime/qiime_test_data/make_otu_table``. The script interface testing works by copying all of the files in ``make_otu_table`` to a temporary directory, changing into that directory, running each of the usage examples, and confirming that the script exited successfully (i.e., with an exit status of ``0``).


.. warning:: Currently the script usage tests only test whether a script exits successfully: they do not check whether the results correspond to the example output. The reasoning is that that would duplicate the functionality of the unit tests (which isn't a bad thing, except that implementing this would be a lot of work). These are tests that the interfaces themselves are working. .. warning:: Currently the script usage tests only test whether a script exits successfully: they do not check whether the results correspond to expected output. The reasoning is that that would duplicate the functionality of the unit tests (which isn't a bad thing, except that implementing this would be a lot of work). These are tests that the interfaces themselves are working.


If you don't see a directory corresponding to a script name in the ``qiime_test_data`` directory, that means that a script interface test has not been defined for the given script. We're currently working on extending this framework to cover all QIIME scripts. If you don't see a directory corresponding to a script name in the ``qiime_test_data`` directory, that means that a script interface test has not been defined for the given script. We're currently working on extending this framework to cover all QIIME scripts.


Adding script interface testing for new scripts Adding script interface testing for new scripts
=============================================== ===============================================


Adding new script interface tests is easy. All you do is create a new test directory in ``qiime_test_data``, where the name of the directory corresponds to the script's name. For example, if you're adding tests for ``my_script.py``, you'd add a directory called ``my_script``. In that directory you would create example input and output files for all of the script usage examples that are defined in your script. Make several usage examples that make use of different paths through your script. Adding new script interface tests is easy. All you need to do is create a new test directory in ``qiime_test_data``, where the name of the directory corresponds to the script's name. For example, if you're adding tests for ``my_script.py``, you'd add a directory called ``my_script``. In that directory you would create example input files for all of the script usage examples that are defined in your script. Make several usage examples that make use of different paths through your script.


Full paths Full paths
---------- ----------
Expand All @@ -94,13 +93,3 @@ We recommend specifying full paths for many of QIIME scripts, and importantly fo
subdirectories as appropriate. ALWAYS SPECIFY ABSOLUTE FILE PATHS (absolute path represented subdirectories as appropriate. ALWAYS SPECIFY ABSOLUTE FILE PATHS (absolute path represented
here as $PWD, but will generally look something like /home/ubuntu/my_analysis/). here as $PWD, but will generally look something like /home/ubuntu/my_analysis/).
pick_de_novo_otus.py -i $PWD/seqs.fna -o $PWD/otus/ pick_de_novo_otus.py -i $PWD/seqs.fna -o $PWD/otus/

Cleaning up output files
------------------------
Some scripts require that the user-specified output directory does not exist when the script runs, but we provide example output in the test directory. To automatically remove output directories prior to running the tests, add the ``script_usage_output_to_remove`` entry to your script info. For example, from ``pick_de_novo_otus.py``::

script_info['script_usage_output_to_remove'] = ['$PWD/otus/']




0 comments on commit de7b2a6

Please sign in to comment.