docs/notebooks/3_run_single_workflow_test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Run a single test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "See the folder *example*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "EXAMPLES_FOLDER = \"../examples\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "... where you can find the following files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['change_case',\n",
       " 'fluxomics_stationary',\n",
       " 'multivariate',\n",
       " 'sacurine',\n",
       " 'workflow-test-suite-full.yml',\n",
       " 'workflow-test-suite-min.yml',\n",
       " 'workflow-test-suite.yml',\n",
       " 'workflows.json']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os, pprint\n",
    "[f for f in os.listdir(EXAMPLES_FOLDER) if not f.startswith('.')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Consider the **definition file** ``workflow-test-suite-min.yml`` (steps[3-4]), which contains the two workflow tests named ``change_case`` and ``multivariate`` respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "suite_conf_filename = os.path.join(EXAMPLES_FOLDER, \"workflow-test-suite-min.yml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"enable_logger\": false, \n",
      "    \"workflows\": {\n",
      "        \"change_case\": {\n",
      "            \"expected\": {\n",
      "                \"OutputText\": \"change_case/expected_output\"\n",
      "            }, \n",
      "            \"inputs\": {\n",
      "                \"InputText\": \"change_case/input\"\n",
      "            }, \n",
      "            \"file\": \"change_case/workflow.ga\"\n",
      "        }, \n",
      "        \"multivariate\": {\n",
      "            \"expected\": {\n",
      "                \"variableMetadata_out\": \"multivariate/variableMetadata_out\", \n",
      "                \"sampleMetadata_out\": \"multivariate/sampleMetadata_out\"\n",
      "            }, \n",
      "            \"inputs\": {\n",
      "                \"DataMatrix\": \"multivariate/dataMatrix.tsv\", \n",
      "                \"SampleMetadata\": \"multivariate/sampleMetadata.tsv\", \n",
      "                \"VariableMetadata\": \"multivariate/variableMetadata.tsv\"\n",
      "            }, \n",
      "            \"params\": {\n",
      "                \"3\": {\n",
      "                    \"predI\": \"1\", \n",
      "                    \"respC\": \"gender\", \n",
      "                    \"orthoI\": \"NA\", \n",
      "                    \"testL\": \"FALSE\"\n",
      "                }\n",
      "            }, \n",
      "            \"file\": \"multivariate/workflow.ga\"\n",
      "        }\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import yaml, json\n",
    "with open(suite_conf_filename, \"r\") as fp:\n",
    "    data = yaml.load(fp)\n",
    "    print(json.dumps(data, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Similar to that in respect of test suite (example [2](2_run_test_suite.ipynb)), we need to load the test configuration within a Python object. The class specialized for prepresenting a test configuration in ``wft4galaxy`` is ``wft4galaxy.core.WorkflowTestCase`` (step [5]):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from wft4galaxy.core import WorkflowTestCase"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "We can create the class instance using the its static ``load`` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "test_case = WorkflowTestCase.load(suite_conf_filename, workflow_test_name=\"change_case\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having the ``test_case`` object, we can simply run the test that it represents by calling its ``run`` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2017-03-30 13:50:18,052 [wft4galaxy] [ INFO]  Create a history '_WorkflowTestHistory_0b688d4c-153f-11e7-87e4-a45e60c4fc6b' (id: u'd3a4d6a5256f2d9a')\n",
      "2017-03-30 13:50:19,480 [wft4galaxy] [ INFO]  Workflow '_WorkflowTest_Change Case (imported from API)' (id: d3a4d6a5256f2d9a) running ...\n",
      "2017-03-30 13:50:24,497 [wft4galaxy] [ INFO]  waiting for datasets\n",
      "2017-03-30 13:50:24,777 [wft4galaxy] [ INFO]  f711c56f400864d1: new\n",
      "2017-03-30 13:50:25,697 [wft4galaxy] [ INFO]  f711c56f400864d1: new\n",
      "2017-03-30 13:50:28,410 [wft4galaxy] [ INFO]  f711c56f400864d1: queued\n",
      "2017-03-30 13:50:29,249 [wft4galaxy] [ INFO]  f711c56f400864d1: ok\n",
      "2017-03-30 13:50:29,754 [wft4galaxy] [ INFO]  Workflow '_WorkflowTest_Change Case (imported from API)' (id: d3a4d6a5256f2d9a) executed\n",
      "2017-03-30 13:50:29,758 [wft4galaxy] [ INFO]  Checking test output: ...\n",
      "2017-03-30 13:50:29,920 [wft4galaxy] [ INFO]  Checking test output: DONE\n"
     ]
    }
   ],
   "source": [
    "test_result = test_case.run(enable_logger=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "``test_result`` is an instance of ``WorkflowTestResult``, a class which contains several information about an executed workflow, like its ID (dynamically generated when the test starts), the workflow definition and the results of the comparator function for each step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test 0b688d4c-153f-11e7-87e4-a45e60c4fc6b:\n",
      "\t - workflow: [_WorkflowTest_Change Case (imported from API)] \n",
      "\t - results: {u'OutputText': True}\n"
     ]
    }
   ],
   "source": [
    "print(\"Test %s:\\n\\t - workflow: [%s] \\n\\t - results: %r\" % \\\n",
    "      (test_result.test_id, test_result.workflow.name, test_result.results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "For example, you can extract the list of tested outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Outputs:  [u'OutputText']\n"
     ]
    }
   ],
   "source": [
    "print \"Outputs: \", test_result.results.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "... or explicitly check if the test is globally passed or failed (all actual outputs are equal to the expected):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, False)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_result.passed(), test_result.failed()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "... or check whether a specific actual output is equal or not to the expected one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_result.check_output(\"OutputText\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}