docs/notebooks/5_config_workflow_test_suite.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Define a test suite"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "See the folder *example*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "EXAMPLES_FOLDER = \"../examples\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "... where you can find the following files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['change_case',\n",
       " 'fluxomics_stationary',\n",
       " 'multivariate',\n",
       " 'sacurine',\n",
       " 'workflow-test-suite-full.yml',\n",
       " 'workflow-test-suite-min.yml',\n",
       " 'workflow-test-suite.yml',\n",
       " 'workflows.json']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os, pprint\n",
    "[f for f in os.listdir(EXAMPLES_FOLDER) if not f.startswith('.')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Consider the **definition file** ``workflow-test-suite-min.yml`` (steps[3-4]), which contains the two workflow tests named ``change_case`` and ``multivariate`` respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "suite_conf_filename = os.path.join(EXAMPLES_FOLDER, \"workflow-test-suite-min.yml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"enable_logger\": false, \n",
      "    \"workflows\": {\n",
      "        \"change_case\": {\n",
      "            \"expected\": {\n",
      "                \"OutputText\": \"change_case/expected_output\"\n",
      "            }, \n",
      "            \"inputs\": {\n",
      "                \"InputText\": \"change_case/input\"\n",
      "            }, \n",
      "            \"file\": \"change_case/workflow.ga\"\n",
      "        }, \n",
      "        \"multivariate\": {\n",
      "            \"expected\": {\n",
      "                \"variableMetadata_out\": \"multivariate/variableMetadata_out\", \n",
      "                \"sampleMetadata_out\": \"multivariate/sampleMetadata_out\"\n",
      "            }, \n",
      "            \"inputs\": {\n",
      "                \"DataMatrix\": \"multivariate/dataMatrix.tsv\", \n",
      "                \"SampleMetadata\": \"multivariate/sampleMetadata.tsv\", \n",
      "                \"VariableMetadata\": \"multivariate/variableMetadata.tsv\"\n",
      "            }, \n",
      "            \"params\": {\n",
      "                \"3\": {\n",
      "                    \"predI\": \"1\", \n",
      "                    \"respC\": \"gender\", \n",
      "                    \"orthoI\": \"NA\", \n",
      "                    \"testL\": \"FALSE\"\n",
      "                }\n",
      "            }, \n",
      "            \"file\": \"multivariate/workflow.ga\"\n",
      "        }\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import yaml, json\n",
    "with open(suite_conf_filename, \"r\") as fp:\n",
    "    data = yaml.load(fp)\n",
    "    print(json.dumps(data, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Suppose that you have two ``WorkflowTestCase`` instances (we load them in steps[5-6] from a definition file, but you can define them programmatically as described in \"[Define a single test case](4_config_single_workflow_test.html)\"):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from wft4galaxy.core import WorkflowTestCase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "wft1 = WorkflowTestCase.load(suite_conf_filename, \"change_case\")\n",
    "wft2 = WorkflowTestCase.load(suite_conf_filename, \"multivariate\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To programmatically create test suite you need to create an instance of the ``WorkflowTestSuite`` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from wft4galaxy.core import WorkflowTestSuite\n",
    "suite = WorkflowTestSuite()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "source": [
    "and register the ``WorkflowTestSuite`` instances to the suite instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "suite.add_workflow_test(wft1)\n",
    "suite.add_workflow_test(wft2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now run your test suite and inspect the results of its execution (steps [9-10]), as described in \"[Run a test suite](2_run_test_suite.html)\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Workflow Test: 'change_case' ... 2017-03-30 15:08:02,477 [wft4galaxy] [ INFO]  Create a history '_WorkflowTestHistory_e79f068a-1549-11e7-823b-a45e60c4fc6b' (id: u'c8a1d1385ed48f8d')\n",
      "2017-03-30 15:08:03,756 [wft4galaxy] [ INFO]  Workflow '_WorkflowTest_Change Case (imported from API)' (id: c8a1d1385ed48f8d) running ...\n",
      "2017-03-30 15:08:06,367 [wft4galaxy] [ INFO]  waiting for datasets\n",
      "2017-03-30 15:08:06,559 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: new\n",
      "2017-03-30 15:08:07,502 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: new\n",
      "2017-03-30 15:08:08,239 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: new\n",
      "2017-03-30 15:08:08,900 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: new\n",
      "2017-03-30 15:08:09,830 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: new\n",
      "2017-03-30 15:08:10,476 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: new\n",
      "2017-03-30 15:08:11,386 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: queued\n",
      "2017-03-30 15:08:12,103 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: running\n",
      "2017-03-30 15:08:12,807 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: running\n",
      "2017-03-30 15:08:13,638 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: running\n",
      "2017-03-30 15:08:14,357 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: running\n",
      "2017-03-30 15:08:15,365 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: ok\n",
      "2017-03-30 15:08:15,872 [wft4galaxy] [ INFO]  Workflow '_WorkflowTest_Change Case (imported from API)' (id: c8a1d1385ed48f8d) executed\n",
      "2017-03-30 15:08:15,875 [wft4galaxy] [ INFO]  Checking test output: ...\n",
      "2017-03-30 15:08:16,054 [wft4galaxy] [ INFO]  Checking test output: DONE\n",
      "ok\n",
      "Workflow Test: 'multivariate' ... 2017-03-30 15:08:17,738 [wft4galaxy] [ INFO]  Create a history '_WorkflowTestHistory_f08acdb0-1549-11e7-bf2e-a45e60c4fc6b' (id: u'2d190d7d262b60f4')\n",
      "2017-03-30 15:08:30,657 [wft4galaxy] [ INFO]  Workflow '_WorkflowTest_Multivariate (imported from API)' (id: 2d190d7d262b60f4) running ...\n",
      "2017-03-30 15:08:39,075 [wft4galaxy] [ INFO]  waiting for datasets\n",
      "2017-03-30 15:08:39,483 [wft4galaxy] [ INFO]  b425b2361a02d64e: queued\n",
      "2017-03-30 15:08:39,970 [wft4galaxy] [ INFO]  eff20c5ba6bf6f87: queued\n",
      "2017-03-30 15:08:40,455 [wft4galaxy] [ INFO]  77efbec71a7e1a47: queued\n",
      "2017-03-30 15:08:40,781 [wft4galaxy] [ INFO]  ea3d97b57366b47f: queued\n",
      "2017-03-30 15:08:41,453 [wft4galaxy] [ INFO]  b425b2361a02d64e: queued\n",
      "2017-03-30 15:08:41,952 [wft4galaxy] [ INFO]  eff20c5ba6bf6f87: queued\n",
      "2017-03-30 15:08:42,266 [wft4galaxy] [ INFO]  77efbec71a7e1a47: queued\n",
      "2017-03-30 15:08:42,443 [wft4galaxy] [ INFO]  ea3d97b57366b47f: queued\n",
      "2017-03-30 15:08:43,290 [wft4galaxy] [ INFO]  b425b2361a02d64e: queued\n",
      "2017-03-30 15:08:43,526 [wft4galaxy] [ INFO]  eff20c5ba6bf6f87: queued\n",
      "2017-03-30 15:08:43,788 [wft4galaxy] [ INFO]  77efbec71a7e1a47: queued\n",
      "2017-03-30 15:08:44,030 [wft4galaxy] [ INFO]  ea3d97b57366b47f: queued\n",
      "2017-03-30 15:08:44,678 [wft4galaxy] [ INFO]  b425b2361a02d64e: queued\n",
      "2017-03-30 15:08:45,228 [wft4galaxy] [ INFO]  eff20c5ba6bf6f87: queued\n",
      "2017-03-30 15:08:45,440 [wft4galaxy] [ INFO]  77efbec71a7e1a47: queued\n",
      "2017-03-30 15:08:45,629 [wft4galaxy] [ INFO]  ea3d97b57366b47f: queued\n",
      "2017-03-30 15:08:46,418 [wft4galaxy] [ INFO]  b425b2361a02d64e: queued\n",
      "2017-03-30 15:08:46,848 [wft4galaxy] [ INFO]  eff20c5ba6bf6f87: queued\n",
      "2017-03-30 15:08:47,146 [wft4galaxy] [ INFO]  77efbec71a7e1a47: ok\n",
      "2017-03-30 15:08:47,662 [wft4galaxy] [ INFO]  ea3d97b57366b47f: ok\n",
      "2017-03-30 15:08:48,312 [wft4galaxy] [ INFO]  b425b2361a02d64e: ok\n",
      "2017-03-30 15:08:48,485 [wft4galaxy] [ INFO]  eff20c5ba6bf6f87: ok\n",
      "2017-03-30 15:08:48,986 [wft4galaxy] [ INFO]  Workflow '_WorkflowTest_Multivariate (imported from API)' (id: 2d190d7d262b60f4) executed\n",
      "2017-03-30 15:08:48,989 [wft4galaxy] [ INFO]  Checking test output: ...\n",
      "2017-03-30 15:08:49,187 [wft4galaxy] [ INFO]  Checking test output: DONE\n",
      "ok\n",
      "\n",
      "----------------------------------------------------------------------\n",
      "Ran 2 tests in 48.039s\n",
      "\n",
      "OK\n"
     ]
    }
   ],
   "source": [
    "test_results = suite.run(enable_logger=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test e79f068a-1549-11e7-823b-a45e60c4fc6b:\n",
      "\t - workflow: [_WorkflowTest_Change Case (imported from API)] \n",
      "\t - results: {u'OutputText': True}\n",
      "Test f08acdb0-1549-11e7-bf2e-a45e60c4fc6b:\n",
      "\t - workflow: [_WorkflowTest_Multivariate (imported from API)] \n",
      "\t - results: {u'variableMetadata_out': True, u'sampleMetadata_out': True}\n"
     ]
    }
   ],
   "source": [
    "for r in test_results:\n",
    "    print(\"Test %s:\\n\\t - workflow: [%s] \\n\\t - results: %r\" % (r.test_id, r.workflow.name, r.results))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}