Scottx611x/td api (#1646)

* Cleanup some more tests * Add id to aid testing * Add id to aid testing * Cleanup login method * Abstract out reused login moethod * Utilize relative import * PEP8 * only run acceptance tests if we detect a build for a PR * Always run our acceptance tests * Add acceptance tests for user registration and login * Don't use separate shell script (for now) * Remove `cd` comands * Refactor selenium utility functions * Add script to run acceptance tests on vagrant or travis * Lower `DEFAULT_WAIT` Now that we aren't relying on SauceLabs VMs * Cleanup setting of breakpoints, and refactor `cleanup_on_error` decorator * Remove unnecessary env var, and re-add `global` as to ensure a single build * Sauce_connect isn't needed * pip caching seems stable * PEP8 * Fail shell script if any step doesn't work properly * GeckoDriver expects a more recent version of Firefox * Remove Comments * Update Default wait time * Bump pytest-selenium version * Update Comment * Utilize `wait_until_id_clickable` * Add tests for global analysis popover * Add tests for Datasets panel * Update comment * Add a status to all generated Analyses * Update stats test to check for actual object counts * include gecko driver in our build process for ease of local selenium-based testing * Refactor old acceptance test implementation in favor of simplicity and uniformness with our current Django unit tests * Make virtual display bigger * Update selenium testing utils * Update assert body text method * Update selenium tests * Update `wait_until_id_visible()` * bump Factory-Boy requirement * Update travis as mentioned issue has been closed * Clean up gecko driver Puppet install * Update docstrings * Remove unnecessary tests * more geckodriver install cleanup * Change Default wait * update wait_until_id_visible() * pull out duplicated code into ui-deletion helper method * Update acceptance tests * Minor changes due to abstraction of deletion code * Fix typo * guest user creation isn't needed here anymore * Merge branch scottx611x/django_1.7_upgrade into scottx611x/acceptance_testing * Flip the order of some installed apps due to Django 1.7 ContentType loading * Decorate function as to not test 3rd party code's functionality * Utilize StaticLiveServerTestCase * Update delete_from_ui() helper * Refactor decorator name to be more verbose * Don't need to take a screenshot here * Add xvfb and python3 to travis * add pyvirtualdisplay req * Rename geckodriver class to selenium * Update order * Update order of INSTALLED_APPS for testing purposes * Update logging levels * Update TEST_RUNNER per Django system-check suggestion * Update tests * Fix `get_or_create()` * Access actual object returned from `get_or_create()` tuple * Update logger statement * Do not need to specify `live server` here since it has a default value * Update string formatting to work properly * Update isa/pre-isa archive deletion to produce less unnecessary errors * Update Tests * Manually save public group to sync db between threads (Specific to use of StaticLiveServerTestCase) * Update `DEFAULT_WAIT` * Share Datasets upon factory_boy creation * Create public group for APIV2 test case * Added group creation to the wrong TestCase * Remove `django-nose` in favor of Django's new: `DiscoverTestRunner` * This file shouldn't exist anymore. Probably wasn't caught in the merge here: 0c727cc * Include ontology fixture content inside of a Data migration (#1581) * Update puppet-archive version * Utilize `puppet-archive` and clean up neo4j.pp * Cleanup selenium manifest * Factor out solr class from init.pp * don't need to chmod here * We are now using StaticLiveServerTestCase * "Users of the `puppet-archive`module are responsible for archive package dependencies" * Remove unused var * Point to new archive download location * Fix string formatting error * Add `TEST_RUNNER` setting to base.py so that someone testing on `prod` locally could also benefeit * update order of `INSTALLED_APPS` addresses: #1353 * Refactor all tests that interact with the ORM to inherit from `TransactionTestCase` (removes need for crazy tearDown()'s) * DEFAULT_WAIT Shouldn't be that long * Manually save `public group` as to persist data between test and web driver threads * Update comment * Don't need to optimize sole's index in tests * Inherit from TransactionTestCase here * `20` was actually a good fail-safe if things get slow for one reason or another * switch order of INSTALLED APPS * Cleanup selenium test cases * Remove other `unittest` usages and unnecessary tests * Override setUp() in a cleaner manner * Add a travis fold * Clean up FileStoreitem and InvestigationLink error handling * Clean up error handling for isa & pre-isa archives * Update comments * Cleanup factory_boy factories and utils * This is already a default value * DRY * Update existing DataMigrations to a more easily manageable format * More robust Data migrations * Remove `pk` * Bump DRF version to latest that supports Django 1.7 * Add `tools` app * Add models * Configure admin ui for new models * Add migration * Add `tools/definitions/` endpoint * Wrap deletion operations in transactions * Add tools app * Add factories and utils method to create dummy ToolDefinitions for Testing * We don't need `django.setup()` here unless script is being run as a standalone * RunPython is already encapsulated in a transaction * refactor `get_isa_archive()` & `get_pre_isa_archive()` * Analysis.status isn't necessary for testing * Add Error Handling to `SeleniumTestBase.setUp()` * Allow for proper rendering of self-referential Many-to-Many rels. * Update models & serializers * Add basic ToolDefinition API test case * Update Models and Migrations file * Add creation of LIST:LIST:PAIR ToolDef to utils script * Update Serializer for more easily readble API Responses * Update Tests * Add MGMT command to genereate Smaple ToolDefinitions * Demand users be authenticated to access DRF api endpoints * Add more tests for ToolDef API * Refactor DRF API related tests due to newly required Authentication * Remove unused `nesting_type` field * Add comment * Add AdminFieldPopulator class * Update MGMT command * Add `TEST_NON_SERIALIZED_APPS` setting due to pre Django 1.9 bug * Don't need to share datasets pubilcly in acceptance tests * Utilize `serialized_rollback` to ensure migration data persists in StaticLiveServerTestCases * Refactor isa/pre_isa archive fetching into one method * Remove other unnecessary instances of `tearDown()` * Update method signature * Update comment * Fix Test * Revert "Fix Test" This reverts commit 673827a. * Don't bundle (pre)isa_archive fetching functionality & Utilize existing `get_investigation()` * Handle potential AttributeErrors that could occcur * Rename `tools` to `tool_manager` * Favor APITestCase to ensure initial DB state persists throughout tests * Provide request auth in test for newly secured endpoints * Remove transaction context manager * Refactor method name * Keeping Python 3 in mind * Rename field for clarity * Add comments * Fix serializer for newly added fieldname * django.setup() not necessary (unless running as a standalone script) * Change `nested_elements` -> `file_relationship` per Model field rename * Reflect lack of POSTing in comment * Fix import ordering * Remove MGMT command since its only needed for a short period of time * Migrations due to GalaxyParameter child class * Add useful comment * Remove already default values * Revert from usage of Generic DRF api perm classes * Change sample data generation to reflect new schema * Ensure that Users accessing ToolDefinitions are authenticated * Rename GalaxyToolParameter to GalaxyParameter * Squash migrations * Old isa_archive deletion tests were of better form * Quoted references only necessary for manual avoidance of circ. imports * We could have a tool without any Parameters * Cleanup imports * Remove unutilized APIClient * Fix ordering of imports * Fix migration * Remove unnesscessary user creation * Remove TearDown * Remove constraint on ToolDefinition description & rename is_editable field * Add `galaxy_workflow_step` field and squash migrations * Scottx611x/td api url routing (#1622) * Add RouterCombiner class to allow for url definitions to be app specific * Don't really need `import as` * Relative import * Fix import ordering * Place hard-coded DRF api urls into their own apps * Fix Import ordering * Add the ability to generate basic ToolDefinitons from properly annota… (#1624) * Add the ability to generate basic ToolDefinitons from properly annotated Galaxy WFs * Add `galaxy_workflow_step` field * Validate incoming workflow annotation data * Add Factory for GalaxyParameter creation * Add tests to ensure proper creation of ToolDefinitions from Galaxy workflow annotations * Handle ConnectionError if we can't reach Galaxy * Rename `create_nesting()` * Add custom Exception * Refactor `validate_workflow_annotation()` * Refactor ToolDefinition generation MGMT command * Update tests * Simplfy validation logic in mgmt command * Utilize jsonschema to validate incoming ToolDefinitions * Update mgmt command * Update tests and test data * Add `jsonschema` requirement * Rename some things for clarity * Add & Rename test data * Properly handle invalid FileTypes in workflow annotations * Fix Typo * Add helpful comment * jsonschema.validate is None upon successful validation, otherwise a ValidaationError is raised * Cleanup generate_tool_definitions mgmt command * Remove old clunky method to create ToolDefs in favor of new one * Fix tests * Clean up utils.py * Update docstring * Changes suggested from CR * Directly access key rather that iterating through * Split large ToolDefinition schema into individual files * We don't need to use `hyper-schema` * Let an empty object denote that we're at the bottom-most FileRelationship * Travis didn't seem to like `os.path.abspath` so lets try `TRAVIS_BUILD_DIR` env var * Try to fix Travis * Trying out paths again * Git doesn't automatically recognize case-sensitive only filename changes * Remove blank lines and "..." * Styling fixes * Update Galaxy Parameter Model to include a single field * Update ToolDefinition schema * Separate Parameter and GalaxyParameter schemas * Add new sample data utilizing Parameters * Add tests to check for good and bad parameters in Annotation data * Avoid code reuse by extending from other schema * Update test-data and tests to utilize Parameters * Add proper deletion of a ToolDefinitions related objects + tests * Update Parameter schema * Update GalaxyParameter schema * Add schema to aid in validating Workflows individual step's annotations * Add test-data and tests to validate Workflows step's annotations * Rename `validate_workflow_annotation` to `validate_tool_annotation` * `clear()` is redundant if we are also performing `delete()` here * Include parameter and output_file parsing from step annotations in MGMT command * Update ParameterSerializer to include/disclude `galaxy_workflow_step` based on its existance * Add sample data and tests for a basic Visualization-based ToolDefinition * Fix typo and update comment * Update comments * Remove extra space * RuntimeError is actually being raised here * Rename method to create_tool_definition * Add API test to check for different Parameter Types * Styling fixes * Fix comment * Rename pre/post delete handlers * Fix styling * Fix more styling * Remove newline characters * Factor out workflow fetching, and cleanup indentation. * Add a sample galaxy `get_workflows()` response for testing purposes * Test management command functionality with proper `mock` usage * Remove comment explaining "what" * Revert "Remove comment explaining "what"" This reverts commit 69d5c65. * Rename `test-data` to `test_data` * Utilize logger statements since not in a MGMT command anymore * Handle RuntimeError
refinery-platform · Mar 28, 2017 · 3cf0725 · 3cf0725
1 parent ae20312
commit 3cf0725
Show file tree

Hide file tree

Showing 31 changed files with 1,667 additions and 212 deletions.
diff --git a/refinery/tool_manager/admin.py b/refinery/tool_manager/admin.py
@@ -1,7 +1,7 @@
 from django.contrib import admin
 
-from .models import (FileRelationship, InputFile, OutputFile, Parameter,
-                     ToolDefinition)
+from .models import (FileRelationship, GalaxyParameter, InputFile,
+                     OutputFile, Parameter, ToolDefinition)
 
 from .utils import AdminFieldPopulator
 
@@ -26,8 +26,12 @@ class ParameterAdmin(AdminFieldPopulator):
     pass
 
 
+class GalaxyParameterAdmin(AdminFieldPopulator):
+    pass
+
 admin.site.register(FileRelationship, FileRelationshipAdmin)
 admin.site.register(InputFile, InputFileAdmin)
 admin.site.register(ToolDefinition, ToolDefinitionAdmin)
 admin.site.register(OutputFile, OutputFileAdmin)
 admin.site.register(Parameter, ParameterAdmin)
+admin.site.register(GalaxyParameter, GalaxyParameterAdmin)
diff --git a/refinery/tool_manager/management/commands/generate_tool_definitions.py b/refinery/tool_manager/management/commands/generate_tool_definitions.py
@@ -3,12 +3,12 @@
 
 from django.core.management.base import BaseCommand, CommandError
 
-from bioblend.galaxy.client import ConnectionError
-
-from core.models import WorkflowEngine
 from tool_manager.models import ToolDefinition
-from ...utils import (create_tool_definition_from_workflow,
-                      validate_workflow_annotation)
+from ...utils import (ANNOTATION_ERROR_MESSAGE,
+                      create_tool_definition,
+                      get_workflow_list,
+                      validate_tool_annotation,
+                      validate_workflow_step_annotation)
 
 
 class Command(BaseCommand):
@@ -25,55 +25,111 @@ def handle(self, **options):
         Workflows.
         """
         sys.stdout.write("Generating ToolDefinitions\n")
-        workflow_engines = WorkflowEngine.objects.all()
-        sys.stdout.write("{} workflow engines found.\n".format(
-            workflow_engines.count())
-        )
 
-        for engine in workflow_engines:
-            sys.stdout.write(
-                "Generating ToolDefinitions from workflow engine {}\n"
-                .format(engine.name)
-            )
+        try:
+            workflow_list = get_workflow_list()
+        except RuntimeError as e:
+            raise CommandError(e)
+
+        # Validate workflow annotation data, and try to create a
+        # ToolDefinition if validation passes.
 
-            galaxy_connection = engine.instance.galaxy_connection()
+        for workflow in workflow_list:
+            workflow["tool_type"] = ToolDefinition.WORKFLOW
             try:
-                workflow_list = galaxy_connection.workflows.get_workflows()
-            except ConnectionError as e:
+                workflow["annotation"] = json.loads(
+                    workflow["annotation"]
+                )
+            except ValueError as e:
                 raise CommandError(
-                    "Unable to retrieve workflows from '{}'. "
-                    "Skipping ToolDefinition generation: {}".format(
-                        engine.instance.base_url, e)
+                    "Workflow: {}'s annotation is not "
+                    "valid JSON: {}".format(workflow["name"], e)
                 )
 
-            # Validate workflow annotation data, and try to create a
-            # ToolDefinition if validation passes.
-            for workflow in workflow_list:
-                workflow_data = galaxy_connection.workflows.show_workflow(
-                    workflow["id"]
+            # Include `parameters` and `output_files` as keys in our
+            # workflow annotation
+            workflow["annotation"]["parameters"] = []
+            workflow["annotation"]["output_files"] = []
+
+            # Iterate through the workflow's step's annotations and
+            # append to the `parameters` and `output_files` lists from
+            # above so they are included in the validation of this
+            # workflow's annotation data
+            for step_index in workflow["steps"]:
+                step = workflow["steps"][step_index]
+
+                if step["annotation"]:
+                    try:
+                        step_annotation = json.loads(step["annotation"])
+                    except ValueError as e:
+                        raise CommandError(
+                            "Workflow: {}'s Step: {}'s annotation data"
+                            " is not valid JSON: {}".format(
+                                workflow["name"],
+                                step_index,
+                                e
+                            )
+                        )
+                    try:
+                        validate_workflow_step_annotation(
+                            step_annotation
+                        )
+                    except RuntimeError as e:
+                        raise CommandError(
+                            "{} {}".format(ANNOTATION_ERROR_MESSAGE, e)
+                            )
+                    try:
+                        parameters = step_annotation["parameters"]
+                    except KeyError:
+                        # `parameters` aren't required for each workflow step
+                        pass
+
+                    for parameter in parameters:
+                        # Check User-defined parameters in
+                        # annotation data against the available
+                        # parameters of the Workflow step's
+                        # `tool_inputs`
+                        if parameter["name"] not in step["tool_inputs"]:
+                            raise CommandError(
+                                "{} is not a valid parameter for {}".
+                                format(
+                                    parameter["name"],
+                                    step["tool_id"]
+                                )
+                            )
+                        else:
+                            parameter["galaxy_workflow_step"] = int(
+                                step_index
+                            )
+                            workflow["annotation"]["parameters"]\
+                                .append(parameter)
+                    try:
+                        output_files = step_annotation["output_files"]
+                    except KeyError:
+                        # `ouput_files` aren't required for each workflow step
+                        pass
+
+                    for output_file in output_files:
+                        workflow["annotation"]["output_files"].append(
+                            output_file
+                        )
+            try:
+                validate_tool_annotation(workflow)
+            except RuntimeError as e:
+                raise CommandError(e)
+            except Exception as e:
+                raise CommandError(
+                    "Something unexpected happened: {}".format(e)
+                )
+            try:
+                create_tool_definition(workflow)
+            except Exception as e:
+                raise CommandError(
+                    "Creation of ToolDefinition failed. Database "
+                    "rolled back to its state before this "
+                    "ToolDefinition's attempted creation: {}".format(e)
                 )
-                workflow_data["tool_type"] = ToolDefinition.WORKFLOW
-                try:
-                    workflow_data["annotation"] = json.loads(
-                        workflow_data["annotation"]
-                    )
-                except ValueError as e:
-                    raise CommandError(
-                        "Workflow annotation is not valid JSON: {}".format(e)
-                    )
-                try:
-                    validate_workflow_annotation(workflow_data)
-                except RuntimeError as e:
-                    raise CommandError(e)
-                except Exception as e:
-                    raise CommandError(
-                        "Something unexpected happened: {}".format(e)
-                    )
-                try:
-                    create_tool_definition_from_workflow(workflow_data)
-                except Exception as e:
-                    raise CommandError(
-                        "Creation of ToolDefinition failed. Database "
-                        "rolled back to its state before this "
-                        "ToolDefinition's attempted creation: {}".format(e)
-                    )
+
+            sys.stdout.write("\nGenerated ToolDefinition for: {}\n".format(
+                    workflow["name"]
+                ))
diff --git a/refinery/tool_manager/migrations/0002_auto_20170321_1628.py b/refinery/tool_manager/migrations/0002_auto_20170321_1628.py
@@ -0,0 +1,22 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import models, migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('tool_manager', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.RemoveField(
+            model_name='galaxyparameter',
+            name='galaxy_tool_id',
+        ),
+        migrations.RemoveField(
+            model_name='galaxyparameter',
+            name='galaxy_tool_parameter',
+        ),
+    ]
diff --git a/refinery/tool_manager/migrations/0003_auto_20170322_1207.py b/refinery/tool_manager/migrations/0003_auto_20170322_1207.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import models, migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('tool_manager', '0002_auto_20170321_1628'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='parameter',
+            name='is_user_adjustable',
+            field=models.BooleanField(default=True),
+            preserve_default=True,
+        ),
+    ]
diff --git a/refinery/tool_manager/models.py b/refinery/tool_manager/models.py
@@ -1,4 +1,6 @@
 from django.db import models
+from django.db.models.signals import post_delete, pre_delete
+from django.dispatch import receiver
 from django_extensions.db.fields import UUIDField
 
 from file_store.models import FileType
@@ -31,22 +33,28 @@ class Parameter(models.Model):
     uuid = UUIDField(unique=True, auto=True)
     name = models.TextField(max_length=100)
     description = models.TextField(max_length=500)
-    is_user_adjustable = models.BooleanField(default=False)
+    is_user_adjustable = models.BooleanField(default=True)
     value_type = models.CharField(choices=VALUE_TYPES, max_length=25)
     default_value = models.TextField(max_length=100)
 
     def __str__(self):
         return "{}: {} - {}".format(self.value_type, self.name, self.uuid)
 
+    def get_galaxy_workflow_step(self):
+        try:
+            return GalaxyParameter.objects.get(
+                uuid=self.uuid
+            ).galaxy_workflow_step
+        except (GalaxyParameter.DoesNotExist,
+                GalaxyParameter.MultipleObjectsReturned):
+            return None
+
 
 class GalaxyParameter(Parameter):
     """
-    Child of Parameter with fields specific to Galaxy tool parameter
+    Extension of Parameter model with fields specific to Galaxy tool parameters
     """
-    # ID of Galaxy tool that the parameter belongs to
-    galaxy_tool_id = models.TextField(max_length=300)
     galaxy_workflow_step = models.IntegerField()
-    galaxy_tool_parameter = models.TextField(max_length=100)
 
 
 class FileRelationship(models.Model):
@@ -135,3 +143,45 @@ class ToolDefinition(models.Model):
 
     def __str__(self):
         return "{}: {} {}".format(self.tool_type, self.name, self.uuid)
+
+
+@receiver(pre_delete, sender=ToolDefinition)
+def delete_parameters_and_output_files(sender, instance, *args, **kwargs):
+    """
+    Delete related parameter and output_file objects upon ToolDefinition
+    deletion
+    """
+    parameters = instance.parameters.all()
+    for parameter in parameters:
+        parameter.delete()
+
+    output_files = instance.output_files.all()
+    for output_file in output_files:
+        output_file.delete()
+
+
+@receiver(post_delete, sender=ToolDefinition)
+def delete_file_relationship(sender, instance, *args, **kwargs):
+    """
+    Delete related (topmost) FileRelationship object after ToolDefinition
+    deletion.
+    """
+    instance.file_relationship.delete()
+
+
+@receiver(pre_delete, sender=FileRelationship)
+def delete_input_files_and_file_relationships(sender, instance, *args,
+                                              **kwargs):
+    """
+    Delete all related nested file_relationships in a recursive manner.
+
+    Due to the nature of the `pre_delete` signal, this approach will delete
+    the bottom-most FileRelationship object first which is desired.
+    """
+    input_files = instance.input_files.all()
+    for input_file in input_files:
+        input_file.delete()
+
+    file_relationships = instance.file_relationship.all()
+    for file_relationship in file_relationships:
+        file_relationship.delete()
diff --git a/refinery/tool_manager/schemas/FileRelationship.json b/refinery/tool_manager/schemas/FileRelationship.json
@@ -26,7 +26,7 @@
     },
     "file_relationship": {
       "oneOf": [
-          {"$ref": "FileRelationship.json#"},
+          {"type": "object", "$ref": "FileRelationship.json#"},
           {"type": "object", "additionalProperties": false, "properties": {}}
         ]
       }

diff --git a/refinery/tool_manager/schemas/GalaxyParameter.json b/refinery/tool_manager/schemas/GalaxyParameter.json
@@ -1,20 +1,12 @@
 {
   "$schema": "http://json-schema.org/draft-04/schema",
-  "type": "object",
-  "required": [
-    "galaxy_tool_id",
-    "galaxy_workflow_step",
-    "galaxy_tool_parameter"
-  ],
+  "extends": {"$ref": "Parameter.json#"},
   "properties": {
-    "galaxy_tool_id": {
-      "type": "string"
-    },
     "galaxy_workflow_step": {
-      "type": "string"
-    },
-    "galaxy_tool_parameter": {
-      "type": "string"
+      "type": "number"
     }
-  }
-}
+  },
+  "required": [
+    "galaxy_workflow_step"
+  ]
+}
diff --git a/refinery/tool_manager/schemas/OutputFile.json b/refinery/tool_manager/schemas/OutputFile.json
@@ -13,7 +13,7 @@
     "description": {
       "type": "string"
     },
-    "file_type": {
+    "filetype": {
       "type": "object",
       "$ref": "FileType.json#"
     }