Analysis refactor gui part7 #2117

josenavas · 2017-04-25T19:14:33Z

This is NOT ready for review

I will comment with relevant information about the PR and remove the WIP tag once this is ready for review

Fix 1505

fix qiita-spots#1987

…estions Processing recommendations and pipelines

fix filter job list

fix get_studies

Fix by blinking

fix qiita-spots#1913

…-part6

…-part7

…nalysis only

josenavas · 2017-05-16T14:55:27Z

@antgonza ready for review. In this PR I've modified the command object so it can be defined in the analysis pipeline only. This PR is also adding some of the missing files that I had locally but not in the branch to be able to execute the analysis as plugins. I've also fixed a couple of small bugs that I found on the job object.

antgonza · 2017-05-16T17:08:42Z

qiita_db/private.py

+        job._set_validator_jobs(val_jobs)
+
+        for j in val_jobs:
+            j.submit()


Just thinking out loud (not sure if that applies to written stuff, anyway ...) but should we add the sleep directly to the j.submit?

I would disagree because the sleep is only needed if submitting multiple jobs, if you're submitting a single one it is not needed, so no reason to stop the processing.

In some cases we found that if we programmatically submit multiple jobs to the torque queue, the resource queue are not updated accordingly and the jobs end up not being spawned fairly across the cluster.

The reason for the delays historically was to spread load on the resource manager and scheduler as they are singleton resources within a shared environment. Generally speaking, the delays were only important for large volumes of submissions (e.g., > 100). I don't think we're doing that here. If there are issues where the scheduler or resource manager are not handling a small number of jobs appropriately, then it suggests either the jobs themselves are flawed or there is a critical issue with the servers handling the requests. Since we routinely hammer these servers outside of Qiita without delays, it suggests the former is the more likely scenario.

If you're not comfortable removing the delay, then at least set it to <= 100ms as 1s is a lifetime.

I wasn't aware Qiita used a library that links to the Torque C API? Aren't these programatic requests actually system calls?

antgonza · 2017-05-16T17:12:00Z

qiita_db/processing_job.py

@@ -1196,7 +1208,7 @@ def _raise_if_not_in_construction(self):
                     WHERE processing_job_workflow_id = %s"""
            qdb.sql_connection.TRN.add(sql, [self.id])
            res = qdb.sql_connection.TRN.execute_fetchflatten()
-            if len(res) != 1 or res[0] != 'in_construction':
+            if (len(res) == 1 and res[0] != 'in_construction') or len(res) > 1:


Could you add some documentation of each test? I'm not following why/when len(res) > 1 ...

Added a comment

antgonza · 2017-05-16T17:35:55Z

qiita_pet/templates/artifact_ajax/processing_artifact.html

+    }
+    else if (p_type == 'integer') {
+      var inp = $('<input>').appendTo("#cmd-opts-div").attr('id', p_name).attr('name', p_name).attr('type', 'number').addClass('required-parameter');
+    }


should there be an else case?

Oops - I moved this file and started working on this function before our meeting last week. This function is the one that I have to modify in the next functionality, so I would suggest leave as it is since I will be completing this function in my next PR. Does this sound good to you?

antgonza · 2017-05-16T17:39:20Z

qiita_pet/templates/artifact_ajax/processing_artifact.html

+   *
+   */
+  function load_artifact_type(p_nodes) {
+    var types = []


antgonza · 2017-05-16T17:40:20Z

scripts/qiita-private-2

@@ -0,0 +1,30 @@
+#!/usr/bin/env python


not sure qiita-private-2 is a good name, sorry no suggestions ...

Agreed - renamed

josenavas

Comments addressed

josenavas · 2017-05-16T18:30:44Z

qiita_db/private.py

+        job._set_validator_jobs(val_jobs)
+
+        for j in val_jobs:
+            j.submit()


I would disagree because the sleep is only needed if submitting multiple jobs, if you're submitting a single one it is not needed, so no reason to stop the processing.

josenavas · 2017-05-16T18:35:32Z

qiita_db/processing_job.py

@@ -1196,7 +1208,7 @@ def _raise_if_not_in_construction(self):
                     WHERE processing_job_workflow_id = %s"""
            qdb.sql_connection.TRN.add(sql, [self.id])
            res = qdb.sql_connection.TRN.execute_fetchflatten()
-            if len(res) != 1 or res[0] != 'in_construction':
+            if (len(res) == 1 and res[0] != 'in_construction') or len(res) > 1:


Added a comment

josenavas · 2017-05-16T18:37:52Z

qiita_pet/templates/artifact_ajax/processing_artifact.html

+    }
+    else if (p_type == 'integer') {
+      var inp = $('<input>').appendTo("#cmd-opts-div").attr('id', p_name).attr('name', p_name).attr('type', 'number').addClass('required-parameter');
+    }


Oops - I moved this file and started working on this function before our meeting last week. This function is the one that I have to modify in the next functionality, so I would suggest leave as it is since I will be completing this function in my next PR. Does this sound good to you?

josenavas · 2017-05-16T18:38:23Z

qiita_pet/templates/artifact_ajax/processing_artifact.html

+   *
+   */
+  function load_artifact_type(p_nodes) {
+    var types = []


josenavas · 2017-05-16T18:38:24Z

qiita_pet/templates/artifact_ajax/processing_artifact.html

+   *
+   */
+  function load_artifact_type(p_nodes) {
+    var types = []


josenavas · 2017-05-16T18:39:06Z

scripts/qiita-private-2

@@ -0,0 +1,30 @@
+#!/usr/bin/env python


Agreed - renamed

wasade

Few comments

wasade · 2017-05-16T20:00:01Z

qiita_db/private.py

+    job.update_heartbeat_state()
+
+    try:
+        task_name = job.command.name


Does the instantiation of this local variable need to be within the try?

Not really, moved outside of the try.

wasade · 2017-05-16T20:01:00Z

qiita_db/private.py

+        task_name = job.command.name
+        TASK_DICT[task_name](job)
+    except Exception:
+        job.complete(False, error="Error executing private task: %s"


what happens if the job does not except, shouldn't there be a job.complete(True, ...)?

Not necessarily, as it depends on the actual private task being executed (right now there is only one type of private tasks, but we will be adding more in the future).

Let's say that we have job A that generates one output O. The output O needs to be validated by a different job B. The way it is currently done is that job A is marked as 'running' until the validator job B has completed. This way, if the validator fails because the output is not correct, the job A is marked as failed (as it should be). However, if the validator B completes successfully, it automatically marks job A as completed and success.

The current private task (build_analysis_files) takes advantage of this behavior by creating the BIOM tables and submitting the validator jobs to the queue. Once those jobs are completed, they will set the correct output status to the current job. Does it make sense?

wasade · 2017-05-16T20:06:28Z

qiita_db/software.py

@@ -222,6 +226,9 @@ def create(cls, software, name, description, parameters, outputs=None):
        outputs : dict, optional
            The description of the outputs that this command generated. The
            format is: {output_name: artifact_type}
+        analysis_only : bool, optional


Does false mean the command is in the analysis pipeline and whatever other pipelines exist? Is it important to be able to restrict commands to non-analysis pipelines? Or, rather, is a two-state variable sufficient to describe the different scenarios faced?

We currently have two pipelines using the artifacts structure: the study pipeline and the (meta-)analysis pipeline. In my opinion, commands in the study pipeline should be available in the meta-analysis pipeline (my example command was open-ref, although we are no longer adding it to the system), but we do want to restrict the analysis pipeline commands on the study pipeline. Given the current status of the project and what we have outlined so far for the future of Qiita I don't foresee adding any other pipeline to the system that will use the artifacts structure as it currently stands.

What is the logical difference between a study command and an analysis command?

wasade · 2017-05-16T20:07:00Z

qiita_db/software.py

+        Returns
+        -------
+        bool
+            Whether the commad is analysis only or not


wasade · 2017-05-16T20:08:09Z

qiita_db/software.py

@@ -508,6 +515,22 @@ def activate(self):
            qdb.sql_connection.TRN.add(sql, [True, self.id])
            return qdb.sql_connection.TRN.execute()

+    @property
+    def analysis_only(self):
+        """Returns if the command is an analysis-only command


Can a command be both an analysis command and "other"? I guess I don't understand what is under the umbrella of "analysis" and what is not

There are currently two options:
(1) The command is available in the analysis pipeline and the study processing pipeline
(2) The command is available only in the analysis pipeline.

I don't think the option "Only on study processing pipeline" should be added as this can potentially limit the opportunities to run meta-analysis, see my previous comment.

josenavas

Thanks for the comments!

josenavas · 2017-05-16T20:22:28Z

qiita_db/private.py

+    job.update_heartbeat_state()
+
+    try:
+        task_name = job.command.name


Not really, moved outside of the try.

josenavas · 2017-05-16T20:27:03Z

qiita_db/private.py

+        task_name = job.command.name
+        TASK_DICT[task_name](job)
+    except Exception:
+        job.complete(False, error="Error executing private task: %s"


Not necessarily, as it depends on the actual private task being executed (right now there is only one type of private tasks, but we will be adding more in the future).

Let's say that we have job A that generates one output O. The output O needs to be validated by a different job B. The way it is currently done is that job A is marked as 'running' until the validator job B has completed. This way, if the validator fails because the output is not correct, the job A is marked as failed (as it should be). However, if the validator B completes successfully, it automatically marks job A as completed and success.

The current private task (build_analysis_files) takes advantage of this behavior by creating the BIOM tables and submitting the validator jobs to the queue. Once those jobs are completed, they will set the correct output status to the current job. Does it make sense?

josenavas · 2017-05-16T20:30:17Z

qiita_db/software.py

@@ -222,6 +226,9 @@ def create(cls, software, name, description, parameters, outputs=None):
        outputs : dict, optional
            The description of the outputs that this command generated. The
            format is: {output_name: artifact_type}
+        analysis_only : bool, optional


We currently have two pipelines using the artifacts structure: the study pipeline and the (meta-)analysis pipeline. In my opinion, commands in the study pipeline should be available in the meta-analysis pipeline (my example command was open-ref, although we are no longer adding it to the system), but we do want to restrict the analysis pipeline commands on the study pipeline. Given the current status of the project and what we have outlined so far for the future of Qiita I don't foresee adding any other pipeline to the system that will use the artifacts structure as it currently stands.

josenavas · 2017-05-16T20:32:02Z

qiita_db/software.py

@@ -508,6 +515,22 @@ def activate(self):
            qdb.sql_connection.TRN.add(sql, [True, self.id])
            return qdb.sql_connection.TRN.execute()

+    @property
+    def analysis_only(self):
+        """Returns if the command is an analysis-only command


There are currently two options:
(1) The command is available in the analysis pipeline and the study processing pipeline
(2) The command is available only in the analysis pipeline.

I don't think the option "Only on study processing pipeline" should be added as this can potentially limit the opportunities to run meta-analysis, see my previous comment.

josenavas · 2017-05-16T20:32:26Z

qiita_db/software.py

+        Returns
+        -------
+        bool
+            Whether the commad is analysis only or not


* Analysis refactor db (#2040) * New DB structure * Adding python patch * Adding a biom so we can actually execute the patch * Fixing the patch to correctly transfer the information from the old structure to the new one * Fixing patch * Fixing patch and a few other bits to make the patch run successfully * These files are no longer needed * Droping analysis status table * Linking the analysis with all the artifacts * Fixing typo * Fixing HTML and dbschema files * Adding analyisis jobs * Adding logging column to the analysis * Addressing @antgonza's comments * Taking into account non-phylogenetic metrics in beta diversity * Analysis refactor fix analysis (#2044) * New DB structure * Adding python patch * Adding a biom so we can actually execute the patch * Fixing the patch to correctly transfer the information from the old structure to the new one * Fixing patch * Fixing patch and a few other bits to make the patch run successfully * These files are no longer needed * Removing unused code * Droping analysis status table * Linking the analysis with all the artifacts * Fixing typo * Fixing HTML and dbschema files * Adding analyisis jobs * Extending the artifact to work with the analysis * Allowing multiomics datatype * Adding private_job_submitter and modifying proc job handler to use it * Adding logging column to the analysis * Adding datatype to the analysis-processing job table * Adding REST endpoint to access the analysis metadata * Adding private jobs to plugin * Fixing typo * Fixing the processing jobs complete * Addressing @antgonza's comments * Taking into account non-phylogenetic metrics in beta diversity * Addressing @antgonza's comments * Addressing @ElDeveloper's comments * Remove old job code (#2045) * New DB structure * Adding python patch * Adding a biom so we can actually execute the patch * Fixing the patch to correctly transfer the information from the old structure to the new one * Fixing patch * Fixing patch and a few other bits to make the patch run successfully * These files are no longer needed * Removing unused code * Droping analysis status table * Linking the analysis with all the artifacts * Fixing typo * Fixing HTML and dbschema files * Adding analyisis jobs * Extending the artifact to work with the analysis * Allowing multiomics datatype * Adding private_job_submitter and modifying proc job handler to use it * Adding logging column to the analysis * Adding datatype to the analysis-processing job table * Adding REST endpoint to access the analysis metadata * Adding private jobs to plugin * Fixing typo * Fixing the processing jobs complete * Removing the old job code * Oops removed the wrong file * Addressing @antgonza's comments * Taking into account non-phylogenetic metrics in beta diversity * Addressing @antgonza's comments * Fixes Qiita db (#2046) * New DB structure * Adding python patch * Adding a biom so we can actually execute the patch * Fixing the patch to correctly transfer the information from the old structure to the new one * Fixing patch * Fixing patch and a few other bits to make the patch run successfully * These files are no longer needed * Removing unused code * Droping analysis status table * Linking the analysis with all the artifacts * Fixing typo * Fixing HTML and dbschema files * Adding analyisis jobs * Extending the artifact to work with the analysis * Allowing multiomics datatype * Adding private_job_submitter and modifying proc job handler to use it * Adding logging column to the analysis * Adding datatype to the analysis-processing job table * Adding REST endpoint to access the analysis metadata * Adding private jobs to plugin * Fixing typo * Fixing the processing jobs complete * Removing the old job code * Oops removed the wrong file * Removing QiitaStatusObject because it is not used * fixing metautil * Fixing porntal, setup and sql tests * Fixing user and util * Fixing qiita_db * Addressing @antgonza's comments * Taking into account non-phylogenetic metrics in beta diversity * Addressing @antgonza's comments * Analysis refactor master (#2070) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * New DB structure * Adding python patch * Adding a biom so we can actually execute the patch * Fixing the patch to correctly transfer the information from the old structure to the new one * Fixing patch * Fixing patch and a few other bits to make the patch run successfully * These files are no longer needed * Removing unused code * Droping analysis status table * Linking the analysis with all the artifacts * Fixing typo * Fixing HTML and dbschema files * Adding analyisis jobs * Extending the artifact to work with the analysis * Allowing multiomics datatype * Adding private_job_submitter and modifying proc job handler to use it * Adding logging column to the analysis * Adding datatype to the analysis-processing job table * Adding REST endpoint to access the analysis metadata * Adding private jobs to plugin * Fixing typo * Fixing the processing jobs complete * Removing the old job code * Oops removed the wrong file * Removing QiitaStatusObject because it is not used * fixing metautil * Fixing porntal, setup and sql tests * Fixing user and util * Fixing qiita_db * fix #1987 * Updating to master * Addressing @antgonza's comments * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * Taking into account non-phylogenetic metrics in beta diversity * fix #1959 * Addressing @antgonza's comments * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * Actually fixing merge conflicts * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * Sync-ing with master (again) (#2073) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Fix qiita ware (#2074) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * Analysis refactor GUI part1 (#2076) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * Addressing @antgonza's comments * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * Analysis refactor GUI part2 (#2077) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Removing unused code * Changing bdiv metrics to single choice * Fixing patch * Various small fixes to be able to run tests on the plugins * Adding private module * Fixing processing job completion * Fixing patch 52 * Fixing call * Fixing complete * small fixes * Analysis refactor gui part3 (#2078) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Fixing typo * Addressing @antgonza's comments * Addressing @antgonza's comments * Analysis refactor gui part4 (#2079) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Analysis refactor gui part5 (#2115) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * Changing bdiv metrics to single choice * Fixing patch * Various small fixes to be able to run tests on the plugins * Addressing @antgonza's comments * Analysis refactor gui part6 (#2116) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * fixing timestamp * rm formats * st -> pt * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * moving to ISO 8601 - wow :'( * fix errors * addressing @wasade comments * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * fixing private/public studies * Changing bdiv metrics to single choice * sanbox-to-sandbox * flake8 * Fixing patch * fixing other issues * adding share documentation * psycopg2 <= 2.7 * psycopg2 < 2.7 * Various small fixes to be able to run tests on the plugins * Solving merge conflicts * Adding processing handlers * Fixing url and bug on processing job workflow * Adding the private script runner * Adding is_analysis column to the command * Adding retrieval of commands excluding analysis commands * Addressing bug on retrieving information from redis * Enabling the command register endpoint to provide if the command is analysis only * Addressing @antgonza's comments * Addressing @wasade's comments * Supporting multiple choice * Adding documentation * Analysis refactor gui part7 (#2117) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * fixing timestamp * rm formats * st -> pt * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * moving to ISO 8601 - wow :'( * fix errors * addressing @wasade comments * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * fixing private/public studies * Changing bdiv metrics to single choice * sanbox-to-sandbox * flake8 * Fixing patch * fixing other issues * adding share documentation * psycopg2 <= 2.7 * psycopg2 < 2.7 * Various small fixes to be able to run tests on the plugins * Adding private module * Fixing processing job completion * Fixing patch 52 * Fixing call * Fixing complete * small fixes * Adding processing handlers * Fixing url and bug on processing job workflow * Adding the private script runner * Adding is_analysis column to the command * Adding retrieval of commands excluding analysis commands * Addressing bug on retrieving information from redis * Enabling the command register endpoint to provide if the command is analysis only * Addressing @antgonza's comments * Addressing @wasade's comments * Modifying handler to pass allow_change_optionals * returning optional parameters * Addressing bug found by @antgonza * Enabling changing the default parameters * Adding correct class * Allowing user to change default parameters * Fixing bug with commands listing * Enabling arbitrary htmls in the summary * Prepping for merge hell * Addressing @wasade's comments * Fixing qiita_db tests * Analysis refactor allow users change dflt params (#2136) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * fixing timestamp * rm formats * st -> pt * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * moving to ISO 8601 - wow :'( * fix errors * addressing @wasade comments * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * fixing private/public studies * Changing bdiv metrics to single choice * sanbox-to-sandbox * flake8 * Fixing patch * fixing other issues * adding share documentation * psycopg2 <= 2.7 * psycopg2 < 2.7 * Various small fixes to be able to run tests on the plugins * Adding private module * Fixing processing job completion * Fixing patch 52 * Fixing call * Fixing complete * small fixes * Adding processing handlers * Fixing url and bug on processing job workflow * Adding the private script runner * Adding is_analysis column to the command * Adding retrieval of commands excluding analysis commands * Addressing bug on retrieving information from redis * Enabling the command register endpoint to provide if the command is analysis only * Addressing @antgonza's comments * Addressing @wasade's comments * Supporting multiple choice * Adding documentation * Modifying handler to pass allow_change_optionals * returning optional parameters * Addressing bug found by @antgonza * Enabling changing the default parameters * Adding correct class * Allowing user to change default parameters * Fixing bug with commands listing * Addressing @wasade's comments * Addressing @antgonza's comment * Solving a few more tests * Arbitrary summary HTML (#2138) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * fixing timestamp * rm formats * st -> pt * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * moving to ISO 8601 - wow :'( * fix errors * addressing @wasade comments * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * fixing private/public studies * Changing bdiv metrics to single choice * sanbox-to-sandbox * flake8 * Fixing patch * fixing other issues * adding share documentation * psycopg2 <= 2.7 * psycopg2 < 2.7 * Various small fixes to be able to run tests on the plugins * Adding private module * Fixing processing job completion * Fixing patch 52 * Fixing call * Fixing complete * small fixes * Adding processing handlers * Fixing url and bug on processing job workflow * Adding the private script runner * Adding is_analysis column to the command * Adding retrieval of commands excluding analysis commands * Addressing bug on retrieving information from redis * Enabling the command register endpoint to provide if the command is analysis only * Addressing @antgonza's comments * Addressing @wasade's comments * Supporting multiple choice * Adding documentation * Modifying handler to pass allow_change_optionals * returning optional parameters * Addressing bug found by @antgonza * Enabling changing the default parameters * Adding correct class * Allowing user to change default parameters * Fixing bug with commands listing * Enabling arbitrary htmls in the summary * Addressing @wasade's comments * Addressing @antgonza's comment * Analysis refactor sync with master - DEPENDS ON #2138 (#2139) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1805 * adding button * fix errors * fix #1816 * fixing failures tests * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * fixing timestamp * rm formats * st -> pt * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * moving to ISO 8601 - wow :'( * fix errors * addressing @wasade comments * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * fixing private/public studies * Changing bdiv metrics to single choice * sanbox-to-sandbox * flake8 * Fixing patch * fixing other issues * adding share documentation * psycopg2 <= 2.7 * psycopg2 < 2.7 * Various small fixes to be able to run tests on the plugins * Adding private module * Fixing processing job completion * Fixing patch 52 * Fixing call * Fixing complete * small fixes * init commit * fixing errors * fixing errors due to update * Making the download work * Fixing tests * working status * adding tags, the right way! * fix error * Addressing @antgonza's comments * Adding missing test * Ignoring tgz - thanks @antgonza * addressing @josenavas comments * list study tags * fix error * adding tags to public * adding docs * addressing @wasade comment * addressing @josenavas and @wasade comments * addressing @wasade request * fix #2091 * option 2: @ElDeveloper and @josenavas * A minimal REST API for Qiita (#2094) * TST: Add initial test cases for study handler * ENH: Add initial study rest api * API: test if a study exists * ENH: oauth2 forced * Get back basic study deets * TST: test for samples collection * API: rest get sample IDs from a study * ENH: samples/info handler * broken routes * API: request sample metadata * ENH/API: Add methods to check for a study person * ENH/API: Add POST methods for study person * TST: Add tests for from_name_and_affiliation * TST: study creation * BUG: Add headers to tests * ENH: create study * Adjust GET on study description * API: Add endpoints for preparation creation * TST: 200 :D * TST: Correctly verify study instantiation * TST: prep artifact creation * ENH/API: associate artifacts with a preparation * TST: test study statys * ENH: study status * Removed trailing whitespace * STY: PEP8 * MAINT: refactor, centralize setup boilerplate * REFACTOR: Remove repeated code * DOC: Remove unnecessary comments * REFACTOR: Missing removal of pattern * STY: Fix PEP8 errors * BUG: Incorrectly changed error code * BUG/TST: Fix typo in tests * Addressing an @antgonza comment * Another @antgonza comment * RVW: Address review comments * ENH: Cleanup webserver and name-spaces * ENH: Improve error messages * ENH: Add more descriptive error message * TST: Exercise different argument types * DOC: Add documentation for REST API * ENH: Remove extra comma * ENH/DOC: update/add samples to sample information via rest (#2097) * Changing how artifact visibility works (#2098) * changing how artifact visibility works * fixing code * fix errors * fixing edit check access * fix * fix #2086 * flak8 * addressing @ElDeveloper comments + fixes * adding the final changes * fix failures * get_qiita_version -> generate_biom_and_metadata_release * download raw data * adding missing empty files * Adding endpoint to retrieve list of person (#2103) * Adding missing endpoint * Addressing @ElDeveloper comment * fix #2086 (#2102) * fix #2086 * flak8 * addressing @ElDeveloper comments + fixes * adding the final changes * fix failures * get_qiita_version -> generate_biom_and_metadata_release * addressing @wasade comments and fix errors * fix error? * rm vfabu + addressing @wasade and @josenavas comments + fix errors * just being dumb! * Deblur quality mention (#2107) * Revised rst values used for section headers (#2108) * Adding processing handlers * Fixing latlongs (#2120) * public studies are being shown in the user own studies * fix #2069 - adding tests * flake8 * Fixing url and bug on processing job workflow * Adding the private script runner * Adding is_analysis column to the command * Adding retrieval of commands excluding analysis commands * Addressing bug on retrieving information from redis * Enabling the command register endpoint to provide if the command is analysis only * Improving study list speed (#2123) * init commit * reading if ... * fixing tests * rethinking listint * split SQL * reseting sql * ignoring prep_total_samples * finishing changes * adding comment: @ElDeveloper * adding message: @josenavas * addressing @ElDeveloper, @josenavas @jdereus comments * fixing download buttons show (#2127) * fixing download buttons show * addressing @ElDeveloper comment * connecting tutorials to CMI * adding link in main page * fixing prep getting-started * mv complex designs * fix idents * Addressing @antgonza's comments * Addressing @wasade's comments * Supporting multiple choice * Adding documentation * limiting number of jobs retrieved * Modifying handler to pass allow_change_optionals * returning optional parameters * Addressing bug found by @antgonza * Enabling changing the default parameters * Adding correct class * Allowing user to change default parameters * Fixing bug with commands listing * Enabling arbitrary htmls in the summary * Prepping for merge hell * Addressing @wasade's comments * Addressing @antgonza's comment * Moving tests around and deleting unused code * Analysis refactor sync w master 2 (#2141) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1805 * adding button * fix errors * fix #1816 * fixing failures tests * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * fixing timestamp * rm formats * st -> pt * moving to ISO 8601 - wow :'( * fix errors * addressing @wasade comments * fixing private/public studies * sanbox-to-sandbox * flake8 * fixing other issues * adding share documentation * psycopg2 <= 2.7 * psycopg2 < 2.7 * init commit * fixing errors * fixing errors due to update * Making the download work * Fixing tests * working status * adding tags, the right way! * fix error * Addressing @antgonza's comments * Adding missing test * Ignoring tgz - thanks @antgonza * addressing @josenavas comments * list study tags * fix error * adding tags to public * adding docs * addressing @wasade comment * addressing @josenavas and @wasade comments * addressing @wasade request * fix #2091 * option 2: @ElDeveloper and @josenavas * A minimal REST API for Qiita (#2094) * TST: Add initial test cases for study handler * ENH: Add initial study rest api * API: test if a study exists * ENH: oauth2 forced * Get back basic study deets * TST: test for samples collection * API: rest get sample IDs from a study * ENH: samples/info handler * broken routes * API: request sample metadata * ENH/API: Add methods to check for a study person * ENH/API: Add POST methods for study person * TST: Add tests for from_name_and_affiliation * TST: study creation * BUG: Add headers to tests * ENH: create study * Adjust GET on study description * API: Add endpoints for preparation creation * TST: 200 :D * TST: Correctly verify study instantiation * TST: prep artifact creation * ENH/API: associate artifacts with a preparation * TST: test study statys * ENH: study status * Removed trailing whitespace * STY: PEP8 * MAINT: refactor, centralize setup boilerplate * REFACTOR: Remove repeated code * DOC: Remove unnecessary comments * REFACTOR: Missing removal of pattern * STY: Fix PEP8 errors * BUG: Incorrectly changed error code * BUG/TST: Fix typo in tests * Addressing an @antgonza comment * Another @antgonza comment * RVW: Address review comments * ENH: Cleanup webserver and name-spaces * ENH: Improve error messages * ENH: Add more descriptive error message * TST: Exercise different argument types * DOC: Add documentation for REST API * ENH: Remove extra comma * ENH/DOC: update/add samples to sample information via rest (#2097) * Changing how artifact visibility works (#2098) * changing how artifact visibility works * fixing code * fix errors * fixing edit check access * fix * fix #2086 * flak8 * addressing @ElDeveloper comments + fixes * adding the final changes * fix failures * get_qiita_version -> generate_biom_and_metadata_release * download raw data * adding missing empty files * Adding endpoint to retrieve list of person (#2103) * Adding missing endpoint * Addressing @ElDeveloper comment * fix #2086 (#2102) * fix #2086 * flak8 * addressing @ElDeveloper comments + fixes * adding the final changes * fix failures * get_qiita_version -> generate_biom_and_metadata_release * addressing @wasade comments and fix errors * fix error? * rm vfabu + addressing @wasade and @josenavas comments + fix errors * just being dumb! * Deblur quality mention (#2107) * Revised rst values used for section headers (#2108) * Fixing latlongs (#2120) * public studies are being shown in the user own studies * fix #2069 - adding tests * flake8 * Improving study list speed (#2123) * init commit * reading if ... * fixing tests * rethinking listint * split SQL * reseting sql * ignoring prep_total_samples * finishing changes * adding comment: @ElDeveloper * adding message: @josenavas * addressing @ElDeveloper, @josenavas @jdereus comments * fixing download buttons show (#2127) * fixing download buttons show * addressing @ElDeveloper comment * connecting tutorials to CMI * adding link in main page * fixing prep getting-started * mv complex designs * fix idents * limiting number of jobs retrieved * Deleting unused code * Fixing travis * Flake8-ing * Fixing travis * Removing unused code * Trying one thing on travis * Trying one thing on travis * Trying to debug * Still debugging * ... * Config biom plugin * Creating the dir * Speed up server by not building the docs * More playing with travis * Fixing config file * Trying with a diff conf file * Fixing path * Adding a print to see the error * Adding another print * Removing debugging bits * Runnning webserver only if running tests * Fixing last failing errors * Removing debugging bits * Fixing tests * fixing config * Fixing stochastic tests * Fixing tests (#2142) * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * Changing bdiv metrics to single choice * Fixing patch * Various small fixes to be able to run tests on the plugins * Adding private module * Fixing processing job completion * Fixing patch 52 * Fixing call * Fixing complete * small fixes * Adding processing handlers * Fixing url and bug on processing job workflow * Adding the private script runner * Adding is_analysis column to the command * Adding retrieval of commands excluding analysis commands * Addressing bug on retrieving information from redis * Enabling the command register endpoint to provide if the command is analysis only * Addressing @antgonza's comments * Addressing @wasade's comments * Supporting multiple choice * Adding documentation * Modifying handler to pass allow_change_optionals * returning optional parameters * Addressing bug found by @antgonza * Enabling changing the default parameters * Adding correct class * Allowing user to change default parameters * Fixing bug with commands listing * Enabling arbitrary htmls in the summary * Prepping for merge hell * Addressing @wasade's comments * Fixing qiita_db tests * Addressing @antgonza's comment * Solving a few more tests * Moving tests around and deleting unused code * Deleting unused code * Fixing travis * Flake8-ing * Fixing travis * Removing unused code * Trying one thing on travis * Trying one thing on travis * Trying to debug * Still debugging * ... * Config biom plugin * Creating the dir * Speed up server by not building the docs * More playing with travis * Fixing config file * Trying with a diff conf file * Fixing path * Adding a print to see the error * Adding another print * Removing debugging bits * Runnning webserver only if running tests * Fixing last failing errors * Removing debugging bits * Fixing tests * fixing config * Fixing stochastic tests * Extending post test call to accept analysis * Extending post test call to accept analysis (#2144) * some minor fixes * adding timer * nose-timer to pip command * nose-timer to pip command * active_children test * improving study listing // artifact info * adding code to both list * fix js length * 300000 * multiprocessing->active_children * 2400000 * 600000 * Trying to fix the tests * Submit the job on commit * Reverting last change * improving sql and display * improving sql * adding some of the requested improvements * addressing comments * addressing @ElDeveloper and @wasade comments * Registering commands after updating plugins (#2156) * Adding codecov (#2157) * Testing codecov for speeding up tests * Fixing test * Fixing ware test * Adding codecov badge * Removing coveralls * Trying to fix the failing test * Expand html summary setter (#2150) * Changing the HTML setter to add support dirs * Extending the patch operation * Fixing remaining calls to html_summary_fp set * Fixing test * Deleting unused function * Removing unused import * Fixing tests * Fixing failing test * Fix artifact type creation (#2158) * Changing the HTML setter to add support dirs * Extending the patch operation * Fixing remaining calls to html_summary_fp set * Fixing test * Deleting unused function * Removing unused import * Fixing tests * Fixing failing test * Creating the mountpoint of the new type * Cleaning up patch and fixing bug * Fixing tests * Adding analysis to the list of acceptable parameter types * Adding the provenance parameter automatically * Adding qzv fileapth type (#2159) * Adding qzv fileapth type * Fixing test * fixing tests * adding ORDER * to debug * splitting the plugin registration * Analysis refactor misc (#2161) * Fixing race condition * Removing unused code and fixing typo * Hiding job step if success * Showing loading gif and fixing typo * fix tests * to debug * improving delete speed (#2165) * improving delete speed * more sleep! * fix artifact summary issue (#2163) * fix artifact summary issue * sleep * fixing errors 1 * missing ] * fix error * barcode_type 8, defaults * WIP: using new info to merge bioms (#2167) * using new info to merge bioms * fix errors * flake8 * fix 55.py (#2168) * fix 55.py * fixing small bug in test env * addressing @ElDeveloper comments * config_test.cfg * rm /test_data/ from folders * fixing * releases * working_dir/releases * working_dir/ * Fix analysis artifact name (#2169) * fix analysis artifact name * fix analysis artifact name * params -> md5 * ignoring params and target subfragment * flake8 * remonving parent_processing * Improving analysis listing (#2170) * improving analysis listing * addressing @ElDeveloper comments * Improving analysis listing b (#2175) * improving analysis listing * addressing @ElDeveloper comments * adding list artifacts * changing names * fixing . -> , * only_biom * actually using only_biom * sleep * moving sleep * import sleep * sleep(0.8) * sleep(1) * addressing @ElDeveloper comments * flake8 * is_qiita_job * job_info -> ji * addressing @ElDeveloper comments * Fix branch (#2176) * improving analysis listing * addressing @ElDeveloper comments * adding list artifacts * changing names * fixing . -> , * only_biom * actually using only_biom * sleep * moving sleep * import sleep * sleep(0.8) * sleep(1) * addressing @ElDeveloper comments * flake8 * is_qiita_job * job_info -> ji * addressing @ElDeveloper comments * fixing branch

antgonza and others added 30 commits January 2, 2017 11:42

fix qiita-spots#1505

3601c29

improving some GUI stuff

0d6788e

improving some GUI stuff - missing lines

12406cc

pull upstream master

958fcbe

addressing all comments

a57ef23

ready for review

2ead7a6

fix qiita-spots#1987

73a78e7

Merge pull request qiita-spots#2036 from antgonza/fix-1505

e64a22a

Fix 1505

Merge pull request qiita-spots#2047 from antgonza/fix-1987

0dcae8b

fix qiita-spots#1987

initial commit

4a5bbbc

requested changes

f99975c

Merge pull request qiita-spots#2049 from antgonza/add-processing-sugg…

ed899a8

…estions Processing recommendations and pipelines

fix filter job list

d508320

Merge pull request qiita-spots#2050 from antgonza/fix-filter-job-list

025cc1e

fix filter job list

Fixing server cert (qiita-spots#2051)

599bcde

fix get_studies

d12ccfe

flake8

b33983b

fix qiita-spots#503

b4f1b1f

fix qiita-spots#2010

62a1b93

fix qiita-spots#1913

2e36141

fix errors

e006e20

Merge pull request qiita-spots#2052 from antgonza/fix-get_studies

c174693

fix get_studies

Merge pull request qiita-spots#2053 from antgonza/fix-by-blinking

131dd6a

Fix by blinking

addressing @josenavas comment

ccb55bd

flake8

dfe2e83

Merge pull request qiita-spots#2056 from antgonza/fix-1913

15fcceb

fix qiita-spots#1913

fix qiita-spots#1010

7f97f2a

fix qiita-spots#1066 (qiita-spots#2058)

9eb9dbb

addressing @josenavas comments

23104d7

fix qiita-spots#1961

1f1e826

josenavas added 20 commits March 15, 2017 11:15

Various small fixes to be able to run tests on the plugins

a77b040

Merging with master

ff9eda9

Adding private module

6145976

Fixing processing job completion

7153efb

Fixing patch 52

46dd73d

Fixing call

08eafaa

Fixing complete

b1a3e99

small fixes

c9580b7

Solving merge conflicts

85d4aa7

Solving merge conflicts

1ffa231

Merge branch 'analysis-refactor-gui-part5' into analysis-refactor-gui…

9e14cc6

…-part6

Merge branch 'analysis-refactor-gui-part6' into analysis-refactor-gui…

4cd34d2

…-part7

Solving merge conflicts

bf33527

Adding processing handlers

3529556

Fixing url and bug on processing job workflow

ca5a331

Adding the private script runner

b934d68

Adding is_analysis column to the command

fa00d60

Adding retrieval of commands excluding analysis commands

b2ac959

Addressing bug on retrieving information from redis

0326a63

Enabling the command register endpoint to provide if the command is a…

cd6b61c

…nalysis only

josenavas changed the title ~~WIP - Analysis refactor gui part7~~ Analysis refactor gui part7 May 16, 2017

antgonza reviewed May 16, 2017

View reviewed changes

Addressing @antgonza's comments

cccb1d4

josenavas commented May 16, 2017

View reviewed changes

wasade reviewed May 16, 2017

View reviewed changes

Addressing @wasade's comments

0a584f3

josenavas commented May 16, 2017

View reviewed changes

antgonza merged commit 39a03e3 into qiita-spots:analysis-refactor May 17, 2017

Analysis refactor gui part7 #2117

Analysis refactor gui part7 #2117

Conversation

josenavas commented Apr 25, 2017

josenavas commented May 16, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josenavas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wasade left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josenavas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment