Analysis refactor db #2040

josenavas · 2017-01-09T14:36:52Z

Doing the PR against the correct branch.

Modifies the DB to transfer the analysis to the plugin system.

Sorry for the length of the SQL patch.

antgonza

Looks good, thanks for all the comments, makes review much easier. My only extra concern is that it seems like the biom to generate the analysis is not actually separate from the analysis so we can reuse for other analyses but perhaps I'm reading this wrongly. @wasade, @ElDeveloper or @mortonjt could you take a look?

antgonza · 2017-01-10T14:37:13Z

qiita_db/support_files/patches/47.sql

+    -- However, these jobs do not have any results and all are marked either
+    -- as queued or error, although no error log has been saved. Since these
+    -- jobs are mainly useleess, we are going to remove them from the system
+    DELETE FROM qiita.analysis_job


I think these are the base for new analysis creation and done at User confirmation: https://github.com/biocore/qiita/blob/master/qiita_db/user.py#L320

antgonza · 2017-01-10T14:41:15Z

qiita_db/support_files/patches/python_patches/47.py

+        # Each element of the filepath list is a 3-tuple with filepath_id,
+        # filepath and filepath type, and we are interested in the filepath
+        # (index 1)
+        t = load_table(a.filepaths[0][1])


This assumes that all analyses were rarefied, which is not the case. You need to check some of the samples and decide if it's rarefied or not.

wasade

I dont know enough about the internals here to really comment in detail

wasade · 2017-01-11T17:15:38Z

qiita_db/support_files/patches/python_patches/47.py

+        # filepath and filepath type, and we are interested in the filepath
+        # (index 1)
+        t = load_table(a.filepaths[0][1])
+        params['depth'] = t[:, 0].sum()


suggest

depths = set(t.sum(axis='sample')) assert len(depths) == 1 depth = depths.pop()

wasade · 2017-01-11T20:59:18Z

qiita_db/support_files/patches/47.sql

+
+            -- Step 1.2: Create the single rarefaction job
+            -- Step 1.2.a: Add the row in the procesisng job table
+            -- depth 1000 -> the depth information is not stored in the database


so the depth information that gets stored here may be incorrect?

…tructure to the new one

antgonza

2 minor comments and clarification. I think this is ready for merge.

antgonza · 2017-01-16T17:36:48Z

qiita_db/support_files/patches/47.sql

+-- database backend that supports the analysis pipeline.
+-- After exploring the data on the database, we realized that
+-- there are a lot of inconsistencies in the data. Unfortunately, this
+-- makes the process of trasnferring the data from the old structure


trasnferring -> transferring

antgonza · 2017-01-16T17:39:34Z

qiita_db/support_files/patches/python_patches/47.py

+        # Create an empty BIOM table to be the new master table
+        new_table = Table([], [], [])
+        ids_map = {}
+        for a_id, samples in samples_by_artifact:


This has an extra level of complexity, right? You need to merge based on the processing params of the artifact, right?

Scratch that, this is for the already existing bioms so there we should had done that and if not we are simply recovering so we should be fine. Leaving here so we keep in mind.

Correct, we should've stored the original table and applied rarefaction afterwards. The way is done now is doing this.

antgonza · 2017-01-16T17:42:45Z

qiita_db/support_files/patches/python_patches/47.py

+                transfer_job()
+
+with TRN:
+    # Unlink the analysis from the biom table filepaths


Due to permissions in our DB, could you put this in a try/except and if it fails print the SQL commands?

antgonza · 2017-01-16T17:44:00Z

@wasade, could you take another look just to make sure that the changes make sense? Thanks!

josenavas · 2017-01-24T23:27:01Z

@wasade @mortonjt @ElDeveloper available for a review in here?

josenavas · 2017-01-26T19:35:00Z

ping @biocore/qiita-dev

wasade · 2017-01-26T19:42:35Z

qiita_db/support_files/patches/python_patches/47.py

+    TRN.add(sql, sql_args, many=True)
+
+    # Step 4: Add the new artifact types
+    sql = """INSERT INTO qiita.artifact_type (


This is a general comment. One thing which came up in Krumlov, which almost resulted in @ElDeveloper's death, was that "artifact" has a very negative meaning to biologists. We pinged Greg about it regarding its use in QIIME2 and it is under discussion for possibly changing that the name used.

Thanks - that make sense. I think we can normalize once a decision is made.

wasade · 2017-01-26T19:50:26Z

qiita_db/support_files/patches/python_patches/47.py

+                    depths.pop(), initial_biom_artifact_id, analysis,
+                    srare_cmd_id)
+                # Create the rarefied artifact
+                rarefied_biom_artifact_id = create_rarefied_biom_artifact(


is this creating a new rarefied version of the unrarefied table? since this is a random process, won't this table be different than what table currently is?

Good question - the name of the function may be misleading. The function is gathering the biom file that is attached to the analysis and creating an artifact with that file (so no new BIOM is created).

wasade · 2017-01-26T19:51:24Z

qiita_db/support_files/patches/python_patches/47.py

+                    # Beta diversity
+                    cmd_id = bdiv_cmd_id
+                    tree_fp = loads(job_data['options'])['--tree_fp']
+                    params = ('{"biom_table":%d,"tree":"%s","metrics":'


the default bdiv metrics differ -- is it the case that all analyses here have a tree?

Now that you mention it I think I remember @antgonza adding the functionality to run other metrics if the tree didn't exist for the given processing parameters. Will change to take that into account.

wasade · 2017-01-26T19:54:38Z

@antgonza @josenavas , I'm not very comfortable commenting on the PR

josenavas

Thanks @wasade !

josenavas · 2017-01-26T20:33:20Z

qiita_db/support_files/patches/python_patches/47.py

+    TRN.add(sql, sql_args, many=True)
+
+    # Step 4: Add the new artifact types
+    sql = """INSERT INTO qiita.artifact_type (


Thanks - that make sense. I think we can normalize once a decision is made.

josenavas · 2017-01-26T20:34:46Z

qiita_db/support_files/patches/python_patches/47.py

+                    depths.pop(), initial_biom_artifact_id, analysis,
+                    srare_cmd_id)
+                # Create the rarefied artifact
+                rarefied_biom_artifact_id = create_rarefied_biom_artifact(


Good question - the name of the function may be misleading. The function is gathering the biom file that is attached to the analysis and creating an artifact with that file (so no new BIOM is created).

josenavas · 2017-01-26T20:36:42Z

qiita_db/support_files/patches/python_patches/47.py

+                    # Beta diversity
+                    cmd_id = bdiv_cmd_id
+                    tree_fp = loads(job_data['options'])['--tree_fp']
+                    params = ('{"biom_table":%d,"tree":"%s","metrics":'


Now that you mention it I think I remember @antgonza adding the functionality to run other metrics if the tree didn't exist for the given processing parameters. Will change to take that into account.

josenavas · 2017-01-30T01:41:53Z

I've fixed @wasade's comments regarding the non-phylogenetic metrics.

* Analysis refactor db (#2040) * New DB structure * Adding python patch * Adding a biom so we can actually execute the patch * Fixing the patch to correctly transfer the information from the old structure to the new one * Fixing patch * Fixing patch and a few other bits to make the patch run successfully * These files are no longer needed * Droping analysis status table * Linking the analysis with all the artifacts * Fixing typo * Fixing HTML and dbschema files * Adding analyisis jobs * Adding logging column to the analysis * Addressing @antgonza's comments * Taking into account non-phylogenetic metrics in beta diversity * Analysis refactor fix analysis (#2044) * New DB structure * Adding python patch * Adding a biom so we can actually execute the patch * Fixing the patch to correctly transfer the information from the old structure to the new one * Fixing patch * Fixing patch and a few other bits to make the patch run successfully * These files are no longer needed * Removing unused code * Droping analysis status table * Linking the analysis with all the artifacts * Fixing typo * Fixing HTML and dbschema files * Adding analyisis jobs * Extending the artifact to work with the analysis * Allowing multiomics datatype * Adding private_job_submitter and modifying proc job handler to use it * Adding logging column to the analysis * Adding datatype to the analysis-processing job table * Adding REST endpoint to access the analysis metadata * Adding private jobs to plugin * Fixing typo * Fixing the processing jobs complete * Addressing @antgonza's comments * Taking into account non-phylogenetic metrics in beta diversity * Addressing @antgonza's comments * Addressing @ElDeveloper's comments * Remove old job code (#2045) * New DB structure * Adding python patch * Adding a biom so we can actually execute the patch * Fixing the patch to correctly transfer the information from the old structure to the new one * Fixing patch * Fixing patch and a few other bits to make the patch run successfully * These files are no longer needed * Removing unused code * Droping analysis status table * Linking the analysis with all the artifacts * Fixing typo * Fixing HTML and dbschema files * Adding analyisis jobs * Extending the artifact to work with the analysis * Allowing multiomics datatype * Adding private_job_submitter and modifying proc job handler to use it * Adding logging column to the analysis * Adding datatype to the analysis-processing job table * Adding REST endpoint to access the analysis metadata * Adding private jobs to plugin * Fixing typo * Fixing the processing jobs complete * Removing the old job code * Oops removed the wrong file * Addressing @antgonza's comments * Taking into account non-phylogenetic metrics in beta diversity * Addressing @antgonza's comments * Fixes Qiita db (#2046) * New DB structure * Adding python patch * Adding a biom so we can actually execute the patch * Fixing the patch to correctly transfer the information from the old structure to the new one * Fixing patch * Fixing patch and a few other bits to make the patch run successfully * These files are no longer needed * Removing unused code * Droping analysis status table * Linking the analysis with all the artifacts * Fixing typo * Fixing HTML and dbschema files * Adding analyisis jobs * Extending the artifact to work with the analysis * Allowing multiomics datatype * Adding private_job_submitter and modifying proc job handler to use it * Adding logging column to the analysis * Adding datatype to the analysis-processing job table * Adding REST endpoint to access the analysis metadata * Adding private jobs to plugin * Fixing typo * Fixing the processing jobs complete * Removing the old job code * Oops removed the wrong file * Removing QiitaStatusObject because it is not used * fixing metautil * Fixing porntal, setup and sql tests * Fixing user and util * Fixing qiita_db * Addressing @antgonza's comments * Taking into account non-phylogenetic metrics in beta diversity * Addressing @antgonza's comments * Analysis refactor master (#2070) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * New DB structure * Adding python patch * Adding a biom so we can actually execute the patch * Fixing the patch to correctly transfer the information from the old structure to the new one * Fixing patch * Fixing patch and a few other bits to make the patch run successfully * These files are no longer needed * Removing unused code * Droping analysis status table * Linking the analysis with all the artifacts * Fixing typo * Fixing HTML and dbschema files * Adding analyisis jobs * Extending the artifact to work with the analysis * Allowing multiomics datatype * Adding private_job_submitter and modifying proc job handler to use it * Adding logging column to the analysis * Adding datatype to the analysis-processing job table * Adding REST endpoint to access the analysis metadata * Adding private jobs to plugin * Fixing typo * Fixing the processing jobs complete * Removing the old job code * Oops removed the wrong file * Removing QiitaStatusObject because it is not used * fixing metautil * Fixing porntal, setup and sql tests * Fixing user and util * Fixing qiita_db * fix #1987 * Updating to master * Addressing @antgonza's comments * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * Taking into account non-phylogenetic metrics in beta diversity * fix #1959 * Addressing @antgonza's comments * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * Actually fixing merge conflicts * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * Sync-ing with master (again) (#2073) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Fix qiita ware (#2074) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * Analysis refactor GUI part1 (#2076) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * Addressing @antgonza's comments * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * Analysis refactor GUI part2 (#2077) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Removing unused code * Changing bdiv metrics to single choice * Fixing patch * Various small fixes to be able to run tests on the plugins * Adding private module * Fixing processing job completion * Fixing patch 52 * Fixing call * Fixing complete * small fixes * Analysis refactor gui part3 (#2078) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Fixing typo * Addressing @antgonza's comments * Addressing @antgonza's comments * Analysis refactor gui part4 (#2079) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Analysis refactor gui part5 (#2115) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * Changing bdiv metrics to single choice * Fixing patch * Various small fixes to be able to run tests on the plugins * Addressing @antgonza's comments * Analysis refactor gui part6 (#2116) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * fixing timestamp * rm formats * st -> pt * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * moving to ISO 8601 - wow :'( * fix errors * addressing @wasade comments * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * fixing private/public studies * Changing bdiv metrics to single choice * sanbox-to-sandbox * flake8 * Fixing patch * fixing other issues * adding share documentation * psycopg2 <= 2.7 * psycopg2 < 2.7 * Various small fixes to be able to run tests on the plugins * Solving merge conflicts * Adding processing handlers * Fixing url and bug on processing job workflow * Adding the private script runner * Adding is_analysis column to the command * Adding retrieval of commands excluding analysis commands * Addressing bug on retrieving information from redis * Enabling the command register endpoint to provide if the command is analysis only * Addressing @antgonza's comments * Addressing @wasade's comments * Supporting multiple choice * Adding documentation * Analysis refactor gui part7 (#2117) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * fixing timestamp * rm formats * st -> pt * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * moving to ISO 8601 - wow :'( * fix errors * addressing @wasade comments * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * fixing private/public studies * Changing bdiv metrics to single choice * sanbox-to-sandbox * flake8 * Fixing patch * fixing other issues * adding share documentation * psycopg2 <= 2.7 * psycopg2 < 2.7 * Various small fixes to be able to run tests on the plugins * Adding private module * Fixing processing job completion * Fixing patch 52 * Fixing call * Fixing complete * small fixes * Adding processing handlers * Fixing url and bug on processing job workflow * Adding the private script runner * Adding is_analysis column to the command * Adding retrieval of commands excluding analysis commands * Addressing bug on retrieving information from redis * Enabling the command register endpoint to provide if the command is analysis only * Addressing @antgonza's comments * Addressing @wasade's comments * Modifying handler to pass allow_change_optionals * returning optional parameters * Addressing bug found by @antgonza * Enabling changing the default parameters * Adding correct class * Allowing user to change default parameters * Fixing bug with commands listing * Enabling arbitrary htmls in the summary * Prepping for merge hell * Addressing @wasade's comments * Fixing qiita_db tests * Analysis refactor allow users change dflt params (#2136) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * fixing timestamp * rm formats * st -> pt * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * moving to ISO 8601 - wow :'( * fix errors * addressing @wasade comments * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * fixing private/public studies * Changing bdiv metrics to single choice * sanbox-to-sandbox * flake8 * Fixing patch * fixing other issues * adding share documentation * psycopg2 <= 2.7 * psycopg2 < 2.7 * Various small fixes to be able to run tests on the plugins * Adding private module * Fixing processing job completion * Fixing patch 52 * Fixing call * Fixing complete * small fixes * Adding processing handlers * Fixing url and bug on processing job workflow * Adding the private script runner * Adding is_analysis column to the command * Adding retrieval of commands excluding analysis commands * Addressing bug on retrieving information from redis * Enabling the command register endpoint to provide if the command is analysis only * Addressing @antgonza's comments * Addressing @wasade's comments * Supporting multiple choice * Adding documentation * Modifying handler to pass allow_change_optionals * returning optional parameters * Addressing bug found by @antgonza * Enabling changing the default parameters * Adding correct class * Allowing user to change default parameters * Fixing bug with commands listing * Addressing @wasade's comments * Addressing @antgonza's comment * Solving a few more tests * Arbitrary summary HTML (#2138) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1816 * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * fixing timestamp * rm formats * st -> pt * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * moving to ISO 8601 - wow :'( * fix errors * addressing @wasade comments * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * fixing private/public studies * Changing bdiv metrics to single choice * sanbox-to-sandbox * flake8 * Fixing patch * fixing other issues * adding share documentation * psycopg2 <= 2.7 * psycopg2 < 2.7 * Various small fixes to be able to run tests on the plugins * Adding private module * Fixing processing job completion * Fixing patch 52 * Fixing call * Fixing complete * small fixes * Adding processing handlers * Fixing url and bug on processing job workflow * Adding the private script runner * Adding is_analysis column to the command * Adding retrieval of commands excluding analysis commands * Addressing bug on retrieving information from redis * Enabling the command register endpoint to provide if the command is analysis only * Addressing @antgonza's comments * Addressing @wasade's comments * Supporting multiple choice * Adding documentation * Modifying handler to pass allow_change_optionals * returning optional parameters * Addressing bug found by @antgonza * Enabling changing the default parameters * Adding correct class * Allowing user to change default parameters * Fixing bug with commands listing * Enabling arbitrary htmls in the summary * Addressing @wasade's comments * Addressing @antgonza's comment * Analysis refactor sync with master - DEPENDS ON #2138 (#2139) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1805 * adding button * fix errors * fix #1816 * fixing failures tests * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * fixing timestamp * rm formats * st -> pt * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * moving to ISO 8601 - wow :'( * fix errors * addressing @wasade comments * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * fixing private/public studies * Changing bdiv metrics to single choice * sanbox-to-sandbox * flake8 * Fixing patch * fixing other issues * adding share documentation * psycopg2 <= 2.7 * psycopg2 < 2.7 * Various small fixes to be able to run tests on the plugins * Adding private module * Fixing processing job completion * Fixing patch 52 * Fixing call * Fixing complete * small fixes * init commit * fixing errors * fixing errors due to update * Making the download work * Fixing tests * working status * adding tags, the right way! * fix error * Addressing @antgonza's comments * Adding missing test * Ignoring tgz - thanks @antgonza * addressing @josenavas comments * list study tags * fix error * adding tags to public * adding docs * addressing @wasade comment * addressing @josenavas and @wasade comments * addressing @wasade request * fix #2091 * option 2: @ElDeveloper and @josenavas * A minimal REST API for Qiita (#2094) * TST: Add initial test cases for study handler * ENH: Add initial study rest api * API: test if a study exists * ENH: oauth2 forced * Get back basic study deets * TST: test for samples collection * API: rest get sample IDs from a study * ENH: samples/info handler * broken routes * API: request sample metadata * ENH/API: Add methods to check for a study person * ENH/API: Add POST methods for study person * TST: Add tests for from_name_and_affiliation * TST: study creation * BUG: Add headers to tests * ENH: create study * Adjust GET on study description * API: Add endpoints for preparation creation * TST: 200 :D * TST: Correctly verify study instantiation * TST: prep artifact creation * ENH/API: associate artifacts with a preparation * TST: test study statys * ENH: study status * Removed trailing whitespace * STY: PEP8 * MAINT: refactor, centralize setup boilerplate * REFACTOR: Remove repeated code * DOC: Remove unnecessary comments * REFACTOR: Missing removal of pattern * STY: Fix PEP8 errors * BUG: Incorrectly changed error code * BUG/TST: Fix typo in tests * Addressing an @antgonza comment * Another @antgonza comment * RVW: Address review comments * ENH: Cleanup webserver and name-spaces * ENH: Improve error messages * ENH: Add more descriptive error message * TST: Exercise different argument types * DOC: Add documentation for REST API * ENH: Remove extra comma * ENH/DOC: update/add samples to sample information via rest (#2097) * Changing how artifact visibility works (#2098) * changing how artifact visibility works * fixing code * fix errors * fixing edit check access * fix * fix #2086 * flak8 * addressing @ElDeveloper comments + fixes * adding the final changes * fix failures * get_qiita_version -> generate_biom_and_metadata_release * download raw data * adding missing empty files * Adding endpoint to retrieve list of person (#2103) * Adding missing endpoint * Addressing @ElDeveloper comment * fix #2086 (#2102) * fix #2086 * flak8 * addressing @ElDeveloper comments + fixes * adding the final changes * fix failures * get_qiita_version -> generate_biom_and_metadata_release * addressing @wasade comments and fix errors * fix error? * rm vfabu + addressing @wasade and @josenavas comments + fix errors * just being dumb! * Deblur quality mention (#2107) * Revised rst values used for section headers (#2108) * Adding processing handlers * Fixing latlongs (#2120) * public studies are being shown in the user own studies * fix #2069 - adding tests * flake8 * Fixing url and bug on processing job workflow * Adding the private script runner * Adding is_analysis column to the command * Adding retrieval of commands excluding analysis commands * Addressing bug on retrieving information from redis * Enabling the command register endpoint to provide if the command is analysis only * Improving study list speed (#2123) * init commit * reading if ... * fixing tests * rethinking listint * split SQL * reseting sql * ignoring prep_total_samples * finishing changes * adding comment: @ElDeveloper * adding message: @josenavas * addressing @ElDeveloper, @josenavas @jdereus comments * fixing download buttons show (#2127) * fixing download buttons show * addressing @ElDeveloper comment * connecting tutorials to CMI * adding link in main page * fixing prep getting-started * mv complex designs * fix idents * Addressing @antgonza's comments * Addressing @wasade's comments * Supporting multiple choice * Adding documentation * limiting number of jobs retrieved * Modifying handler to pass allow_change_optionals * returning optional parameters * Addressing bug found by @antgonza * Enabling changing the default parameters * Adding correct class * Allowing user to change default parameters * Fixing bug with commands listing * Enabling arbitrary htmls in the summary * Prepping for merge hell * Addressing @wasade's comments * Addressing @antgonza's comment * Moving tests around and deleting unused code * Analysis refactor sync w master 2 (#2141) * fix #1505 * improving some GUI stuff * improving some GUI stuff - missing lines * addressing all comments * ready for review * fix #1987 * initial commit * requested changes * fix filter job list * Fixing server cert (#2051) * fix get_studies * flake8 * fix #503 * fix #2010 * fix #1913 * fix errors * addressing @josenavas comment * flake8 * fix #1010 * fix #1066 (#2058) * addressing @josenavas comments * fix #1961 * fix #1837 * Automatic jobs & new stats (#2057) * fix #814, fix #1636 * fixing error in test-env * fixing stats.html call * adding img * addressing @josenavas comments * rm for loops * addresssing @ElDeveloper comments * generalizing this functionality * fix #1805 * adding button * fix errors * fix #1816 * fixing failures tests * fix #1959 * addressing @josenavas comments * addressing @josenavas comments * fixing error * fixed? * addressing @josenavas comments * addressing @wasade comments * fix flake8 * generate biom and metadata release (#2066) * initial commit * adding portal * addressing @josenavas comments * pid -> qiita_artifact_id * addressing @josenavas comments * addressing @ElDeveloper comments * rm 50.sql * database changes to fix 969 * adding delete * addressing @josenavas comments * addressing @ElDeveloper comments * duh! * fix generate_biom_and_metadata_release (#2072) * fix generate_biom_and_metadata_release * addressing @ElDeveloper comment * fixing timestamp * rm formats * st -> pt * moving to ISO 8601 - wow :'( * fix errors * addressing @wasade comments * fixing private/public studies * sanbox-to-sandbox * flake8 * fixing other issues * adding share documentation * psycopg2 <= 2.7 * psycopg2 < 2.7 * init commit * fixing errors * fixing errors due to update * Making the download work * Fixing tests * working status * adding tags, the right way! * fix error * Addressing @antgonza's comments * Adding missing test * Ignoring tgz - thanks @antgonza * addressing @josenavas comments * list study tags * fix error * adding tags to public * adding docs * addressing @wasade comment * addressing @josenavas and @wasade comments * addressing @wasade request * fix #2091 * option 2: @ElDeveloper and @josenavas * A minimal REST API for Qiita (#2094) * TST: Add initial test cases for study handler * ENH: Add initial study rest api * API: test if a study exists * ENH: oauth2 forced * Get back basic study deets * TST: test for samples collection * API: rest get sample IDs from a study * ENH: samples/info handler * broken routes * API: request sample metadata * ENH/API: Add methods to check for a study person * ENH/API: Add POST methods for study person * TST: Add tests for from_name_and_affiliation * TST: study creation * BUG: Add headers to tests * ENH: create study * Adjust GET on study description * API: Add endpoints for preparation creation * TST: 200 :D * TST: Correctly verify study instantiation * TST: prep artifact creation * ENH/API: associate artifacts with a preparation * TST: test study statys * ENH: study status * Removed trailing whitespace * STY: PEP8 * MAINT: refactor, centralize setup boilerplate * REFACTOR: Remove repeated code * DOC: Remove unnecessary comments * REFACTOR: Missing removal of pattern * STY: Fix PEP8 errors * BUG: Incorrectly changed error code * BUG/TST: Fix typo in tests * Addressing an @antgonza comment * Another @antgonza comment * RVW: Address review comments * ENH: Cleanup webserver and name-spaces * ENH: Improve error messages * ENH: Add more descriptive error message * TST: Exercise different argument types * DOC: Add documentation for REST API * ENH: Remove extra comma * ENH/DOC: update/add samples to sample information via rest (#2097) * Changing how artifact visibility works (#2098) * changing how artifact visibility works * fixing code * fix errors * fixing edit check access * fix * fix #2086 * flak8 * addressing @ElDeveloper comments + fixes * adding the final changes * fix failures * get_qiita_version -> generate_biom_and_metadata_release * download raw data * adding missing empty files * Adding endpoint to retrieve list of person (#2103) * Adding missing endpoint * Addressing @ElDeveloper comment * fix #2086 (#2102) * fix #2086 * flak8 * addressing @ElDeveloper comments + fixes * adding the final changes * fix failures * get_qiita_version -> generate_biom_and_metadata_release * addressing @wasade comments and fix errors * fix error? * rm vfabu + addressing @wasade and @josenavas comments + fix errors * just being dumb! * Deblur quality mention (#2107) * Revised rst values used for section headers (#2108) * Fixing latlongs (#2120) * public studies are being shown in the user own studies * fix #2069 - adding tests * flake8 * Improving study list speed (#2123) * init commit * reading if ... * fixing tests * rethinking listint * split SQL * reseting sql * ignoring prep_total_samples * finishing changes * adding comment: @ElDeveloper * adding message: @josenavas * addressing @ElDeveloper, @josenavas @jdereus comments * fixing download buttons show (#2127) * fixing download buttons show * addressing @ElDeveloper comment * connecting tutorials to CMI * adding link in main page * fixing prep getting-started * mv complex designs * fix idents * limiting number of jobs retrieved * Deleting unused code * Fixing travis * Flake8-ing * Fixing travis * Removing unused code * Trying one thing on travis * Trying one thing on travis * Trying to debug * Still debugging * ... * Config biom plugin * Creating the dir * Speed up server by not building the docs * More playing with travis * Fixing config file * Trying with a diff conf file * Fixing path * Adding a print to see the error * Adding another print * Removing debugging bits * Runnning webserver only if running tests * Fixing last failing errors * Removing debugging bits * Fixing tests * fixing config * Fixing stochastic tests * Fixing tests (#2142) * Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * Changing bdiv metrics to single choice * Fixing patch * Various small fixes to be able to run tests on the plugins * Adding private module * Fixing processing job completion * Fixing patch 52 * Fixing call * Fixing complete * small fixes * Adding processing handlers * Fixing url and bug on processing job workflow * Adding the private script runner * Adding is_analysis column to the command * Adding retrieval of commands excluding analysis commands * Addressing bug on retrieving information from redis * Enabling the command register endpoint to provide if the command is analysis only * Addressing @antgonza's comments * Addressing @wasade's comments * Supporting multiple choice * Adding documentation * Modifying handler to pass allow_change_optionals * returning optional parameters * Addressing bug found by @antgonza * Enabling changing the default parameters * Adding correct class * Allowing user to change default parameters * Fixing bug with commands listing * Enabling arbitrary htmls in the summary * Prepping for merge hell * Addressing @wasade's comments * Fixing qiita_db tests * Addressing @antgonza's comment * Solving a few more tests * Moving tests around and deleting unused code * Deleting unused code * Fixing travis * Flake8-ing * Fixing travis * Removing unused code * Trying one thing on travis * Trying one thing on travis * Trying to debug * Still debugging * ... * Config biom plugin * Creating the dir * Speed up server by not building the docs * More playing with travis * Fixing config file * Trying with a diff conf file * Fixing path * Adding a print to see the error * Adding another print * Removing debugging bits * Runnning webserver only if running tests * Fixing last failing errors * Removing debugging bits * Fixing tests * fixing config * Fixing stochastic tests * Extending post test call to accept analysis * Extending post test call to accept analysis (#2144) * some minor fixes * adding timer * nose-timer to pip command * nose-timer to pip command * active_children test * improving study listing // artifact info * adding code to both list * fix js length * 300000 * multiprocessing->active_children * 2400000 * 600000 * Trying to fix the tests * Submit the job on commit * Reverting last change * improving sql and display * improving sql * adding some of the requested improvements * addressing comments * addressing @ElDeveloper and @wasade comments * Registering commands after updating plugins (#2156) * Adding codecov (#2157) * Testing codecov for speeding up tests * Fixing test * Fixing ware test * Adding codecov badge * Removing coveralls * Trying to fix the failing test * Expand html summary setter (#2150) * Changing the HTML setter to add support dirs * Extending the patch operation * Fixing remaining calls to html_summary_fp set * Fixing test * Deleting unused function * Removing unused import * Fixing tests * Fixing failing test * Fix artifact type creation (#2158) * Changing the HTML setter to add support dirs * Extending the patch operation * Fixing remaining calls to html_summary_fp set * Fixing test * Deleting unused function * Removing unused import * Fixing tests * Fixing failing test * Creating the mountpoint of the new type * Cleaning up patch and fixing bug * Fixing tests * Adding analysis to the list of acceptable parameter types * Adding the provenance parameter automatically * Adding qzv fileapth type (#2159) * Adding qzv fileapth type * Fixing test * fixing tests * adding ORDER * to debug * splitting the plugin registration * Analysis refactor misc (#2161) * Fixing race condition * Removing unused code and fixing typo * Hiding job step if success * Showing loading gif and fixing typo * fix tests * to debug * improving delete speed (#2165) * improving delete speed * more sleep! * fix artifact summary issue (#2163) * fix artifact summary issue * sleep * fixing errors 1 * missing ] * fix error * barcode_type 8, defaults * WIP: using new info to merge bioms (#2167) * using new info to merge bioms * fix errors * flake8 * fix 55.py (#2168) * fix 55.py * fixing small bug in test env * addressing @ElDeveloper comments * config_test.cfg * rm /test_data/ from folders * fixing * releases * working_dir/releases * working_dir/ * Fix analysis artifact name (#2169) * fix analysis artifact name * fix analysis artifact name * params -> md5 * ignoring params and target subfragment * flake8 * remonving parent_processing * Improving analysis listing (#2170) * improving analysis listing * addressing @ElDeveloper comments * Improving analysis listing b (#2175) * improving analysis listing * addressing @ElDeveloper comments * adding list artifacts * changing names * fixing . -> , * only_biom * actually using only_biom * sleep * moving sleep * import sleep * sleep(0.8) * sleep(1) * addressing @ElDeveloper comments * flake8 * is_qiita_job * job_info -> ji * addressing @ElDeveloper comments * Fix branch (#2176) * improving analysis listing * addressing @ElDeveloper comments * adding list artifacts * changing names * fixing . -> , * only_biom * actually using only_biom * sleep * moving sleep * import sleep * sleep(0.8) * sleep(1) * addressing @ElDeveloper comments * flake8 * is_qiita_job * job_info -> ji * addressing @ElDeveloper comments * fixing branch

josenavas added 3 commits January 8, 2017 17:49

New DB structure

931e566

Adding python patch

a2b883e

Adding a biom so we can actually execute the patch

e737f64

antgonza requested changes Jan 10, 2017

View reviewed changes

wasade requested changes Jan 11, 2017

View reviewed changes

josenavas added 10 commits January 12, 2017 13:00

Fixing the patch to correctly transfer the information from the old s…

6bc0303

…tructure to the new one

Fixing patch

331012d

Fixing patch and a few other bits to make the patch run successfully

eabe25d

These files are no longer needed

0c40bc0

Droping analysis status table

ad4deea

Linking the analysis with all the artifacts

aa3e96f

Fixing typo

d222e08

Fixing HTML and dbschema files

39f6beb

Adding analyisis jobs

4dd357c

Adding logging column to the analysis

ab65fa2

josenavas mentioned this pull request Jan 16, 2017

Analysis refactor fix analysis #2044

Merged

antgonza approved these changes Jan 16, 2017

View reviewed changes

Addressing @antgonza's comments

43eb1ed

josenavas mentioned this pull request Jan 24, 2017

Remove old job code #2045

Merged

wasade reviewed Jan 26, 2017

View reviewed changes

josenavas commented Jan 26, 2017

View reviewed changes

Taking into account non-phylogenetic metrics in beta diversity

d016833

wasade approved these changes Jan 30, 2017

View reviewed changes

antgonza merged commit c92231c into qiita-spots:analysis-refactor Jan 30, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Analysis refactor db #2040

Analysis refactor db #2040

josenavas commented Jan 9, 2017

antgonza left a comment

antgonza Jan 10, 2017

antgonza Jan 10, 2017

wasade left a comment

wasade Jan 11, 2017

wasade Jan 11, 2017

antgonza left a comment

antgonza Jan 16, 2017

antgonza Jan 16, 2017

antgonza Jan 16, 2017

josenavas Jan 16, 2017

antgonza Jan 16, 2017

josenavas Jan 16, 2017

antgonza commented Jan 16, 2017

josenavas commented Jan 24, 2017

josenavas commented Jan 26, 2017

wasade Jan 26, 2017

josenavas Jan 26, 2017

wasade Jan 26, 2017

josenavas Jan 26, 2017

wasade Jan 26, 2017

josenavas Jan 26, 2017

wasade commented Jan 26, 2017

josenavas left a comment

josenavas Jan 26, 2017

josenavas Jan 26, 2017

josenavas Jan 26, 2017

josenavas commented Jan 30, 2017

Analysis refactor db #2040

Analysis refactor db #2040

Conversation

josenavas commented Jan 9, 2017

antgonza left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wasade left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antgonza left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antgonza commented Jan 16, 2017

josenavas commented Jan 24, 2017

josenavas commented Jan 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wasade commented Jan 26, 2017

josenavas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josenavas commented Jan 30, 2017