Skip to content

Commit bf2299e

Browse files
josenavasantgonza
josenavas
authored and
antgonza
committed
Fixing tests (#2142)
* Removing qiita ware code that will not be used anymore * Organizing the handlers and new analysis description page * Connecting the analysis creation and making interface responsive * Addressing @antgonza's comments * Initial artifact GUI refactor * Removing unused code * Adding can_edit call to the analysis * Fixing artifact rest API since not all artifacts have study * Adding can_be_publicized call to analysis * Adding QiitaHTTPError to handle errors gracefully * Adding safe_execution contextmanager * Fixing typo * Adding qiita test checker * Adapting some artifact handlers * Abstracting the graph reloading and adding some documentation * Fixing typo * Fixing changing artifact visibility * Fixing delete * Fixing artifact deletion * Adding default parameters to the commands * Fixing processing page * Fixing variable name * Changing bdiv metrics to single choice * Fixing patch * Various small fixes to be able to run tests on the plugins * Adding private module * Fixing processing job completion * Fixing patch 52 * Fixing call * Fixing complete * small fixes * Adding processing handlers * Fixing url and bug on processing job workflow * Adding the private script runner * Adding is_analysis column to the command * Adding retrieval of commands excluding analysis commands * Addressing bug on retrieving information from redis * Enabling the command register endpoint to provide if the command is analysis only * Addressing @antgonza's comments * Addressing @wasade's comments * Supporting multiple choice * Adding documentation * Modifying handler to pass allow_change_optionals * returning optional parameters * Addressing bug found by @antgonza * Enabling changing the default parameters * Adding correct class * Allowing user to change default parameters * Fixing bug with commands listing * Enabling arbitrary htmls in the summary * Prepping for merge hell * Addressing @wasade's comments * Fixing qiita_db tests * Addressing @antgonza's comment * Solving a few more tests * Moving tests around and deleting unused code * Deleting unused code * Fixing travis * Flake8-ing * Fixing travis * Removing unused code * Trying one thing on travis * Trying one thing on travis * Trying to debug * Still debugging * ... * Config biom plugin * Creating the dir * Speed up server by not building the docs * More playing with travis * Fixing config file * Trying with a diff conf file * Fixing path * Adding a print to see the error * Adding another print * Removing debugging bits * Runnning webserver only if running tests * Fixing last failing errors * Removing debugging bits * Fixing tests * fixing config * Fixing stochastic tests
1 parent 8d7ad00 commit bf2299e

File tree

21 files changed

+120
-672
lines changed

21 files changed

+120
-672
lines changed

‎.travis.yml‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,15 @@ install:
3030
- pip install sphinx sphinx-bootstrap-theme coveralls 'ipython[all]==2.4.1'
3131
- travis_retry pip install . --process-dependency-links
3232
- 'echo "backend: Agg" > matplotlibrc'
33-
script:
33+
# Install the biom plugin so we can run the analysis tests
34+
- pip install https://github.com/qiita-spots/qiita_client/archive/master.zip
35+
- pip install https://github.com/qiita-spots/qtp-biom/archive/master.zip --process-dependency-links
36+
- export QIITA_SERVER_CERT=`pwd`/qiita_core/support_files/server.crt
37+
- mkdir ~/.qiita_plugins
38+
- cp $PWD/qiita_core/support_files/BIOM\ type_2.1.4.conf ~/.qiita_plugins
39+
before_script:
40+
# Some of the tests rely on the plugin system to complete successfuly.
41+
# Thus, we need a qiita webserver running to be able to execute the tests.
3442
- export MOI_CONFIG_FP=`pwd`/qiita_core/support_files/config_test.cfg
3543
- if [ ${TRAVIS_PULL_REQUEST} == "false" ]; then
3644
export QIITA_CONFIG_FP=`pwd`/qiita_core/support_files/config_test_travis.cfg;
@@ -39,6 +47,12 @@ script:
3947
- ipython profile create qiita-general --parallel
4048
- qiita-env start_cluster qiita-general
4149
- qiita-env make --no-load-ontologies
50+
- |
51+
if [ ${TEST_ADD_STUDIES} == "False" ]; then
52+
qiita pet webserver --no-build-docs start &
53+
fi
54+
script:
55+
- sleep 5
4256
- if [ ${TEST_ADD_STUDIES} == "True" ]; then test_data_studies/commands.sh ; fi
4357
- if [ ${TEST_ADD_STUDIES} == "True" ]; then qiita-cron-job ; fi
4458
- if [ ${TEST_ADD_STUDIES} == "False" ]; then qiita-test-install ; fi

‎qiita_core/support_files/BIOM type_2.1.4.conf‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
[main]
2+
NAME = BIOM type
3+
VERSION = 2.1.4
4+
DESCRIPTION = The Biological Observation Matrix format
5+
ENVIRONMENT_SCRIPT = source activate qtp-biom
6+
START_SCRIPT = start_biom
7+
PLUGIN_TYPE = artifact definition
8+
PUBLICATIONS =
9+
10+
[oauth2]
11+
SERVER_CERT = /home/travis/miniconda3/envs/qiita/lib/python2.7/site-packages/qiita_core/support_files/server.crt
12+
CLIENT_ID = dHgaXDwq665ksFPqfIoD3Jt8KRXdSioTRa4lGa5mGDnz6JTIBf
13+
CLIENT_SECRET = xqx61SD4M2EWbaS0WYv3H1nIemkvEAMIn16XMLjy5rTCqi7opCcWbfLINEwtV48bQ

‎qiita_core/support_files/config_test.cfg‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ LOG_DIR =
2323
REQUIRE_APPROVAL = True
2424

2525
# Base URL: DO NOT ADD TRAILING SLASH
26-
BASE_URL = https://localhost
26+
BASE_URL = https://localhost:21174
2727

2828
# Download path files
2929
UPLOAD_DATA_DIR = /tmp/
@@ -35,7 +35,7 @@ WORKING_DIR = /tmp/
3535
MAX_UPLOAD_SIZE = 100
3636

3737
# Path to the base directory where the data files are going to be stored
38-
BASE_DATA_DIR =
38+
BASE_DATA_DIR = /home/travis/miniconda3/envs/qiita/lib/python2.7/site-packages/qiita_db/support_files/test_data/
3939

4040
# Valid upload extension, comma separated. Empty for no uploads
4141
VALID_UPLOAD_EXTENSION = fastq,fastq.gz,txt,tsv,sff,fna,qual

‎qiita_db/artifact.py‎

Lines changed: 15 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -311,8 +311,7 @@ def create(cls, filepaths, artifact_type, name=None, prep_template=None,
311311
# them execute a set of common operations. Declare functions to avoid
312312
# code duplication. These functions should not be used outside of the
313313
# create function, hence declaring them here
314-
def _common_creation_steps(atype, cmd_id, data_type, cmd_parameters,
315-
fps, mv_files):
314+
def _common_creation_steps(atype, cmd_id, data_type, cmd_parameters):
316315
gen_timestamp = datetime.now()
317316
visibility_id = qdb.util.convert_to_id("sandbox", "visibility")
318317
atype_id = qdb.util.convert_to_id(atype, "artifact_type")
@@ -328,15 +327,6 @@ def _common_creation_steps(atype, cmd_id, data_type, cmd_parameters,
328327
cmd_parameters, visibility_id, atype_id, False]
329328
qdb.sql_connection.TRN.add(sql, sql_args)
330329
a_id = qdb.sql_connection.TRN.execute_fetchlast()
331-
# Associate the artifact with its filepaths
332-
fp_ids = qdb.util.insert_filepaths(
333-
fps, a_id, atype, "filepath",
334-
move_files=mv_files, copy=(not mv_files))
335-
sql = """INSERT INTO qiita.artifact_filepath
336-
(artifact_id, filepath_id)
337-
VALUES (%s, %s)"""
338-
sql_args = [[a_id, fp_id] for fp_id in fp_ids]
339-
qdb.sql_connection.TRN.add(sql, sql_args, many=True)
340330
qdb.sql_connection.TRN.execute()
341331

342332
return cls(a_id)
@@ -398,8 +388,7 @@ def _associate_with_analysis(instance, analysis_id):
398388

399389
instance = _common_creation_steps(
400390
artifact_type, processing_parameters.command.id,
401-
dtypes.pop(), processing_parameters.dump(), filepaths,
402-
move_files)
391+
dtypes.pop(), processing_parameters.dump())
403392

404393
_associate_with_study(instance, study_id)
405394
else:
@@ -411,8 +400,7 @@ def _associate_with_analysis(instance, analysis_id):
411400
if len(dtypes) > 1 else dtypes.pop())
412401
instance = _common_creation_steps(
413402
artifact_type, processing_parameters.command.id,
414-
data_type, processing_parameters.dump(), filepaths,
415-
move_files)
403+
data_type, processing_parameters.dump())
416404
_associate_with_analysis(instance, analysis_id)
417405

418406
# Associate the artifact with its parents
@@ -436,20 +424,28 @@ def _associate_with_analysis(instance, analysis_id):
436424
# This artifact is uploaded by the user in the
437425
# processing pipeline
438426
instance = _common_creation_steps(
439-
artifact_type, None, prep_template.data_type(), None,
440-
filepaths, move_files)
427+
artifact_type, None, prep_template.data_type(), None)
441428
# Associate the artifact with the prep template
442429
prep_template.artifact = instance
443430
# Associate the artifact with the study
444431
_associate_with_study(instance, prep_template.study_id)
445432
else:
446433
# This artifact is an initial artifact of an analysis
447434
instance = _common_creation_steps(
448-
artifact_type, None, data_type, None, filepaths,
449-
move_files)
435+
artifact_type, None, data_type, None)
450436
# Associate the artifact with the analysis
451437
analysis.add_artifact(instance)
452438

439+
# Associate the artifact with its filepaths
440+
fp_ids = qdb.util.insert_filepaths(
441+
filepaths, instance.id, artifact_type, "filepath",
442+
move_files=move_files, copy=(not move_files))
443+
sql = """INSERT INTO qiita.artifact_filepath
444+
(artifact_id, filepath_id)
445+
VALUES (%s, %s)"""
446+
sql_args = [[instance.id, fp_id] for fp_id in fp_ids]
447+
qdb.sql_connection.TRN.add(sql, sql_args, many=True)
448+
453449
if name:
454450
instance.name = name
455451

‎qiita_db/handlers/tests/test_plugin.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,9 @@ def test_get(self):
5656
'for performing microbiome analysis from raw DNA '
5757
'sequencing data',
5858
'commands': ['Split libraries FASTQ', 'Split libraries',
59-
'Pick closed-reference OTUs'],
59+
'Pick closed-reference OTUs', 'Summarize Taxa',
60+
'Beta Diversity', 'Alpha Rarefaction',
61+
'Single Rarefaction'],
6062
'publications': [{'DOI': '10.1038/nmeth.f.303',
6163
'PubMed': '20383131'}],
6264
'default_workflows': ['FASTQ upstream workflow',

‎qiita_db/metadata_template/test/test_prep_template.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -930,7 +930,6 @@ def test_create_data_type_id(self):
930930
def test_create_warning(self):
931931
"""Warns if a required columns is missing for a given functionality
932932
"""
933-
fp_count = qdb.util.get_count("qiita.filepath")
934933
del self.metadata['barcode']
935934
pt = npt.assert_warns(
936935
qdb.exceptions.QiitaDBWarning,

‎qiita_db/test/test_analysis.py‎

Lines changed: 2 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,8 @@ def tearDown(self):
6666
def _wait_for_jobs(self, analysis):
6767
for j in analysis.jobs:
6868
wait_for_processing_job(j.id)
69+
if j.status == 'error':
70+
print j.log.msg
6971

7072
def _create_analyses_with_samples(self, user='demo@microbio.me',
7173
merge=False):
@@ -129,21 +131,11 @@ def test_can_be_publicized(self):
129131
analysis = qdb.analysis.Analysis(1)
130132
self.assertFalse(analysis.can_be_publicized)
131133
a4 = qdb.artifact.Artifact(4)
132-
a5 = qdb.artifact.Artifact(5)
133-
a6 = qdb.artifact.Artifact(6)
134134

135135
a4.visibility = 'public'
136-
self.assertFalse(analysis.can_be_publicized)
137-
138-
a5.visibility = 'public'
139-
self.assertFalse(analysis.can_be_publicized)
140-
141-
a6.visibility = 'public'
142136
self.assertTrue(analysis.can_be_publicized)
143137

144138
a4.visibility = 'private'
145-
a5.visibility = 'private'
146-
a6.visibility = 'private'
147139
self.assertFalse(analysis.can_be_publicized)
148140

149141
def test_add_artifact(self):

‎qiita_db/test/test_setup.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ def test_study_experimental_factor(self):
3636
self.assertEqual(get_count("qiita.study_experimental_factor"), 1)
3737

3838
def test_filepath(self):
39-
self.assertEqual(get_count("qiita.filepath"), 26)
39+
self.assertEqual(get_count("qiita.filepath"), 25)
4040

4141
def test_filepath_type(self):
4242
self.assertEqual(get_count("qiita.filepath_type"), 21)
@@ -66,7 +66,7 @@ def test_analysis(self):
6666
self.assertEqual(get_count("qiita.analysis"), 10)
6767

6868
def test_analysis_filepath(self):
69-
self.assertEqual(get_count("qiita.analysis_filepath"), 2)
69+
self.assertEqual(get_count("qiita.analysis_filepath"), 1)
7070

7171
def test_analysis_sample(self):
7272
self.assertEqual(get_count("qiita.analysis_sample"), 31)

0 commit comments

Comments
 (0)