Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data objects #75

Merged
merged 43 commits into from
Jun 11, 2014
Merged
Show file tree
Hide file tree
Changes from 40 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
a17f67f
Changing static methods by class methods
josenavas May 29, 2014
12a7643
Fixing allowed types in sql_args
josenavas May 29, 2014
f0e04f6
Implementation of the BaseData class and RawData class create and is_…
josenavas May 29, 2014
96a2a68
Adding util functions for checking table existance
josenavas May 29, 2014
9783d09
Adding an extra test case
josenavas May 29, 2014
f4ef0f1
Adding PreprocessedData class creation and its tests
josenavas May 29, 2014
6bf81eb
Adapting date format
josenavas May 29, 2014
626e384
Merge branch 'master' of github.com:qiime/QiiTa into data_obj
josenavas May 29, 2014
bcfa67a
Merge branch 'master' of github.com:qiime/QiiTa into data_obj
josenavas May 29, 2014
0e39691
Adding get_filepaths functionality and tests
josenavas May 30, 2014
399565d
Fix typo
josenavas May 30, 2014
64288c4
Adding some documentation
josenavas May 30, 2014
87e2ff3
Merging master
josenavas May 30, 2014
18138b9
Removing partially added delete code since we are referring its imple…
josenavas Jun 2, 2014
7d05a7d
Solving merge conflicts
josenavas Jun 2, 2014
adc686b
fixing some tests
josenavas Jun 3, 2014
eb133b2
Solving merge conflicts
josenavas Jun 4, 2014
82e42f2
Fixing tests due to merging upstream
josenavas Jun 4, 2014
6b3f7f4
Adding get_db_files_base_dir function to util
josenavas Jun 5, 2014
a21373f
Creating controlled directory for db files as discussed with @adamrp
josenavas Jun 5, 2014
fc337c6
add_filepaths now moves the files to the controlled directory
josenavas Jun 6, 2014
3a06ec0
Partially fixing tests to reflect file movement
josenavas Jun 6, 2014
f441c24
Partially fixing tests to reflect file movement round 2
josenavas Jun 6, 2014
bd1ed09
Partially fixing tests to reflect file movement aaaand 2
josenavas Jun 6, 2014
fead25c
Fixing merge conflicts
josenavas Jun 9, 2014
f77675b
100% coveragegit status
josenavas Jun 9, 2014
773d409
Making sure that we clean up the user environment after tests
josenavas Jun 9, 2014
83321c1
Adding studies property to RawData
josenavas Jun 9, 2014
ee05c54
Making all documentation raw strings
josenavas Jun 10, 2014
ad902a2
adding getters for downstream data
josenavas Jun 10, 2014
dd2bf7a
Fixing merge conflicts
josenavas Jun 10, 2014
d863d63
Adding checksums
josenavas Jun 10, 2014
2158d3c
Fixing setup.py to reflect the test_data changes
josenavas Jun 11, 2014
96fbdf5
Fixing setup.py to reflect the test_data changes
josenavas Jun 11, 2014
122eee5
Fixing setup.py to reflect the test_data changes arghhghghghghhg
josenavas Jun 11, 2014
a755ebd
Fixing doctest
josenavas Jun 11, 2014
8dc5b80
Solving py3 issues
josenavas Jun 11, 2014
b12314b
Fixing test error
josenavas Jun 11, 2014
a13c8de
Figuring out what's wrong with Travis
josenavas Jun 11, 2014
eb11486
Hopefully all tests will pass now
josenavas Jun 11, 2014
b3be47e
Using scrub_data, improving its docstring and adding tests for it
josenavas Jun 11, 2014
435f856
Using fetchall instead of fetchone
josenavas Jun 11, 2014
55008f6
Fixing last comments from @squirrelo
josenavas Jun 11, 2014
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
492 changes: 486 additions & 6 deletions qiita_db/data.py

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions qiita_db/sql_connection.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,9 @@ def _check_sql_args(self, sql_args):
otherwise it just returns the execution to the caller
"""
# Check that sql arguments have the correct type
if sql_args and type(sql_args) not in [tuple, list]:
raise TypeError("sql_args should be tuple or list. Found %s " %
type(sql_args))
if sql_args and type(sql_args) not in [tuple, list, dict]:
raise TypeError("sql_args should be tuple, list or dict. Found %s "
% type(sql_args))

@contextmanager
def _sql_executor(self, sql, sql_args=None, many=False):
Expand Down
12 changes: 6 additions & 6 deletions qiita_db/support_files/populate_test_db.sql
Original file line number Diff line number Diff line change
Expand Up @@ -47,13 +47,13 @@ INSERT INTO qiita.investigation_study (investigation_id, study_id) VALUES (1, 1)
INSERT INTO qiita.study_experimental_factor (study_id, efo_id) VALUES (1, 1);

-- Insert the raw data filepaths for study 1
INSERT INTO qiita.filepath (filepath, filepath_type_id, checksum, checksum_algorithm_id) VALUES ('s_G1_L001_sequences.fastq.gz', 1, '852952723', 1), ('s_G1_L001_sequences_barcodes.fastq.gz', 2, '852952723', 1), ('sequences.fastq.gz', 1, '852952723', 1), ('sequences_barcodes.fastq.gz', 2, '852952723', 1);
INSERT INTO qiita.filepath (filepath, filepath_type_id, checksum, checksum_algorithm_id) VALUES ('1_s_G1_L001_sequences.fastq.gz', 1, '852952723', 1), ('1_s_G1_L001_sequences_barcodes.fastq.gz', 2, '852952723', 1), ('2_sequences.fastq.gz', 1, '852952723', 1), ('2_sequences_barcodes.fastq.gz', 2, '852952723', 1);

-- Insert the raw data information for study 1
INSERT INTO qiita.raw_data (filetype_id, submitted_to_insdc) VALUES (2, FALSE), (2, TRUE);

-- Insert (link) the raw data with the raw filepaths
INSERT INTO qiita.raw_filepath (raw_data_id, filepath_id) VALUES (1, 1), (1, 2), (1, 3), (1, 4);
INSERT INTO qiita.raw_filepath (raw_data_id, filepath_id) VALUES (1, 1), (1, 2), (2, 3), (2, 4);

-- Insert (link) the study with the raw data
INSERT INTO qiita.study_raw_data (study_id, raw_data_id) VALUES (1, 1), (1, 2);
Expand Down Expand Up @@ -287,10 +287,10 @@ INSERT INTO qiita.preprocessed_data (raw_data_id, preprocessed_params_table, pre
INSERT INTO qiita.study_preprocessed_data (preprocessed_data_id, study_id) VALUES (1, 1), (2, 1);

-- Insert the preprocessed filepath for raw data 1
INSERT INTO qiita.filepath (filepath, filepath_type_id, checksum, checksum_algorithm_id) VALUES ('seqs.fna', 4, '852952723', 1), ('seqs.qual', 5, '852952723', 1);
INSERT INTO qiita.filepath (filepath, filepath_type_id, checksum, checksum_algorithm_id) VALUES ('1_seqs.fna', 4, '852952723', 1), ('1_seqs.qual', 5, '852952723', 1);

-- Insert (link) the preprocessed data with the preprocessed filepaths
INSERT INTO qiita.preprocessed_filepath (preprocessed_data_id, filepath_id) VALUES (1, 3), (1, 4);
INSERT INTO qiita.preprocessed_filepath (preprocessed_data_id, filepath_id) VALUES (1, 5), (1, 6);

-- Insert the preprocessed illumina params used for raw data 1
INSERT INTO qiita.preprocessed_sequence_illumina_params (trim_length) VALUES (151), (100);
Expand All @@ -305,7 +305,7 @@ INSERT INTO qiita.reference (reference_name, reference_version, sequence_filepat
INSERT INTO qiita.processed_params_uclust (similarity, enable_rev_strand_match, suppress_new_clusters, reference_id) VALUES (0.97, TRUE, TRUE, 1);

-- Insert the biom table filepath for processed data 1
INSERT INTO qiita.filepath (filepath, filepath_type_id, checksum, checksum_algorithm_id) VALUES ('study_1001_closed_reference_otu_table.biom', 6, '852952723', 1);
INSERT INTO qiita.filepath (filepath, filepath_type_id, checksum, checksum_algorithm_id) VALUES ('1_study_1001_closed_reference_otu_table.biom', 6, '852952723', 1);

-- Insert (link) the processed data with the processed filepath
INSERT INTO qiita.processed_filepath (processed_data_id, filepath_id) VALUES (1, 5);
INSERT INTO qiita.processed_filepath (processed_data_id, filepath_id) VALUES (1, 7);
Loading