Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixes after 2022.04 #3199

Merged
merged 2 commits into from
May 12, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 29 additions & 17 deletions qiita_db/artifact.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,26 @@ class Artifact(qdb.base.QiitaObject):
"""
_table = "artifact"

@classmethod
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to iter over all Artifacts more often than before, thus this method.

def iter(cls):
"""Iterate over all artifacts in the database

Returns
-------
generator
Yields a `Artifact` object for each artifact in the database,
in order of ascending artifact_id
"""
with qdb.sql_connection.TRN:
sql = """SELECT artifact_id FROM qiita.{}
ORDER BY artifact_id""".format(cls._table)
qdb.sql_connection.TRN.add(sql)

ids = qdb.sql_connection.TRN.execute_fetchflatten()

for id_ in ids:
yield Artifact(id_)

@classmethod
def iter_by_visibility(cls, visibility):
r"""Iterator over the artifacts with the given visibility
Expand All @@ -79,17 +99,6 @@ def iter_by_visibility(cls, visibility):
for a_id in qdb.sql_connection.TRN.execute_fetchflatten():
yield cls(a_id)

@classmethod
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not used anywhere in the code.

def iter_public(cls):
r"""Iterator over the public artifacts available in the system

Returns
-------
generator of qiita_db.artifact.Artifact
The public artifacts available in the system
"""
return cls.iter_by_visibility('public')

@staticmethod
def types():
"""Returns list of all artifact types available and their descriptions
Expand Down Expand Up @@ -615,12 +624,15 @@ def delete(cls, artifact_id):
qdb.util.move_filepaths_to_upload_folder(
study.id, filepaths)

sql = """UPDATE qiita.prep_template
SET artifact_id = NULL
WHERE prep_template_id IN %s"""
qdb.sql_connection.TRN.add(
sql, [tuple(pt.id for a in all_artifacts
for pt in a.prep_templates)])
# there are cases that an artifact would not be linked to a
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now we have artifacts (like those generated by the sequencing processing pipeline) without prep/analysis linkages.

# study
pt_ids = [tuple([pt.id]) for a in all_artifacts
for pt in a.prep_templates]
if pt_ids:
sql = """UPDATE qiita.prep_template
SET artifact_id = NULL
WHERE prep_template_id IN %s"""
qdb.sql_connection.TRN.add(sql, pt_ids)
else:
sql = """DELETE FROM qiita.parent_artifact
WHERE artifact_id IN %s"""
Expand Down
6 changes: 2 additions & 4 deletions qiita_db/test/test_artifact.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,8 @@ def test_iter(self):
qdb.artifact.Artifact(7)]
self.assertEqual(obs, exp)

def test_iter_public(self):
obs = list(qdb.artifact.Artifact.iter_public())
exp = []
self.assertEqual(obs, exp)
exp.extend([qdb.artifact.Artifact(8), qdb.artifact.Artifact(9)])
self.assertEqual(list(qdb.artifact.Artifact.iter()), exp)

def test_create_type(self):
obs = qdb.artifact.Artifact.types()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,14 @@ Qiita currently has one active shotgun metagenomics data analysis pipeline: a pe
bowtie2 alignment step with Woltka classification using either the WoLr1 or Rep200 databases.
Below you will find more information about each of these options.

.. note::
The bowtie2 settings are maximum and minimum mismatch penalties (mp=[1,1]), a
penalty for ambiguities (np=1; default), read and reference gap open- and
extend penalties (rdg=[0,1], rfg=[0,1]), a minimum alignment score for an
alignment to be considered valid (score-min=[L,0,-0.05]), a defined number of
distinct, valid alignments (k=16), and the suppression of SAM records for
unaligned reads, as well as SAM headers (no-unal, no-hd).

The current workflow is as follows:

#. A single step per sample adapter removal (via `fastp <https://academic.oup.com/bioinformatics/article/34/17/i884/5093234>`_) and host filtering (via `minimap2 <https://academic.oup.com/bioinformatics/article/34/18/3094/4994778>`_); more information below.
Expand Down
6 changes: 1 addition & 5 deletions qiita_pet/templates/sitebase.html
Original file line number Diff line number Diff line change
Expand Up @@ -532,11 +532,7 @@
</li>
<li role="separator" class="divider"></li>
<li>
{% if qiita_sha=="" %}
<a href="#">
{% else %}
<a href="https://github.com/biocore/qiita/commit/{{qiita_sha}}">
{% end %}
<a href="https://github.com/qiita-spots/qiita/releases">
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should point to the releases page vs. the sha that is too dev oriented.

<h6>
<b>Current version:</b></br>
{{qiita_version}} {{qiita_sha[0:7]}}
Expand Down
12 changes: 6 additions & 6 deletions qiita_ware/commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,8 @@ def _ssh_session(p_url, private_key):
"""
scheme = p_url.scheme
hostname = p_url.hostname
# if port is '' Python 2.7.6 will raise an error
try:
port = p_url.port
except Exception:
port = 22

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ssh connection is currently broken due to newer python, this fixes it.

port = p_url.port
username = p_url.username

if scheme == 'scp':
Expand All @@ -61,8 +58,11 @@ def _ssh_session(p_url, private_key):

# step 2: connect to fileserver
key = RSAKey.from_private_key_file(private_key)
# we need pkeys now based on
# https://github.com/paramiko/paramiko/issues/1961
pkeys = dict(pubkeys=['rsa-sha2-256', 'rsa-sha2-512'])
ssh.connect(hostname, port=port, username=username,
pkey=key, look_for_keys=False)
pkey=key, look_for_keys=False, disabled_algorithms=pkeys)
return ssh
else:
raise ValueError(
Expand Down