Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Transaction #1318

Merged
merged 94 commits into from
Jul 9, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
2fc1a3d
Empty commit for the PR
josenavas Jul 1, 2015
269d340
Adding transaction object, applying singleton pattern and adding tests
josenavas Jul 2, 2015
5f813c2
Fixing environment manager so we can drop/make environments
josenavas Jul 2, 2015
a619d34
Fixing the base object
josenavas Jul 2, 2015
d9c7441
Removing conn_handler parameter from check_table_cols
josenavas Jul 2, 2015
60078b5
Fixing user object
josenavas Jul 2, 2015
a4949de
Fixing call to check_table_cols
josenavas Jul 2, 2015
178d70d
Fixing comments
josenavas Jul 2, 2015
05f1c56
Merge branch 'pr-1318-transaction' into pr-1318-base-user
josenavas Jul 2, 2015
b14b0ab
transaction -> TRN
josenavas Jul 2, 2015
ee4a84e
Fixing analysis and job calls so tests pass
josenavas Jul 2, 2015
7712758
Merge pull request #1320 from josenavas/pr-1318-transaction
ElDeveloper Jul 2, 2015
673a0a4
Addressing comments
josenavas Jul 2, 2015
7785805
Adding execute_fetchindex and tests
josenavas Jul 2, 2015
f610409
execute()[-1] -> execute_fetchindex()
josenavas Jul 2, 2015
f0a34fc
Merge pull request #1323 from josenavas/pr-1318-base-user
ElDeveloper Jul 2, 2015
62d2e9c
Fixing util
josenavas Jul 3, 2015
c2a4293
Fixing calls to util functions that changed the API
josenavas Jul 3, 2015
a6a176c
Fixing small bug
josenavas Jul 3, 2015
1d3c0fe
Fixing error in get_lat_longs
josenavas Jul 3, 2015
2e04a1d
Addressing comments
josenavas Jul 3, 2015
eb07511
Fixing base template
josenavas Jul 3, 2015
52d65a7
Adding post commit/rollback functionality
josenavas Jul 4, 2015
7a4aedd
Addressing comments
josenavas Jul 4, 2015
0bc8638
Ensuring FS-DB sync with post commit/rollback funcs
josenavas Jul 4, 2015
90b985d
Addressing comments
josenavas Jul 4, 2015
ad9f87e
Adding comment
josenavas Jul 4, 2015
c0c72ff
Adding execute_flatten function
josenavas Jul 4, 2015
49c302f
Make use of fetchflatten
josenavas Jul 4, 2015
703c01f
Improving add function methods
josenavas Jul 4, 2015
8d18255
Removing unused import
josenavas Jul 4, 2015
e4764b2
Merge pull request #1324 from josenavas/pr-1318-util
ElDeveloper Jul 4, 2015
534a109
Merge branch 'transaction' of https://github.com/biocore/qiita into p…
josenavas Jul 4, 2015
27cc242
Fixing base template
josenavas Jul 5, 2015
5273385
Fixing sample template
josenavas Jul 5, 2015
76a35f6
Fixing ST tests
josenavas Jul 5, 2015
417ea55
Fixing prep template
josenavas Jul 5, 2015
4a0dcf8
Fixing PT tests
josenavas Jul 5, 2015
40345ab
Fixing checks in the subclass
josenavas Jul 5, 2015
e329982
Fixing base class tests
josenavas Jul 5, 2015
0fc73a0
Fixing qiita_db/data.py
josenavas Jul 5, 2015
86923a7
fixing pep8 errors and test in qiita_pet
josenavas Jul 5, 2015
b8a73ee
Fixing comments
josenavas Jul 6, 2015
d5c16d5
Merge pull request #1327 from josenavas/pr-1318-data
ElDeveloper Jul 6, 2015
8b481d2
Addressing comment
josenavas Jul 6, 2015
366f8ab
fixing analysis obj
josenavas Jul 6, 2015
4338726
Fixing tests
josenavas Jul 6, 2015
186f955
Fixing job object
josenavas Jul 6, 2015
7f0c8d9
Merge branch 'transaction' of https://github.com/biocore/qiita into p…
josenavas Jul 7, 2015
d1c0025
Adding the 'd' character that @ElDeveloper requested in the other PR …
josenavas Jul 7, 2015
b89dcd9
Fixing study and studyperson
josenavas Jul 7, 2015
6506118
Fixing tests and adding more tests
josenavas Jul 7, 2015
304b50d
Adding missing test
josenavas Jul 7, 2015
e963a76
Fixing reference object
josenavas Jul 7, 2015
89f829e
Fixing meta_util.py
josenavas Jul 7, 2015
11b688e
Fixing parameters.py
josenavas Jul 7, 2015
99ea9bf
Fixing ontology object
josenavas Jul 7, 2015
5d5ee12
Fixing the logger object
josenavas Jul 7, 2015
6a45c6a
Merge pull request #1326 from josenavas/pr-1318-metadata
ElDeveloper Jul 7, 2015
5c7a6b8
Merge branch 'transaction' of https://github.com/biocore/qiita into p…
josenavas Jul 7, 2015
511c893
Merge branch 'transaction' of https://github.com/biocore/qiita into p…
josenavas Jul 7, 2015
67ae8de
Merge branch 'master' of https://github.com/biocore/qiita into pr-131…
josenavas Jul 7, 2015
507f3c3
Fixing the search engine, and adding a check in the sql connection ha…
josenavas Jul 7, 2015
da51816
Fixing commands
josenavas Jul 7, 2015
f7402ec
Merge branch 'transaction' of https://github.com/biocore/qiita into p…
josenavas Jul 7, 2015
59becaa
Reverting changes in StudyPerson.create
josenavas Jul 7, 2015
9264500
Fixing patches
josenavas Jul 7, 2015
be388d0
Fixing environment mgr
josenavas Jul 7, 2015
ff299a7
Removing code duplication in tornado_test_base
josenavas Jul 7, 2015
8954a61
Addressing comments
josenavas Jul 7, 2015
20b5c97
JOIN ... ON ... -> JOIN ... USING (...)
josenavas Jul 7, 2015
ab223ff
FROm -> FROM
josenavas Jul 7, 2015
c4e6a09
Fixing test
josenavas Jul 7, 2015
4d5ac06
pep8ing
josenavas Jul 7, 2015
f95983d
Addressing comment
josenavas Jul 7, 2015
7f1ace0
Merge pull request #1335 from josenavas/pr-1318-search-cmds
squirrelo Jul 7, 2015
1c4db37
Merge pull request #1332 from josenavas/pr-1318-analysis-job
squirrelo Jul 7, 2015
09149f5
Merge pull request #1333 from josenavas/pr-1318-study-reference
squirrelo Jul 7, 2015
c0bb23d
Merge pull request #1334 from josenavas/pr-1318-metaparatologylog
squirrelo Jul 7, 2015
a4773e1
Merge pull request #1337 from josenavas/pr-1318-patches-env-mgr
squirrelo Jul 7, 2015
0ac0c22
Fixing merge conflicts
josenavas Jul 7, 2015
f01dd8a
Adding transactionizer wrapper
josenavas Jul 7, 2015
09a0f8b
fixing test
josenavas Jul 7, 2015
236f089
Merge pull request #1338 from josenavas/transaction-mc
squirrelo Jul 7, 2015
2e588ab
Fixing import so we make sure it happens in the workers
josenavas Jul 8, 2015
4483a1f
Merge branch 'transaction' of https://github.com/biocore/qiita into t…
josenavas Jul 8, 2015
e82e3c0
Transactionize portals
josenavas Jul 8, 2015
5b4d741
Merge pull request #1340 from josenavas/pr-1318-portal
squirrelo Jul 8, 2015
f015338
Adding decorators
josenavas Jul 8, 2015
17d6fb2
Removing queue code
josenavas Jul 8, 2015
8eac15d
Transactionize everything
josenavas Jul 8, 2015
acc92e0
Yay pep8....
josenavas Jul 8, 2015
5fe0630
Merge pull request #1347 from josenavas/pr-1318-qiita-pet
squirrelo Jul 8, 2015
df93a50
Merge pull request #1353 from josenavas/pr-1318-remove-query
antgonza Jul 8, 2015
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions qiita_core/util.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
# -----------------------------------------------------------------------------
from smtplib import SMTP, SMTP_SSL, SMTPException
from future import standard_library
from functools import wraps

from qiita_core.qiita_settings import qiita_config
from qiita_db.sql_connection import SQLConnectionHandler
Expand Down Expand Up @@ -83,3 +84,13 @@ def tearDown(self):

return DecoratedClass
return class_modifier


def execute_as_transaction(func):
"""Decorator to make a method execute inside a transaction"""
@wraps(func)
def wrapper(*args, **kwargs):
from qiita_db.sql_connection import TRN
with TRN:
return func(*args, **kwargs)
return wrapper
1,126 changes: 596 additions & 530 deletions qiita_db/analysis.py

Large diffs are not rendered by default.

126 changes: 63 additions & 63 deletions qiita_db/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,10 @@
# -----------------------------------------------------------------------------

from __future__ import division

from qiita_core.exceptions import IncompetentQiitaDeveloperError
from qiita_core.qiita_settings import qiita_config
from .sql_connection import SQLConnectionHandler
from .sql_connection import TRN
from .exceptions import (QiitaDBNotImplementedError, QiitaDBUnknownIDError,
QiitaDBError)

Expand Down Expand Up @@ -131,11 +132,12 @@ def _check_id(self, id_):
the other classes. However, still defining here as there is only one
subclass that doesn't follow this convention and it can override this.
"""
conn_handler = SQLConnectionHandler()

return conn_handler.execute_fetchone(
"SELECT EXISTS(SELECT * FROM qiita.{0} WHERE "
"{0}_id=%s)".format(self._table), (id_, ))[0]
with TRN:
sql = """SELECT EXISTS(
SELECT * FROM qiita.{0}
WHERE {0}_id=%s)""".format(self._table)
TRN.add(sql, [id_])
return TRN.execute_fetchlast()

def _check_portal(self, id_):
"""Checks that object is accessible in current portal
Expand All @@ -149,15 +151,15 @@ def _check_portal(self, id_):
# assume not portal limited object
return True

conn_handler = SQLConnectionHandler()

return conn_handler.execute_fetchone(
"""SELECT EXISTS(
SELECT * from qiita.{0}
JOIN qiita.portal_type using (portal_type_id)
WHERE {1}_id = %s AND portal = %s)""".format(
self._portal_table, self._table),
[id_, qiita_config.portal])[0]
with TRN:
sql = """SELECT EXISTS(
SELECT *
FROM qiita.{0}
JOIN qiita.portal_type USING (portal_type_id)
WHERE {1}_id = %s AND portal = %s
)""".format(self._portal_table, self._table)
TRN.add(sql, [id_, qiita_config.portal])
return TRN.execute_fetchlast()

def __init__(self, id_):
r"""Initializes the object
Expand All @@ -171,14 +173,15 @@ def __init__(self, id_):
QiitaDBUnknownIDError
If `id_` does not correspond to any object
"""
self._check_subclass()
if not self._check_id(id_):
raise QiitaDBUnknownIDError(id_, self._table)
with TRN:
self._check_subclass()
if not self._check_id(id_):
raise QiitaDBUnknownIDError(id_, self._table)

if not self._check_portal(id_):
raise QiitaDBError("%s with id %d inaccessible in current portal: "
"%s" % (self.__class__.__name__, id_,
qiita_config.portal))
if not self._check_portal(id_):
raise QiitaDBError(
"%s with id %d inaccessible in current portal: %s"
% (self.__class__.__name__, id_, qiita_config.portal))

self._id = id_

Expand Down Expand Up @@ -216,18 +219,16 @@ class QiitaStatusObject(QiitaObject):
@property
def status(self):
r"""String with the current status of the analysis"""
# Check that self._table is actually defined
self._check_subclass()

# Get the DB status of the object
conn_handler = SQLConnectionHandler()
return conn_handler.execute_fetchone(
"SELECT status FROM qiita.{0}_status WHERE {0}_status_id = "
"(SELECT {0}_status_id FROM qiita.{0} WHERE "
"{0}_id = %s)".format(self._table),
(self._id, ))[0]

def _status_setter_checks(self, conn_handler):
with TRN:
sql = """SELECT status FROM qiita.{0}_status
WHERE {0}_status_id = (
SELECT {0}_status_id FROM qiita.{0}
WHERE {0}_id = %s)""".format(self._table)
TRN.add(sql, [self._id])
return TRN.execute_fetchlast()

def _status_setter_checks(self):
r"""Perform any extra checks that needed to be done before setting the
object status on the database. Should be overwritten by the subclasses
"""
Expand All @@ -242,18 +243,18 @@ def status(self, status):
status: str
The new object status
"""
# Check that self._table is actually defined
self._check_subclass()

# Perform any extra checks needed before we update the status in the DB
conn_handler = SQLConnectionHandler()
self._status_setter_checks(conn_handler)

# Update the status of the object
conn_handler.execute(
"UPDATE qiita.{0} SET {0}_status_id = "
"(SELECT {0}_status_id FROM qiita.{0}_status WHERE status = %s) "
"WHERE {0}_id = %s".format(self._table), (status, self._id))
with TRN:
# Perform any extra checks needed before
# we update the status in the DB
self._status_setter_checks()

# Update the status of the object
sql = """UPDATE qiita.{0} SET {0}_status_id = (
SELECT {0}_status_id FROM qiita.{0}_status
WHERE status = %s)
WHERE {0}_id = %s""".format(self._table)
TRN.add(sql, [status, self._id])
TRN.execute()

def check_status(self, status, exclude=False):
r"""Checks status of object.
Expand Down Expand Up @@ -285,21 +286,20 @@ def check_status(self, status, exclude=False):
Table setup:
foo: foo_status_id ----> foo_status: foo_status_id, status
"""
# Check that self._table is actually defined
self._check_subclass()

# Get all available statuses
conn_handler = SQLConnectionHandler()

statuses = [x[0] for x in conn_handler.execute_fetchall(
"SELECT DISTINCT status FROM qiita.{0}_status".format(self._table),
(self._id, ))]

# Check that all the provided statuses are valid statuses
if set(status).difference(statuses):
raise ValueError("%s are not valid status values"
% set(status).difference(statuses))

# Get the DB status of the object
dbstatus = self.status
return dbstatus not in status if exclude else dbstatus in status
with TRN:
# Get all available statuses
sql = "SELECT DISTINCT status FROM qiita.{0}_status".format(
self._table)
TRN.add(sql)
# We need to access to the results of the last SQL query,
# hence indexing using -1
avail_status = [x[0] for x in TRN.execute_fetchindex()]

# Check that all the provided status are valid status
if set(status).difference(avail_status):
raise ValueError("%s are not valid status values"
% set(status).difference(avail_status))

# Get the DB status of the object
dbstatus = self.status
return dbstatus not in status if exclude else dbstatus in status
Loading