Skip to content

[FIX] qc_report consuming all db connections + queue jobs timing out#332

Merged
DESm1th merged 5 commits intoTIGRLab:masterfrom
DESm1th:qc_report_fix
Jul 19, 2022
Merged

[FIX] qc_report consuming all db connections + queue jobs timing out#332
DESm1th merged 5 commits intoTIGRLab:masterfrom
DESm1th:qc_report_fix

Conversation

@DESm1th
Copy link
Copy Markdown
Contributor

@DESm1th DESm1th commented Jul 7, 2022

After the move to the new dynamic QC pages tons of QC jobs were submitted to generate metrics that were (for some reason) missing from the QC folders. This large number of jobs consumed all available database connections. This was because SQLAlchemy opens a transaction whenever database reads happen and doesn't close them until the next commit/rollback. dm_qc_report was reading some values from the database prior to generating metrics, which left connections open 'idle in transaction' for 30+ minutes while the metrics were created.

This PR:

  • Adds a decorator that will ensure a transaction ends after a function it wraps exits (8e248ce)
  • Refactors dm_qc_report a bit to ensure all transactions are closed before metric generation happens (20a3c5f)
  • Fixes a bug where one DTI file missing its bvec/bval file could crash the whole script (a0d7604)
  • Increases the memory requested for queue jobs, as it turns out that newer studies have data large enough to require more than 1G of memory per job (88c2bfb)

@DESm1th DESm1th merged commit 95fab20 into TIGRLab:master Jul 19, 2022
@DESm1th DESm1th deleted the qc_report_fix branch July 19, 2022 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant