Refactor data analysis and pipeline to workflow #391

henrikstranneheim · 2024-02-09T15:58:49Z

Description

Added

Changed

Refactor data analysis and pipeline to workflow

Fixed

How to prepare for test

ssh to hasta (depending on type of change)
activate stage: us
request trailblazer-stage on hasta: paxa

install on stage:

bash /home/proj/production/servers/resources/hasta.scilifelab.se/update-tool-stage.sh -e S_trailblazer -t trailblazer -b refactor-data_analysis-and-pipeline-to-workflow -a

ssh to clinical-db (depending on type of change)
install on stage:
bash /home/proj/production/servers/resources/clinical-db.scilifelab.se/update-trailblazer-ui-stage.sh [THIS-BRANCH-NAME]

How to test

login to ...
do ...

Expected test outcome

check that ...
Take a screenshot and attach or copy/paste the output.

Review

tests executed by
"Merge and deploy" approved by
Thanks for filling in who performed the code review and the test!

This version is a

MAJOR - when you make incompatible API changes
MINOR - when you add functionality in a backwards compatible manner
PATCH - when you make backwards compatible bug fixes or documentation/instructions

Linked to: Clinical-Genomics/cg#2932

…kflow

henrikstranneheim · 2024-02-13T07:30:35Z

cg workflow mip-dna start justhusky 👍

cg workflow mip-dna start justhusky
[INFO] 2024/02/13 08:21:57 MIP_ANALYSE - Wrote: /home/proj/stage/rare-disease/cases/justhusky/analysis/justhusky_deliverables.yaml
Action running set for case justhusky
mip-dna run started!

Feb 13 08:21:15 cg-vm1.scilifelab.se podman[2678621]: [2024-02-13 08:21:15 +0100] [6] [DEBUG] POST /api/v1/get-latest-analysis
Feb 13 08:21:15 cg-vm1.scilifelab.se podman[2678621]: 10.0.2.100 - - [13/Feb/2024:08:21:15 +0100] "POST /api/v1/get-latest-analysis HTTP/1.0" 200 564 "-" "python-requests/2.31.0"
Feb 13 08:21:57 cg-vm1.scilifelab.se podman[2678621]: [2024-02-13 08:21:57 +0100] [5] [DEBUG] POST /api/v1/get-latest-analysis
Feb 13 08:21:57 cg-vm1.scilifelab.se podman[2678621]: 10.0.2.100 - - [13/Feb/2024:08:21:57 +0100] "POST /api/v1/get-latest-analysis HTTP/1.0" 200 564 "-" "python-requests/2.31.0"
Feb 13 08:21:57 cg-vm1.scilifelab.se podman[2678621]: [2024-02-13 08:21:57 +0100] [5] [DEBUG] POST /api/v1/add-pending-analysis
Feb 13 08:21:57 cg-vm1.scilifelab.se podman[2678621]: 10.0.2.100 - - [13/Feb/2024:08:21:57 +0100] "POST /api/v1/add-pending-analysis HTTP/1.0" 201 719 "-" "python-requests/2.31.0"
Feb 13 08:22:07 cg-vm1.scilifelab.se podman[2678621]: [2024-02-13 08:22:07 +0100] [5] [DEBUG] OPTIONS /api/v1/analyses
Feb 13 08:22:07 cg-vm1.scilifelab.se podman[2678621]: 10.0.2.100 - - [13/Feb/2024:08:22:07 +0100] "OPTIONS /api/v1/analyses?pipeline=&pageSize=50&page=1 HTTP/1.0" 204 0 "https://cigrid-stage.scilifelab.se/" "Mozil>
Feb 13 08:22:07 cg-vm1.scilifelab.se podman[2678621]: [2024-02-13 08:22:07 +0100] [6] [DEBUG] GET /api/v1/analyses
Feb 13 08:22:07 cg-vm1.scilifelab.se podman[2678621]: 10.0.2.100 - - [13/Feb/2024:08:22:07 +0100] "GET /api/v1/analyses?pipeline=&pageSize=50&page=1 HTTP/1.0" 200 28135 "https://cigrid-stage.scilifelab.se/" "Mozil>
lines 1-3

Trailblazer scan 👍

θ63° [henrik.stranneheim@hasta:~] [S_base] 130 $ trailblazer scan
Error updating analysis for: case - grandaphid : [Errno 13] Permission denied: '/home/proj/production/cancer/cases/grandaphid/analysis/slurm_jobids.yaml'
Error updating analysis for: case - goodbengal : [Errno 2] No such file or directory: '/home/proj/production/nipt/cases/goodbengal/output/sacct/submitted_jobs.yaml'
Error updating analysis for: case - helpedlab : [Errno 2] No such file or directory: '/home/proj/production/nipt/cases/helpedlab/output/sacct/submitted_jobs.yaml'
Error updating analysis for: case - newgopher : [Errno 13] Permission denied: '/home/proj/production/microbial/results/reports/trailblazer/ACC9662_slurm_ids.yaml'
Error updating analysis for: case - onestag : [Errno 13] Permission denied: '/home/proj/production/microbial/results/reports/trailblazer/ACC9966_slurm_ids.yaml'
Error updating analysis for: case - funnyaphid : [Errno 2] No such file or directory: '/home/proj/production/mutant/cases/funnyaphid/results/trailblazer_config.yaml'
Error updating analysis for: case - proudfish : [Errno 13] Permission denied: '/home/proj/production/microbial/results/reports/trailblazer/ACC9337_slurm_ids.yaml'
Error updating analysis for: case - setcolt : [Errno 13] Permission denied: '/home/proj/production/microbial/results/reports/trailblazer/ACC9963_slurm_ids.yaml'
Error updating analysis for: case - handyshrimp : [Errno 2] No such file or directory: '/home/proj/production/mutant/cases/handyshrimp/results/trailblazer_config.yaml'
Error updating analysis for: case - divinetomcat : [Errno 13] Permission denied: '/home/proj/production/microbial/results/reports/trailblazer/ACC5119_slurm_ids.yaml'
Error updating analysis for: case - HW2V2DRX2 : [Errno 2] No such file or directory: '/home/proj/production/flowcells/novaseq/runs/230504_A00689_0805_AHW2V2DRX2/slurm_job_ids.yaml'
Error updating analysis for: case - savingstag : [Errno 2] No such file or directory: '/home/proj/production/mutant/cases/savingstag/results/trailblazer_config.yaml'
Error updating analysis for: case - topewe : [Errno 13] Permission denied: '/home/proj/production/microbial/results/reports/trailblazer/ACC12566_slurm_ids.yaml'
Error updating analysis for: case - fullgoat : [Errno 13] Permission denied: '/home/proj/production/microbial/results/reports/trailblazer/ACC12567_slurm_ids.yaml'
Status in Tower: coolkodiak - 282044 - WPjrjs23yxQwq
Error logging case - coolkodiak :  JSONDecodeError
Updated status correctplatypus - 287509: pending
Error updating analysis for: case - 22FYCNLT3 : [Errno 2] No such file or directory: '/home/proj/production/flow_cells/20231222_LH00188_0053_A22FYCNLT3/slurm_job_ids.yaml'
Updated status hotglider - 288340: failed
Updated status refinedboar - 288356: completed
Updated status gladbuzzard - 288357: completed
Updated status definitemoray - 288359: completed
Updated status absoluteglowworm - 288360: completed
Error updating analysis for: case - lastinghoneybee : [Errno 2] No such file or directory: '/home/proj/production/rare-disease/cases/lastinghoneybee/analysis/slurm_job_ids.yaml'
Updated status choicepuma - 288389: failed
Error updating analysis for: case - HYVYMCCXY : [Errno 2] No such file or directory: '/home/proj/production/flow_cells/190307_ST-E00266_0341_AHYVYMCCXY/slurm_job_ids.yaml'
Updated status setamoeba - 288397: cancelled
Error updating analysis for: case - HK33MDRX3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/230912_A00187_1009_AHK33MDRX3/slurm_job_ids.yaml'
Error updating analysis for: case - HK33MDRX3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/230912_A00187_1009_AHK33MDRX3/slurm_job_ids.yaml'
Error updating analysis for: case - HGYFNBCX2 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180509_D00450_0598_BHGYFNBCX2/slurm_job_ids.yaml'
Error updating analysis for: case - HJCFFALXX : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/170517_ST-E00266_0210_BHJCFFALXX/slurm_job_ids.yaml'
Error updating analysis for: case - HLYWYDSXX : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/190927_A00689_0069_BHLYWYDSXX/slurm_job_ids.yaml'
Error updating analysis for: case - HM2LNBCX2 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/181005_D00410_0735_BHM2LNBCX2/slurm_job_ids.yaml'
Error updating analysis for: case - 22F52TLT3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/20231108_LH00188_0028_B22F52TLT3/slurm_job_ids.yaml'
Error updating analysis for: case - HL32LCCXY : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180508_ST-E00269_0269_AHL32LCCXY/slurm_job_ids.yaml'
Error updating analysis for: case - HK33MDRX3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/230912_A00187_1009_AHK33MDRX3/slurm_job_ids.yaml'
Error updating analysis for: case - HGYFNBCX2 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180509_D00450_0598_BHGYFNBCX2/slurm_job_ids.yaml'
Error updating analysis for: case - HJCFFALXX : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/170517_ST-E00266_0210_BHJCFFALXX/slurm_job_ids.yaml'
Error updating analysis for: case - HLYWYDSXX : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/190927_A00689_0069_BHLYWYDSXX/slurm_job_ids.yaml'
Error updating analysis for: case - HM2LNBCX2 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/181005_D00410_0735_BHM2LNBCX2/slurm_job_ids.yaml'
Error updating analysis for: case - 22F52TLT3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/20231108_LH00188_0028_B22F52TLT3/slurm_job_ids.yaml'
Error updating analysis for: case - HL32LCCXY : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180508_ST-E00269_0269_AHL32LCCXY/slurm_job_ids.yaml'
Error updating analysis for: case - HK33MDRX3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/230912_A00187_1009_AHK33MDRX3/slurm_job_ids.yaml'
Error updating analysis for: case - HGYFNBCX2 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180509_D00450_0598_BHGYFNBCX2/slurm_job_ids.yaml'
Error updating analysis for: case - HJCFFALXX : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/170517_ST-E00266_0210_BHJCFFALXX/slurm_job_ids.yaml'
Error updating analysis for: case - HLYWYDSXX : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/190927_A00689_0069_BHLYWYDSXX/slurm_job_ids.yaml'
Error updating analysis for: case - HM2LNBCX2 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/181005_D00410_0735_BHM2LNBCX2/slurm_job_ids.yaml'
Error updating analysis for: case - 22F52TLT3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/20231108_LH00188_0028_B22F52TLT3/slurm_job_ids.yaml'
Error updating analysis for: case - HL32LCCXY : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180508_ST-E00269_0269_AHL32LCCXY/slurm_job_ids.yaml'
Error updating analysis for: case - HK33MDRX3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/230912_A00187_1009_AHK33MDRX3/slurm_job_ids.yaml'
Error updating analysis for: case - HL32LCCXY : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180508_ST-E00269_0269_AHL32LCCXY/slurm_job_ids.yaml'
Error updating analysis for: case - HL32LCCXY : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180508_ST-E00269_0269_AHL32LCCXY/slurm_job_ids.yaml'
Error updating analysis for: case - 22F52TLT3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/20231108_LH00188_0028_B22F52TLT3/slurm_job_ids.yaml'
Updated status 22F52TLT3 - 288425: completed
Updated status HK33MDRX3 - 288426: completed
Updated status HM2LNBCX2 - 288427: completed
Status in Tower: cuddlyhen - 288428 - 3YsiGXRdGQtp6S
Status in Tower: tendergoose - 288431 - 4pfgqVdbxWSXhr
Updated status justhusky - 288432: running
Updating upload job 664516416
Updating upload job 664516417
Updating upload job 664516418
Updating upload job 664516419
Updating upload job 664516420
Updating upload job 664516421
Updating upload job 664516422
Updating upload job 664516423
Updating upload job 664516424
Updating upload job 664516425
Updating upload job 664516426
Updating upload job 664516427
Updating upload job 664516428
Updating upload job 664516429
Updating upload job 664516430
Updating upload job 664516431
Updating upload job 664516432
Updating upload job 664516433
Updating upload job 664516434
Updating upload job 664516435
Updating upload job 664516436
Updating upload job 664516437
Updating upload job 664516438
Updating upload job 664516439
Updating upload job 664516440
Updating upload job 664516441
Updating upload job 664516442
Updating upload job 664516443
Updating upload job 664516444
Updating upload job 664516445
Updating upload job 664516446
Updating upload job 664516447
All analyses updated!

henrikstranneheim · 2024-02-13T08:49:21Z

@moedarrah A bit surprised that cigrid still work when deploying and testing these PRs. Could you help me understand why it does not fail even though I changed the response?

moedarrah · 2024-02-13T09:24:32Z

@moedarrah A bit surprised that cigrid still work when deploying and testing these PRs. Could you help me understand why it does not fail even though I changed the response?

I believe the issue might be related to the cache; the Trailblaze endpoints aren't functioning for me on the stage.

henrikstranneheim · 2024-02-13T09:53:00Z

@moedarrah Right now cg-vm1 is down for maintenance.

islean · 2024-02-13T09:57:23Z

@henrikstranneheim Did you mean that the Pipeline column was populated? Or were you surprised that it rendered at all? My guess is that the column would simply be empty with this change, since no value called data_analysis is being exported.

henrikstranneheim · 2024-02-13T10:00:56Z

@henrikstranneheim Did you mean that the Pipeline column was populated? Or were you surprised that it rendered at all? My guess is that the column would simply be empty with this change, since no value called data_analysis is being exported.

Right, and it is empty. That makes sense. @moedarrah do you have time this week to make a PR to accommodate this?

moedarrah · 2024-02-13T11:13:25Z

@henrikstranneheim Did you mean that the Pipeline column was populated? Or were you surprised that it rendered at all? My guess is that the column would simply be empty with this change, since no value called data_analysis is being exported.

Right, and it is empty. That makes sense. @moedarrah do you have time this week to make a PR to accommodate this?

Sure 👍

moedarrah · 2024-02-13T11:51:40Z

@henrikstranneheim Did you mean that the Pipeline column was populated? Or were you surprised that it rendered at all? My guess is that the column would simply be empty with this change, since no value called data_analysis is being exported.

Right, and it is empty. That makes sense. @moedarrah do you have time this week to make a PR to accommodate this?

Yes, we're using conditional rendering on the frontend to prevent UI breaks if the value is missing.

trailblazer/dto/create_analysis_request.py

seallard

Some changes needed in cg as well

…kflow # Conflicts: # trailblazer/store/base.py

…ine-to-workflow' into refactor-data_analysis-and-pipeline-to-workflow

sonarcloud · 2024-02-16T07:12:07Z

Quality Gate passed

Issues
0 New issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

henrikstranneheim · 2024-02-16T07:17:30Z

@moedarrah Could you link the Cigrid PR here when you are done and I can test cg/TB and Cigrid on stage?

moedarrah · 2024-02-16T08:16:12Z

@moedarrah Could you link the Cigrid PR here when you are done and I can test cg/TB and Cigrid on stage?

The PR has been deployed to the stage: https://github.com/Clinical-Genomics/cigrid-ui/pull/509

henrikstranneheim · 2024-02-16T09:16:27Z

👍

islean · 2024-02-16T09:19:03Z

Works well with the new order views as well 💪

henrikstranneheim · 2024-02-16T09:25:24Z

Planning to M&D on Monday!

henrikstranneheim · 2024-02-19T08:27:37Z

Deployed in production:

Log deploy... done.
trailblazer, version 21.0.0

henrikstranneheim added 6 commits February 9, 2024 15:46

feat(workflow): Refactor

4bf2930

feat(workflow): Refactor

c74c7e2

feat(workflow): Refactor

08fb728

feat(workflow): Refactor

dd2e5e5

feat(workflow): Refactor db model

fae9567

feat(workflow): Refactor to mutant

f742569

henrikstranneheim added the Enhancement label Feb 9, 2024

henrikstranneheim self-assigned this Feb 9, 2024

Merge branch 'master' into refactor-data_analysis-and-pipeline-to-wor…

710aad3

…kflow

clingen-sthlm temporarily deployed to stage February 13, 2024 07:07 Inactive

henrikstranneheim marked this pull request as ready for review February 13, 2024 07:30

henrikstranneheim requested a review from a team as a code owner February 13, 2024 07:30

seallard reviewed Feb 14, 2024

View reviewed changes

trailblazer/dto/create_analysis_request.py Show resolved Hide resolved

seallard approved these changes Feb 14, 2024

View reviewed changes

clingen-sthlm temporarily deployed to stage February 14, 2024 15:26 Inactive

clingen-sthlm temporarily deployed to stage February 14, 2024 15:28 Inactive

clingen-sthlm temporarily deployed to stage February 14, 2024 15:42 Inactive

henrikstranneheim added 2 commits February 16, 2024 08:11

Merge branch 'master' into refactor-data_analysis-and-pipeline-to-wor…

8407323

…kflow # Conflicts: # trailblazer/store/base.py

Merge remote-tracking branch 'origin/refactor-data_analysis-and-pipel…

bd0ef13

…ine-to-workflow' into refactor-data_analysis-and-pipeline-to-workflow

clingen-sthlm temporarily deployed to stage February 16, 2024 09:05 Inactive

henrikstranneheim merged commit 99e3da5 into master Feb 19, 2024
5 checks passed

henrikstranneheim deleted the refactor-data_analysis-and-pipeline-to-workflow branch February 19, 2024 08:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor data analysis and pipeline to workflow #391

Refactor data analysis and pipeline to workflow #391

henrikstranneheim commented Feb 9, 2024 •

edited

Loading

henrikstranneheim commented Feb 13, 2024

henrikstranneheim commented Feb 13, 2024

moedarrah commented Feb 13, 2024

henrikstranneheim commented Feb 13, 2024

islean commented Feb 13, 2024

henrikstranneheim commented Feb 13, 2024

moedarrah commented Feb 13, 2024

moedarrah commented Feb 13, 2024

seallard left a comment

sonarcloud bot commented Feb 16, 2024

henrikstranneheim commented Feb 16, 2024

moedarrah commented Feb 16, 2024

henrikstranneheim commented Feb 16, 2024

islean commented Feb 16, 2024

henrikstranneheim commented Feb 16, 2024

henrikstranneheim commented Feb 19, 2024

Refactor data analysis and pipeline to workflow #391

Refactor data analysis and pipeline to workflow #391

Conversation

henrikstranneheim commented Feb 9, 2024 • edited Loading

Description

Added

Changed

Fixed

How to prepare for test

How to test

Expected test outcome

Review

This version is a

henrikstranneheim commented Feb 13, 2024

cg workflow mip-dna start justhusky 👍

Trailblazer scan 👍

henrikstranneheim commented Feb 13, 2024

moedarrah commented Feb 13, 2024

henrikstranneheim commented Feb 13, 2024

islean commented Feb 13, 2024

henrikstranneheim commented Feb 13, 2024

moedarrah commented Feb 13, 2024

moedarrah commented Feb 13, 2024

seallard left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Feb 16, 2024

Quality Gate passed

henrikstranneheim commented Feb 16, 2024

moedarrah commented Feb 16, 2024

henrikstranneheim commented Feb 16, 2024

islean commented Feb 16, 2024

henrikstranneheim commented Feb 16, 2024

henrikstranneheim commented Feb 19, 2024

henrikstranneheim commented Feb 9, 2024 •

edited

Loading