New pipeline runner image. #841

bpblanken · 2024-07-22T15:24:19Z

No description provided.

* Add mito local constraint * Fix tests * lint

bpblanken · 2024-07-25T19:22:18Z

.cloudbuild/vep-docker.cloudbuild.yaml

@@ -1,11 +1,11 @@
 # Run locally with:
 #
-# gcloud builds submit --quiet --substitutions='_VEP_VERSION=110' --config .cloudbuild/vep-docker.cloudbuild.yaml v03_pipeline/deploy
+# gcloud builds submit --quiet --substitutions='_REFERENCE_GENOME=GRCh38' --config .cloudbuild/vep-docker.cloudbuild.yaml v03_pipeline/deploy


I've left automated builds to another ticket to avoid scope creep:

#848

…b.com:broadinstitute/seqr-loading-pipelines into benb/pipeline-runner-docker-electric-boogaloo

bpblanken · 2024-07-26T20:47:15Z

.github/workflows/dev-release.yml

@@ -44,5 +44,6 @@ jobs:
      - name: Copy files to release directory
        run: |-
          gcloud storage rm -r gs://seqr-luigi/releases/dev/latest/ || echo 'No latest release'
-          gcloud storage cp v03_pipeline/bin/* gs://seqr-luigi/releases/dev/latest/
+          gcloud storage cp v03_pipeline/bin gs://seqr-luigi/releases/dev/latest/bin/
+          gcloud storage cp v03_pipeline/var/vep_config gs://seqr-luigi/releases/dev/latest/var/vep_config


Changed the structure of the deploy artifact a bit so the vep configs are deployed and accessible by the dataproc init.

bpblanken · 2024-07-26T20:48:02Z

requirements.txt

@@ -146,7 +146,7 @@ grpcio==1.63.0
    #   grpcio-status
 grpcio-status==1.48.2
    # via google-api-core
-hail==0.2.130
+hail==0.2.132


There was a hail bug blocking the docker container from working

bpblanken · 2024-07-26T20:48:36Z

v03_pipeline/api/__main__.py

+from v03_pipeline.lib.logger import get_logger
+
+
+def run():


Just a shim to get rolling.

bpblanken · 2024-07-26T20:49:28Z

v03_pipeline/bin/dataproc_vep_init_38.sh

+gcc -Wall -Werror -O2 /vep.c -o /vep
+chmod u+s /vep
+
+gcloud storage cp gs://seqr-luigi/releases/$ENVIRONMENT/latest/bin/download_vep_data.bash /download_vep_data.bash


I refactored this so that shared code with the local deploy would be modularized (in bash!! 😬). I'm endeavoring to test on dataproc right now.

bpblanken · 2024-07-26T20:54:31Z

v03_pipeline/var/vep_config/vep-GRCh38.json

@@ -2,7 +2,7 @@
  "command": [
    "bash",
    "-c",
-    "/vep --warning_file STDERR --format vcf -json --hgvs --biotype --canonical --mane --minimal --numbers --regulatory --allele_number --no_stats --cache --offline --assembly GRCh38 --fasta /opt/vep/.vep/Homo_sapiens.GRCh38.dna.primary_assembly.fa.gz --check_ref --dont_skip --plugin LoF,loftee_path:/plugins,gerp_bigwig:/opt/vep/.vep/gerp_conservation_scores.homo_sapiens.GRCh38.bw,human_ancestor_fa:/opt/vep/.vep/human_ancestor.fa.gz,conservation_file:/opt/vep/.vep/loftee.sql --plugin UTRAnnotator,file=/opt/vep/.vep/uORF_5UTR_GRCh38_PUBLIC.txt --plugin SpliceRegion,Extended --plugin AlphaMissense,file=/opt/vep/.vep/AlphaMissense_hg38.tsv.gz --dir_plugins /plugins -o STDOUT | sed s/5utr/fiveutr/g"
+    "/vep GRCh38 --warning_file STDERR --format vcf -json --hgvs --biotype --canonical --mane --minimal --numbers --regulatory --allele_number --no_stats --cache --offline --assembly GRCh38 --fasta /opt/vep/.vep/Homo_sapiens.GRCh38.dna.primary_assembly.fa.gz --check_ref --dont_skip --plugin LoF,loftee_path:/plugins,gerp_bigwig:/opt/vep/.vep/gerp_conservation_scores.homo_sapiens.GRCh38.bw,human_ancestor_fa:/opt/vep/.vep/human_ancestor.fa.gz,conservation_file:/opt/vep/.vep/loftee.sql --plugin UTRAnnotator,file=/opt/vep/.vep/uORF_5UTR_GRCh38_PUBLIC.txt --plugin SpliceRegion,Extended --plugin AlphaMissense,file=/opt/vep/.vep/AlphaMissense_hg38.tsv.gz --dir_plugins /plugins -o STDOUT | sed s/5utr/fiveutr/g"


The vep binary was updated to accept a reference genome arg, which we then use to volume mount a subdirectory of /vep_data up into /opt/vep/.vep/

bpblanken · 2024-07-26T20:55:07Z

v03_pipeline/var/vep_config/vep-GRCh37.json

+    "--minimal",
+    "--assembly", "GRCh37",
+    "--fasta", "/opt/vep/.vep/homo_sapiens/Homo_sapiens.GRCh37.dna.primary_assembly.fa.gz",
+    "--plugin", "LoF,human_ancestor_fa:/opt/vep/.vep/loftee_data/GRCh37/human_ancestor.fa.gz,filter_position:0.05,min_intron_size:15,conservation_file:/opt/vep/.vep/loftee_data/GRCh37/phylocsf_gerp.sql,gerp_file:/opt/vep/.vep/loftee_data/GRCh37/GERP_scores.final.sorted.txt.gz",


The loftee_data for GRCh37 is nested but it isn't nested for 38. So I think this is right as is.

bpblanken · 2024-07-26T20:55:14Z

v03_pipeline/var/vep_config/vep-GRCh37.json

+ "env": {
+     "PERL5LIB": ""
+ },
+ "vep_json_schema": "Struct{assembly_name:String,allele_string:String,ancestral:String,colocated_variants:Array[Struct{aa_allele:String,aa_maf:Float64,afr_allele:String,afr_maf:Float64,allele_string:String,amr_allele:String,amr_maf:Float64,clin_sig:Array[String],end:Int32,eas_allele:String,eas_maf:Float64,ea_allele:String,ea_maf:Float64,eur_allele:String,eur_maf:Float64,exac_adj_allele:String,exac_adj_maf:Float64,exac_allele:String,exac_afr_allele:String,exac_afr_maf:Float64,exac_amr_allele:String,exac_amr_maf:Float64,exac_eas_allele:String,exac_eas_maf:Float64,exac_fin_allele:String,exac_fin_maf:Float64,exac_maf:Float64,exac_nfe_allele:String,exac_nfe_maf:Float64,exac_oth_allele:String,exac_oth_maf:Float64,exac_sas_allele:String,exac_sas_maf:Float64,id:String,minor_allele:String,minor_allele_freq:Float64,phenotype_or_disease:Int32,pubmed:Array[Int32],sas_allele:String,sas_maf:Float64,somatic:Int32,start:Int32,strand:Int32}],context:String,end:Int32,id:String,input:String,intergenic_consequences:Array[Struct{allele_num:Int32,consequence_terms:Array[String],impact:String,minimised:Int32,variant_allele:String}],most_severe_consequence:String,motif_feature_consequences:Array[Struct{allele_num:Int32,consequence_terms:Array[String],high_inf_pos:String,impact:String,minimised:Int32,motif_feature_id:String,motif_name:String,motif_pos:Int32,motif_score_change:Float64,strand:Int32,variant_allele:String}],regulatory_feature_consequences:Array[Struct{allele_num:Int32,biotype:String,consequence_terms:Array[String],impact:String,minimised:Int32,regulatory_feature_id:String,variant_allele:String}],seq_region_name:String,start:Int32,strand:Int32,transcript_consequences:Array[Struct{allele_num:Int32,amino_acids:String,biotype:String,canonical:Int32,ccds:String,cdna_start:Int32,cdna_end:Int32,cds_end:Int32,cds_start:Int32,codons:String,consequence_terms:Array[String],distance:Int32,domains:Array[Struct{db:String,name:String}],exon:String,gene_id:String,gene_pheno:Int32,gene_symbol:String,gene_symbol_source:String,hgnc_id:String,hgvsc:String,hgvsp:String,hgvs_offset:Int32,impact:String,intron:String,lof:String,lof_flags:String,lof_filter:String,lof_info:String,minimised:Int32,polyphen_prediction:String,polyphen_score:Float64,protein_end:Int32,protein_start:Int32,protein_id:String,sift_prediction:String,sift_score:Float64,strand:Int32,swissprot:String,transcript_id:String,trembl:String,uniparc:String,variant_allele:String}],variant_class:String}"


Unaltered thus far.

bpblanken · 2024-07-26T21:41:00Z

v03_pipeline/deploy/Dockerfile

@@ -0,0 +1,23 @@
+FROM docker:dind as BUILD


This is the actual new Dockerfile, replacing https://github.com/broadinstitute/seqr-loading-pipelines/blob/main/docker/Dockerfile

bpblanken added 3 commits July 19, 2024 18:03

Push

1a0753a

This works!

592cfef

cleanup

5553e6b

bpblanken changed the title ~~Benb/pipeline runner docker electric boogaloo~~ New pipeline runner image. Jul 22, 2024

bpblanken and others added 13 commits July 22, 2024 11:29

lint

8b9be0f

lint

41ffab4

Tweak spark config location

c301012

add dockerignore

0b91c65

vep bumps

2641764

Finish download

9a5a61c

lines

24af8d0

correct path

0259140

Move vep files (#844)

f7992eb

Add mito local constraint (#845)

63191c3

* Add mito local constraint * Fix tests * lint

merge

597438b

bump vep version

d4bc209

path updates

0acb973

bpblanken commented Jul 25, 2024

View reviewed changes

bpblanken and others added 12 commits July 26, 2024 13:08

Add reference genome to path

688f879

rename to REFERENCE_GENOME

609f986

refactor yet again

787a3e4

Finish

75463f0

Merge branch 'main' into benb/pipeline-runner-docker-electric-boogaloo

fcc21fd

cleanup

b19da16

Merge branch 'benb/pipeline-runner-docker-electric-boogaloo' of githu…

eabd0a2

…b.com:broadinstitute/seqr-loading-pipelines into benb/pipeline-runner-docker-electric-boogaloo

redo reference data

bbbc425

more refactoring

b661a7e

more refactor

2a723a2

bump

83a9d37

absolute path

8e0c49a

bpblanken commented Jul 26, 2024

View reviewed changes

v03_pipeline/api/__main__.py

from v03_pipeline.lib.logger import get_logger

def run():

Copy link

Collaborator Author

bpblanken Jul 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a shim to get rolling.

bpblanken commented Jul 26, 2024

View reviewed changes

Fix var

8acf0fc

bpblanken commented Jul 26, 2024

View reviewed changes

bpblanken added 2 commits July 26, 2024 17:24

default back to /vep_data

614acc8

rearrange vep data

dd87fef

bpblanken marked this pull request as ready for review July 26, 2024 21:39

bpblanken requested a review from a team as a code owner July 26, 2024 21:39

bpblanken commented Jul 26, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New pipeline runner image. #841

New pipeline runner image. #841

bpblanken commented Jul 22, 2024

bpblanken Jul 25, 2024

bpblanken Jul 26, 2024

bpblanken Jul 26, 2024

bpblanken Jul 26, 2024

bpblanken Jul 26, 2024

bpblanken Jul 26, 2024

bpblanken Jul 26, 2024

bpblanken Jul 26, 2024

bpblanken Jul 26, 2024

New pipeline runner image. #841

Are you sure you want to change the base?

New pipeline runner image. #841

Conversation

bpblanken commented Jul 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment