Staging -> Master #837
Staging -> Master #837
Conversation
* increase disk size for MakeOptionalBam * changelogs and pipeline versions Co-authored-by: Nikelle Petrillo <npetrill@broadinstitute.org>
* Point fingerprinting file keys to publicly available files * Remove redundant input files from germline single sample pipeline
* Update Skylab STAR Dockerfile to use Alpine Linux image and remove unneeded packages * Create build script, version file, and readme for STAR Docker image * Update StarAlign.wdl to use new STAR docker image * Update changelogs * Add --platform flag to STAR Dockerfile
…other ARM machines and fix typos (#777)
* Update ValidateATAC.wdl and ValidateSCATAC.wdl to use Broad samtools image * Remove redundant samtools Dockerfile and directory * Remove skylab/hca_adapter directory
* Remove unused TagGeneExon.wdl * Remove dropseq_tools Docker directories and additional files
Change memory allocation and enable maxRetries in fastp and groupbyUMIs tasks. Temporarily add monitoring scripts to these tasks to monitor memory usage. Update the transcriptome bam to be the RSEM post-processed one. Removing tdr_staging_bucket from test file since it is no longer an input Co-authored-by: M. Morgan Taylor <marymorg@broadinstitute.org> Co-authored-by: Nikelle Petrillo <npetrill@broadinstitute.org>
* Update imputation bcf/vcftools and minimac4 docker images * Update changelogs
* Rewrite TrimAdapters Dockerfile, build script * Create TrimAdapters version list, readme * Replace TrimAdapters.wdl runtime docker URL with GCR URL * Rename trim-adapters to ea-utils, push a new image to GCR, and replace all docker image URLs * Update changelog and version number * Minor corrections to Minimac4 version list and STAR readme
Remove umi-tools docker directory
* Update Optimus MergeStarOutput task, LoomUtils.wdl tasks to use new pytools docker image * Standardize paths to scripts in docker image in affected tasks * Add Docker image files for new pytools docker and reorganize scripts * Fix print function syntax in scATAC MakeCompliantBAM task * Update changelogs and version numbers * Remove old Dockerfiles, build scripts, requirements files * Revert some changes from #687 in create_snrna_optimus.py to fix a runtime error
* Create new bwa and snaptools-bwa docker files and documentation * Build images and replace URLs in tasks
* Document pytools docker scripts, remove unused create-npz-output.py script and ConvertStarOutput task * Update changelogs
* Update bcf/vcftools docker files
* update get_changed_pipeline_worklow_test_args.sh * Test * remove Test
…with additionally added required parameter
* update WGS pipeline docs * Update README.md * Update website/docs/Pipelines/Whole_Genome_Germline_Single_Sample_Pipeline/README.md Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
* Rebuild featureCounts image into a general subread image, add documentation and docker files, and update changelogs and URL in tasks
…tute/warp into BroadInternalArraysImputation
* Create new docker files and update tasks and changelogs
…gelogs, and version numbers (#822)
* Correct test string and update testing framework script
#829) * Remove /cromwell_root/ prefix in file paths and update changelogs
#827) * Consistency and formatting improvements * Update some tasks to use rebuilt images * Update changelogs
…DL (#830) * Checkpoint. Renamed the package and made modifications based on recommendations in Warp's docker style guide. Local build currently fails even without modifications, currently investigating why. * Modifications based on WARP's docker style guide * Updated to conform to the docker style guide of warp. Also upgraded to a more recent Debian version (Bullseye) so tini is part of apt. An upgrade of python was attempted too but pip failed as it can't find old versions of libraries listed in requirements.txt. * Parameterizing STAR version. * Adding build script * Updating build script to push to GCR. * Changed the WDL to utilize the new container. * Changing biotypes type from String to File so it localizes. * Fixing the PATH, rebuilding the container, then setting the WDL to use the new build. * Ran into an error where STAR was looking for a fasta file that didn't exist. This fixes that error. * Added Readme for the container. * Updated changelog and version. * Changing the list of versions to match the other containers' format
BroadInternalArrays and BroadInternalImputation: change docker from dockerhub to gcr
dev -> staging
|
DO NOT SQUASH MERGE!!! |
|
retest this please |
3 similar comments
|
retest this please |
|
retest this please |
|
retest this please |
|
UG specific tests running in dev env: |
|
retest this please |
|
Note on the failing smart tests: UltimaGenomicsJointGenotyping: This was previously passing here: https://gotc-jenkins.dsp-techops.broadinstitute.org/job/warp-workflow-tests/35963/console. Since this test, we have updated the UltimaGenomicsJointGenotyping truth after this PR was merged into develop. Once this PR is merged into staging, the scientific tests should pass. scATAC: It has been determined that there is some non-determinism in this pipeline. We run this test every night in dev, but the cache broke when we ran it in staging. UltimaGenomicsWholeGenomeGermline: This test is having permissions issues when it is run in staging. Here is a link to the passing scientific test in dev: https://gotc-jenkins.dsp-techops.broadinstitute.org/job/warp-workflow-tests/36657/console. Here is the Jira ticket to investigate the permissions issue. BroadInternalUltimaGenomics: This test is having permissions issues when it is run in staging. Here is a link to the passing scientific test in dev: https://gotc-jenkins.dsp-techops.broadinstitute.org/job/warp-workflow-tests/36715/console. Here is the Jira ticket to investigate the permissions issue. RNAWithUMIsPipeline: From Chris K: "This new non-determinism is in umi-tools, there are a few PRs/issues talking about it in the repo (CGATOxford/UMI-tools#550, CGATOxford/UMI-tools#365, CGATOxford/UMI-tools#349). This happens pretty rarely, which is I suspect why we haven’t hit it in plumbing tests. For now, I think it is fine to merge, since the differences are all very small and understood." BroadInternalRNAWithUMIsPipeline: https://gotc-jenkins.dsp-techops.broadinstitute.org/job/warp-workflow-tests/37138/console is failing in the same way RNAWithUMIsPipeline is failing. These differences are due to non-determinism |
No description provided.