Skip to content
This repository was archived by the owner on May 31, 2024. It is now read-only.

feat: Added output file creation snakemake + copying efs files to s3#316

Merged
waliag87 merged 3 commits into
mainfrom
gwalia/add_outputs_and_sync
Feb 18, 2022
Merged

feat: Added output file creation snakemake + copying efs files to s3#316
waliag87 merged 3 commits into
mainfrom
gwalia/add_outputs_and_sync

Conversation

@waliag87
Copy link
Copy Markdown
Contributor

@waliag87 waliag87 commented Feb 10, 2022 •
edited
Loading

Issue #, if available:

Description of Changes

This change does the following:

  • Creates an output file for snakemake in the customers s3 account.
  • Copies the sm outputs from the batch mounted efs to the customers s3 bucket, where execution outputs are expected.
  • Updates the wes adapter on how to get outputs.

Outputs look like:

OUTPUT	id	84f32737-35e6-49ec-b654-a3ffd207451f
OUTPUT	outputs.snakemake_output.0	s3://agc-987650158744-us-west-2/project/SnakemakeDemo/userid/gwalia4Dhwlt/context/onDemandContext/snakemake-execution/84f32737-35e6-49ec-b654-a3ffd207451f/output/Bonjour.txt
OUTPUT	outputs.snakemake_output.1	s3://agc-987650158744-us-west-2/project/SnakemakeDemo/userid/gwalia4Dhwlt/context/onDemandContext/snakemake-execution/84f32737-35e6-49ec-b654-a3ffd207451f/output/Ciao.txt
OUTPUT	outputs.snakemake_output.2	s3://agc-987650158744-us-west-2/project/SnakemakeDemo/userid/gwalia4Dhwlt/context/onDemandContext/snakemake-execution/84f32737-35e6-49ec-b654-a3ffd207451f/output/Hello.txt
OUTPUT	outputs.snakemake_output.3	s3://agc-987650158744-us-west-2/project/SnakemakeDemo/userid/gwalia4Dhwlt/context/onDemandContext/snakemake-execution/84f32737-35e6-49ec-b654-a3ffd207451f/output/Hola.txt

Validated by updating unit tests and manually running integration tests with example workflows.

Checklist

  • If this change would make any existing documentation invalid, I have included those updates within this PR
  • I have added unit tests that prove my fix is effective or that my feature works
  • I have linted my code before raising the PR
  • Title of this Pull Request follows Conventional Commits standard: https://www.conventionalcommits.org/en/v1.0.0/

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@waliag87 waliag87 temporarily deployed to slack February 10, 2022 18:28 Inactive
@waliag87 waliag87 force-pushed the gwalia/add_outputs_and_sync branch from 7a8a85a to c10474d Compare February 10, 2022 18:28
tneely
tneely reviewed Feb 10, 2022
View reviewed changes
Comment thread packages/engines/snakemake/snakemake.aws.sh Outdated
Comment on lines 36 to 38
if [[ $ENGINE_PROJECT != *"://"* ]] ; then
ENGINE_PROJECT="${ENGINE_PROJECT_DIRECTORY}/${ENGINE_PROJECT}"
ENGINE_PROJECT="${ENGINE_PROJECT}"
fi
Copy link
Copy Markdown
Contributor

@tneely tneely Feb 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can just be deleted - you're reassigning to the same variable/value

@waliag87 waliag87 requested review from tneely and removed request for elliot-smith February 18, 2022 17:00
@waliag87 waliag87 merged commit bb4a42f into main Feb 18, 2022
@waliag87 waliag87 deleted the gwalia/add_outputs_and_sync branch February 18, 2022 19:32
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@tneely tneely Awaiting requested review from tneely

2 more reviewers

@evanyui evanyui evanyui approved these changes

@elliot-smith elliot-smith elliot-smith approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@waliag87 @tneely @evanyui @elliot-smith