[WORKFLOWS-220] Create py-orca demonstration script
#23
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## main #23 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 28 28
Lines 854 869 +15
Branches 134 137 +3
=========================================
+ Hits 854 869 +15
|
thomasyu888
reviewed
May 19, 2023
|
@thomasyu888 @BWMac Could at least one of your try to run the script locally as per my instructions in the README? In theory, the |
|
@BrunoGrandePhD Giving it a shot now! After working out a couple of kinks with Bruno, the whole process was executed successfully. I think the instructions are great and it's easy for someone to try it out for themselves. |
BWMac
approved these changes
May 31, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The goal of this script is to demonstrate how you can use
py-orcato process a dataset (in this case, RNA-seq) using a series of workflow runs:nf-synstage,nf-core/rnaseq, andnf-synindex. I made some minor modifications to thepy-orcamodule to make it easier to write this script. Its current location is temporary. We can discuss whether we want to move this to a subfolder or a separate repository.One key improvement over the scripts that I wrote for NTAP is that I wanted to create an actual DAG. Doing this in Airflow would make this challenging for users to play around with it. Instead, I decided to experiment with Metaflow, which makes it easy to test a DAG locally (but offers the chance to deploy it to AWS later). It's similar to Prefect and Dagster. We can revisit this once we settle on a strategy for consistency between Airflow and non-Airflow DAGs.
python3 demo.py run --dataset_id syn51514585Some of the components in this script (e.g.
RnaseqDatasetandTowerRnaseqFlow) could be reused in different contexts. We should also discuss if this is desirable and if so, where to store those abstract components (e.g. inpy-orcaor elsewhere).demo.pyin READMECommand and Output