Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tower actions testing #981

Merged
merged 26 commits into from
Mar 29, 2023
Merged

Tower actions testing #981

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
0fb0648
First pass at a GitHub Actions workflow for Tower
robsyme Mar 28, 2023
ce0bb67
Use add-wait branch of tower action
robsyme Mar 28, 2023
8813318
Remove if statements
robsyme Mar 29, 2023
4c07497
Add full-sized cloud tests
robsyme Mar 29, 2023
c513d27
Trailing commas are not allowed in YAML files.
robsyme Mar 29, 2023
bf8c05a
Unset igenomes_base from small test workflow
robsyme Mar 29, 2023
a759554
Apply suggestions from code review
drpatelh Mar 29, 2023
d21abe6
Move to self-hosted runners
robsyme Mar 29, 2023
56bf2ed
Add matrix
robsyme Mar 29, 2023
053d16e
Change to self-hosted runners
robsyme Mar 29, 2023
7399f56
Include aligner in run names
robsyme Mar 29, 2023
22db014
Update work directory to match old runners
robsyme Mar 29, 2023
3ee41ab
Use env for run name
robsyme Mar 29, 2023
29cf6ab
Remove github.event.pull_request.head.label
robsyme Mar 29, 2023
79c354f
Rely on action to remove colons from branch name
robsyme Mar 29, 2023
8fab0d9
Remove double underscore
robsyme Mar 29, 2023
1a88aa8
Bump actions versions
robsyme Mar 29, 2023
7957933
Minimize run names to remove tag/branch/commit
robsyme Mar 29, 2023
bb41bbc
Fix #975
drpatelh Mar 27, 2023
baa7aa3
Replace System.exit(1) calls in pipeline template
drpatelh Mar 27, 2023
7b8abfc
Fix nf-core lint
drpatelh Mar 27, 2023
aa407f4
Add explicit test_full profiles for all cloud providers
drpatelh Mar 28, 2023
565e970
Update nextflow.config
drpatelh Mar 28, 2023
257fd7e
Update nextflow.config
drpatelh Mar 28, 2023
530c813
Update nextflow.config
drpatelh Mar 29, 2023
fe39f59
Minor fixes for JSON sent to Slack.
robsyme Mar 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 93 additions & 0 deletions .github/workflows/cloud_tests_full.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
name: Run full-sized tests on cloud providers
run-name: Submitting workflow to all cloud providers
on:
pull_request:
drpatelh marked this conversation as resolved.
Show resolved Hide resolved
workflow_dispatch:
inputs:
platform:
description: 'Platform to run test'
required: true
default: 'all'
type: choice
options:
- all
- aws
- azure
- gcp
jobs:
run-full-tests-on-aws:
# if: ${{ github.event.inputs.platform == 'all' || github.event.inputs.platform == 'aws' }}
runs-on: self-hosted
strategy:
matrix:
aligner: ["star_salmon", "star_rsem"]
steps:
- uses: seqeralabs/action-tower-launch@add-wait
drpatelh marked this conversation as resolved.
Show resolved Hide resolved
with:
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
compute_env: ${{ secrets.TOWER_CE_AWS_CPU }}
workdir: "${{ secrets.TOWER_BUCKET_AWS }}/work/${{ github.sha }}"
drpatelh marked this conversation as resolved.
Show resolved Hide resolved
run_name: AWS_${{ github.event.pull_request.head.label || github.head_ref || github.ref }}
drpatelh marked this conversation as resolved.
Show resolved Hide resolved
drpatelh marked this conversation as resolved.
Show resolved Hide resolved
profiles: test_full
parameters: |
{
"hook_url": "${{ secrets.MEGATESTS_ALERTS_SLACK_HOOK_URL }}",
"outdir": "${{ secrets.TOWER_BUCKET_AWS }}/rnaseq/results-${{ github.sha }}/aligner_${{ matrix.aligner }}"
}
wait: false
- uses: actions/upload-artifact@v3
with:
name: Tower debug log file
path: tower_action_*.log
run-full-tests-on-gcp:
# if: ${{ github.event.inputs.platform == 'all' || github.event.inputs.platform == 'gcp' }}
runs-on: self-hosted
strategy:
matrix:
aligner: ["star_salmon", "star_rsem"]
steps:
- uses: seqeralabs/action-tower-launch@add-wait
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- uses: seqeralabs/action-tower-launch@add-wait
- uses: seqeralabs/action-tower-launch@v1

with:
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
compute_env: ${{ secrets.TOWER_CE_GCP_CPU }}
workdir: "${{ secrets.TOWER_BUCKET_GCP }}/work/${{ github.sha }}"
run_name: GCP_${{ github.event.pull_request.head.label || github.head_ref || github.ref }}
profiles: test_full
drpatelh marked this conversation as resolved.
Show resolved Hide resolved
parameters: |
{
"hook_url": "${{ secrets.MEGATESTS_ALERTS_SLACK_HOOK_URL }}",
"outdir": "${{ secrets.TOWER_BUCKET_GCP }}/rnaseq/results-${{ github.sha }}/aligner_${{ matrix.aligner }}"
}
wait: false
- uses: actions/upload-artifact@v3
with:
name: Tower debug log file
path: tower_action_*.log
run-full-tests-on-azure:
# if: ${{ github.event.inputs.platform == 'all' || github.event.inputs.platform == 'azure' }}
runs-on: self-hosted
strategy:
matrix:
aligner: ["star_salmon", "star_rsem"]
steps:
- uses: seqeralabs/action-tower-launch@add-wait
drpatelh marked this conversation as resolved.
Show resolved Hide resolved
with:
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
compute_env: ${{ secrets.TOWER_CE_AZURE_CPU }}
workdir: "${{ secrets.TOWER_BUCKET_AZURE }}/work/${{ github.sha }}"
run_name: AZURE_${{ github.event.pull_request.head.label || github.head_ref || github.ref }}
profiles: test_full
drpatelh marked this conversation as resolved.
Show resolved Hide resolved
parameters: |
{
"hook_url": "${{ secrets.MEGATESTS_ALERTS_SLACK_HOOK_URL }}",
"outdir": "${{ secrets.TOWER_BUCKET_AZURE }}/rnaseq/results-${{ github.sha }}/aligner_${{ matrix.aligner }}",
"igenomes_base": "${{ secrets.TOWER_IGENOMES_BASE_AZURE }}"
}
wait: false
- uses: actions/upload-artifact@v3
with:
name: Tower debug log file
path: tower_action_*.log
80 changes: 80 additions & 0 deletions .github/workflows/cloud_tests_small.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
name: Run small tests on cloud providers
run-name: Submitting workflow to all cloud providers
on:
pull_request:
drpatelh marked this conversation as resolved.
Show resolved Hide resolved
workflow_dispatch:
inputs:
platform:
description: 'Platform to run test'
required: true
default: 'all'
type: choice
options:
- all
- aws
- azure
- gcp
jobs:
run-small-tests-on-aws:
# if: ${{ github.event.inputs.platform == 'all' || github.event.inputs.platform == 'aws' }}
runs-on: self-hosted
steps:
- uses: seqeralabs/action-tower-launch@add-wait
drpatelh marked this conversation as resolved.
Show resolved Hide resolved
with:
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
compute_env: ${{ secrets.TOWER_CE_AWS_CPU }}
workdir: "${{ secrets.TOWER_BUCKET_AWS }}/work/${{ github.sha }}"
run_name: AWS_${{ github.event.pull_request.head.label || github.head_ref || github.ref }}
profiles: test
parameters: |
{
"outdir": "${{ secrets.TOWER_BUCKET_AWS }}/rnaseq/results-test-${{ github.sha }}"
}
wait: false
- uses: actions/upload-artifact@v3
with:
name: Tower debug log file
path: tower_action_*.log
run-small-tests-on-gcp:
# if: ${{ github.event.inputs.platform == 'all' || github.event.inputs.platform == 'gcp' }}
runs-on: self-hosted
steps:
- uses: seqeralabs/action-tower-launch@add-wait
drpatelh marked this conversation as resolved.
Show resolved Hide resolved
with:
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
compute_env: ${{ secrets.TOWER_CE_GCP_CPU }}
workdir: "${{ secrets.TOWER_BUCKET_GCP }}/work/${{ github.sha }}"
run_name: GCP_${{ github.event.pull_request.head.label || github.head_ref || github.ref }}
profiles: test
parameters: |
{
"outdir": "${{ secrets.TOWER_BUCKET_GCP }}/rnaseq/results-test-${{ github.sha }}"
}
wait: false
- uses: actions/upload-artifact@v3
with:
name: Tower debug log file
path: tower_action_*.log
run-small-tests-on-azure:
# if: ${{ github.event.inputs.platform == 'all' || github.event.inputs.platform == 'azure' }}
runs-on: self-hosted
steps:
- uses: seqeralabs/action-tower-launch@add-wait
drpatelh marked this conversation as resolved.
Show resolved Hide resolved
with:
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
compute_env: ${{ secrets.TOWER_CE_AZURE_CPU }}
workdir: "${{ secrets.TOWER_BUCKET_AZURE }}/work/${{ github.sha }}"
run_name: AZURE_${{ github.event.pull_request.head.label || github.head_ref || github.ref }}
profiles: test
parameters: |
{
"outdir": "${{ secrets.TOWER_BUCKET_AZURE }}/rnaseq/results-test-${{ github.sha }}"
}
wait: false
- uses: actions/upload-artifact@v3
with:
name: Tower debug log file
path: tower_action_*.log