Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge develop with faster tests #199

Merged
merged 195 commits into from
May 31, 2024
Merged
Changes from all commits
Commits
Show all changes
195 commits
Select commit Hold shift + click to select a range
2e2f122
Refactor HeatmapFiles class to include make_heatmaps property in desi…
alsmith151 Apr 26, 2024
efa0f0c
Refactor NonRNAOutput class in design.py to remove computed_field dec…
alsmith151 Apr 26, 2024
65bc64c
fix: correct query function to provide both the ip and control if req…
alsmith151 Apr 26, 2024
6036f76
Refactor control lambda function in peak_call_chip.smk to allow null …
alsmith151 Apr 26, 2024
686825c
Refactor query function in DesignIP class to return IPExperiment object
alsmith151 Apr 26, 2024
fcf7bca
Refactor FastqFile model to handle resolved and absolute paths
alsmith151 May 17, 2024
b7e4006
Refactor peak_call_grouped.smk to add runtime and mem resources for l…
alsmith151 May 19, 2024
47a596d
feat: Refactor pileup_norm.smk to use tile instead of tiles
alsmith151 May 19, 2024
69e21ce
Merge branch 'fix-csaw-norm' into experimental
alsmith151 May 23, 2024
4b4b6ed
Merge branch 'fix-increase-grouped-peak-call-mem' into experimental
alsmith151 May 23, 2024
1fbe50c
Merge branch 'fix-heatmaps-always-created' into experimental
alsmith151 May 23, 2024
024b97a
fix typo design
CChahrour May 28, 2024
9338a77
fix tests fastq path
CChahrour May 28, 2024
7397902
fix merge in design
CChahrour May 28, 2024
81c6e2b
Merge branch 'develop' into experimental
CChahrour May 28, 2024
b22ed22
Merge pull request #191 from alsmith151/experimental
CChahrour May 28, 2024
f5df8c3
fix config
CChahrour May 28, 2024
e57a26e
Merge branch 'develop' of https://github.com/alsmith151/SeqNado into …
CChahrour May 28, 2024
902cdfd
reinstate snp into design
CChahrour May 28, 2024
f6aad2d
revert peaks and design
CChahrour May 28, 2024
4c03a07
fixed typo from merge peak calls
CChahrour May 28, 2024
e63deac
simplify tests
CChahrour May 28, 2024
a808bb1
update test order
CChahrour May 28, 2024
c644e1e
correct test order
CChahrour May 28, 2024
45b21a8
revert test order
CChahrour May 28, 2024
cb66d35
split tests
CChahrour May 28, 2024
90c902e
increase cores for tests
CChahrour May 28, 2024
00e774c
revert single test
CChahrour May 28, 2024
74965c0
merge
CChahrour May 28, 2024
5732241
merge develop to master
CChahrour May 28, 2024
a0cd8e3
Merge branch 'master' into develop
CChahrour May 28, 2024
159eff5
remove codecov
CChahrour May 29, 2024
b8cd8d4
subset data
CChahrour May 30, 2024
524135f
parallel testing and cache conda
CChahrour May 30, 2024
43f2b52
remove conda cache
CChahrour May 30, 2024
76e3643
matrix for parallel testing and cache conda and simg
CChahrour May 30, 2024
0fa2050
fix restore conda env
CChahrour May 30, 2024
4f052b6
fix error in extract env
CChahrour May 30, 2024
229d9d8
fix conda tmp paths
CChahrour May 30, 2024
e676629
add mkdir for conda extract
CChahrour May 30, 2024
10e090f
fix lack of pytest
CChahrour May 30, 2024
9cf3ab3
merge test and min envs
CChahrour May 30, 2024
fb4c16f
rename conda env
CChahrour May 30, 2024
c97deac
fix mamba use for consistency
CChahrour May 30, 2024
59d6785
activate mamba
CChahrour May 30, 2024
70b871b
separate out singularity from setup
CChahrour May 30, 2024
ada89bb
remove setup as dependency in sing
CChahrour May 30, 2024
30d457d
add matrix for sing pull
CChahrour May 30, 2024
ca73a4d
update sing matrix
CChahrour May 30, 2024
63a88f6
fix pull matrix
CChahrour May 30, 2024
8a5cbc5
remove --output from apptainer pull
CChahrour May 30, 2024
f3ee9c8
remove cache steps
CChahrour May 30, 2024
91fcf4b
check env install
CChahrour May 30, 2024
25bf2d0
explicit env naming
CChahrour May 30, 2024
5af7893
add symlink to env
CChahrour May 30, 2024
f8276d2
fix paths to simg
CChahrour May 30, 2024
f142f0a
installs mamba
CChahrour May 30, 2024
39370d8
activate conda env
CChahrour May 30, 2024
d8c3c6a
explicit mamba path
CChahrour May 30, 2024
8708be5
fix test cmd
CChahrour May 30, 2024
d5a9f26
check conda env extract path
CChahrour May 30, 2024
2276831
mk env dir
CChahrour May 30, 2024
bd5c793
remove sing for now
CChahrour May 30, 2024
7932ad1
make all conda
CChahrour May 30, 2024
9bcec43
debug extract env
CChahrour May 30, 2024
ac3a7cd
update env creation
CChahrour May 30, 2024
8df87e3
update env create
CChahrour May 30, 2024
f4ef7bc
verify env creation
CChahrour May 30, 2024
8cab05c
revert to original conda
CChahrour May 30, 2024
59e63d3
keep name of env as test
CChahrour May 30, 2024
8d3e3f1
debug env path
CChahrour May 30, 2024
93a1945
update env create
CChahrour May 30, 2024
db6e111
remove activate env
CChahrour May 30, 2024
f10da9c
cache dependencies
CChahrour May 30, 2024
27a6243
check seqnado v
CChahrour May 30, 2024
603a158
add conda env to path
CChahrour May 30, 2024
0f5a65a
caching only conda
CChahrour May 30, 2024
ea7336c
fix pytest cmd
CChahrour May 30, 2024
44e9a22
add conditions for download
CChahrour May 30, 2024
cc4b0dc
clean saving env
CChahrour May 30, 2024
01408c3
clean saving
CChahrour May 30, 2024
3dc56a4
add upload back
CChahrour May 30, 2024
8bba47f
get cache output for test jobs
CChahrour May 30, 2024
7f17e58
validate conda env
CChahrour May 30, 2024
77ebdaf
edit verify env
CChahrour May 30, 2024
29a1786
fix finding conda
CChahrour May 30, 2024
40b5c1d
clean up verify env
CChahrour May 30, 2024
b9b7a0a
setup and test
CChahrour May 30, 2024
db51c20
remove cond case for verify env
CChahrour May 30, 2024
fd1a9c4
remove source
CChahrour May 30, 2024
dda9c40
simplify
CChahrour May 30, 2024
23d6b5d
update verify env
CChahrour May 30, 2024
0bd437f
use setup env better
CChahrour May 30, 2024
23a5f6d
update caching
CChahrour May 30, 2024
0f2dd74
check conda envs
CChahrour May 30, 2024
ec1a5b5
specify env path
CChahrour May 30, 2024
4025a0d
update caching env
CChahrour May 30, 2024
ded3f8d
fix conda fields
CChahrour May 30, 2024
839f298
fix conda cache
CChahrour May 30, 2024
56a0788
try test
CChahrour May 30, 2024
63fd59a
conda envs from cache
CChahrour May 30, 2024
a38d15d
change path to conda env
CChahrour May 30, 2024
818421c
fix paths
CChahrour May 30, 2024
b84588e
fix mamba version
CChahrour May 30, 2024
ff07cbd
rmove bad options for setup env
CChahrour May 30, 2024
1cd4b08
remove conda channels
CChahrour May 30, 2024
a0fd9ca
edit conda setup opts
CChahrour May 30, 2024
a51cbea
add channels
CChahrour May 30, 2024
ba7c1a2
remove updates
CChahrour May 30, 2024
9b9f536
revert conda
CChahrour May 30, 2024
d7f3fad
update path to source
CChahrour May 30, 2024
ddd2e6c
dont activate env
CChahrour May 30, 2024
4c613d9
fix type in pytest cmd
CChahrour May 30, 2024
1409db4
add apptainer
CChahrour May 30, 2024
6d93f11
typo in test matrix
CChahrour May 30, 2024
02e262f
tidy up
CChahrour May 30, 2024
4cc012b
update names
CChahrour May 30, 2024
6364cac
revert tests but with matrix
CChahrour May 30, 2024
07e17ae
add caching back
CChahrour May 30, 2024
79bbfdd
design for all assays
CChahrour May 30, 2024
2d2954b
remove python setup
CChahrour May 30, 2024
7a6c1c7
add debugging for config test
CChahrour May 30, 2024
b3df2bc
move design to pytest
CChahrour May 30, 2024
786ea90
cache conda
CChahrour May 30, 2024
068bfd2
update pytest cmd
CChahrour May 30, 2024
cb4d353
cache conda
CChahrour May 30, 2024
941fffe
separate pip cache
CChahrour May 30, 2024
a813b1b
update tests
CChahrour May 30, 2024
c8d26a8
update pip cache
CChahrour May 30, 2024
e107206
updates pytest cmd
CChahrour May 30, 2024
5e47bb0
fix config defaults
CChahrour May 30, 2024
ff00486
tidy up test names
CChahrour May 30, 2024
3cd1f57
remove paths from cache
CChahrour May 30, 2024
2bfb95b
set caches
CChahrour May 30, 2024
1734193
try path for cache
CChahrour May 30, 2024
8d5e197
combine caching
CChahrour May 30, 2024
9c2dd21
only install conda if cache not avail
CChahrour May 30, 2024
a6a6d31
restore tests
CChahrour May 30, 2024
d01febe
update env
CChahrour May 30, 2024
9492da0
remove conditions
CChahrour May 30, 2024
8a92072
save simgs as artefacts
CChahrour May 30, 2024
c748673
access artefact images
CChahrour May 30, 2024
9fd6a27
move simg
CChahrour May 30, 2024
a970611
set image files
CChahrour May 30, 2024
8b155ea
set matrix images
CChahrour May 30, 2024
7cd2f21
remove prior load
CChahrour May 30, 2024
81b5516
exclude combos
CChahrour May 30, 2024
004acaa
fix pull cmd
CChahrour May 30, 2024
edffd5b
fix simg paths
CChahrour May 30, 2024
4a9e350
fix matrix
CChahrour May 30, 2024
9905cb2
get list of simgs
CChahrour May 30, 2024
e117f07
get simgs
CChahrour May 31, 2024
d8ec524
fix use simgs
CChahrour May 31, 2024
dc7dc78
pull no retries
CChahrour May 31, 2024
e2f8152
get and extract simg
CChahrour May 31, 2024
5038081
revert tests
CChahrour May 31, 2024
41a2062
remove sing pull dep
CChahrour May 31, 2024
ffd4069
add apptainer
CChahrour May 31, 2024
e8f3338
only setup apptainer for pipeline
CChahrour May 31, 2024
189a75d
cache id
CChahrour May 31, 2024
15d2c8f
cache simgs
CChahrour May 31, 2024
2e48e3b
specify imgs for each assay
CChahrour May 31, 2024
afb14db
fix cond assay
CChahrour May 31, 2024
7576e44
correct cond
CChahrour May 31, 2024
de02507
clean tests
CChahrour May 31, 2024
d34b9ca
Merge pull request #195 from alsmith151/add-tests
CChahrour May 31, 2024
0f79060
Merge branch 'master' of https://github.com/alsmith151/SeqNado into d…
CChahrour May 31, 2024
27f3a16
add download and save simg
CChahrour May 31, 2024
27cd561
fix simg
CChahrour May 31, 2024
094adef
keep pwsh
CChahrour May 31, 2024
e761497
pull using bash
CChahrour May 31, 2024
8b2c868
fix pull cmd
CChahrour May 31, 2024
46ad4a5
check for artefacts before pull
CChahrour May 31, 2024
cac8357
add remotes
CChahrour May 31, 2024
f9ed8ea
cache simg
CChahrour May 31, 2024
e7de2bc
check caching
CChahrour May 31, 2024
e90e6c9
fix conditions for apptainer
CChahrour May 31, 2024
2707120
change conditions for singularity
CChahrour May 31, 2024
841bde7
minor changes to check caching
CChahrour May 31, 2024
19b329b
change simg dir
CChahrour May 31, 2024
c5be50d
add tmp to apptainer mounts
CChahrour May 31, 2024
f45704a
reduce sing cache names
CChahrour May 31, 2024
dd5e294
only cache sing for pipeline
CChahrour May 31, 2024
5c0a80d
clean up cache
CChahrour May 31, 2024
b05c928
check cache
CChahrour May 31, 2024
2c28c22
change cache dirs
CChahrour May 31, 2024
82457fd
update sing cache dir
CChahrour May 31, 2024
be13e25
mount apptainer cache in pytest and use tmp in workflow
CChahrour May 31, 2024
08a5322
remove simg names
CChahrour May 31, 2024
34661ba
correct apptainer cache dir in tests
CChahrour May 31, 2024
ad12ff4
Merge pull request #197 from alsmith151/fix-tests
CChahrour May 31, 2024
8a0bea8
fix typo in workflow
CChahrour May 31, 2024
e9434a0
Merge branch 'master' of https://github.com/alsmith151/SeqNado into d…
CChahrour May 31, 2024
14a65da
merge error in workflow file
CChahrour May 31, 2024
7a6b7f3
Merge branch 'master' into develop
CChahrour May 31, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 1 addition & 10 deletions .github/workflows/test_pipelines.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,13 @@ name: Python package
on: [push]

jobs:
Test:
if: github.event_name == 'pull_request' || github.event_name == 'push'
Test:
if: github.event_name == 'pull_request' || github.event_name == 'push'
runs-on: ubuntu-latest
strategy:
matrix:
test: ["config", "design", "pipeline"]
assay: ["atac", "chip-rx", "chip", "rna-rx", "rna", "snp"]
test: ["config", "design", "pipeline"]
assay: ["atac", "chip-rx", "chip", "rna-rx", "rna", "snp"]

steps:
- uses: actions/checkout@v3
Expand All @@ -28,7 +24,6 @@ jobs:
key: ${{ runner.os }}-env-${{ hashFiles('**/testing.yml') }}
restore-keys: |
${{ runner.os }}-env-

- name: Setup conda
uses: conda-incubator/setup-miniconda@v3
with:
Expand All @@ -41,7 +36,6 @@ jobs:
shell: pwsh
run: |
pip install .

- name: Cache singularity
if: matrix.test == 'pipeline'
id: cache-singularity
Expand All @@ -51,7 +45,6 @@ jobs:
key: ${{ runner.os }}-singularity-${{ hashFiles('tmp/*') }}
restore-keys: |
${{ runner.os }}-singularity-

- uses: eWaterCycle/setup-apptainer@v2
if: matrix.test == 'pipeline'
with:
Expand All @@ -63,7 +56,6 @@ jobs:
run: |
apptainer remote add --no-login SylabsCloud cloud.sylabs.io
apptainer remote use SylabsCloud

- name: Pull singularity images
if: matrix.test == 'pipeline' && steps.cache-singularity.outputs.cache-hit != 'true'
shell: bash
Expand All @@ -77,10 +69,9 @@ jobs:
elif [[ "${{ matrix.assay }}" == "rna-rx" ]]; then
apptainer pull --force library://asmith151/seqnado/seqnado_report:latest
fi

- name: Test ${{ matrix.test }} ${{ matrix.assay }}
shell: pwsh
env:
TMPDIR: /tmp
run: |
pytest tests/test_pipelines.py::test_${{ matrix.test }}[${{ matrix.assay }}] -vv -s --cores 4
pytest tests/test_pipelines.py::test_${{ matrix.test }}[${{ matrix.assay }}] -vv -s --cores 4
Loading