Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix per sample custom config #66

Merged
merged 2 commits into from May 26, 2021
Merged

Conversation

alexvpickering
Copy link
Contributor

@alexvpickering alexvpickering commented May 24, 2021

Linked Issue

Also fixed in data-ingest: https://github.com/biomage-ltd/data-ingest/pull/34

Background

The function add_custom_config_per_sample wasn't subsetting the dataset. This created a bug where the entire object was being used to calculate default cellSizeDistribution, numGenesVsNumUmis and doubletScore thresholds. For example all the doubletScore thresholds end up the same:

image

This is the same value as if I calculate the threshold with the full seurat object:

image

Whereas e.g. for an actual sample subset we get something different for each sample:

image

Copy link
Contributor

@JuanluOnieva-biomage JuanluOnieva-biomage left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's weird because so far it has been working well. Moreover, if it gives wrong parameter for numGeneVsnumUMIs it should also fail for the cellSizeDistribution. What could happen is that since the numGeneVsnumUMIs' config parameter depends on the number of cells (p.level = 1/numCells), in the case that of larger experiments with similar size across samples we will have similar value.

Nevertheless, I like the refactoring. I think it is better to use the subset function (which belongs to the SeuratObject package), however filtering by columns seems to give the same result so I leave it to your choice and I approve the PR.

@alexvpickering
Copy link
Contributor Author

It seems to have been a bug introduced by the fact that we have both sample and samples as meta.data columns:

# samples was unique samples
samples <- unique(scseq$samples)
samples
# [1] "T0_1"   "T0_2"   "Ctrl_1" "Ctrl_2" "Gent_1" "Gent_2"

dim(scseq)
# [1]  18689 267867

dim(subset(scseq, samples %in% "T0_1"))
# [1] 18689  2103

sample <-  "T0_1"
dim(subset(scseq, samples %in% sample))
# [1]  18689 267867

I'm a fan of the slice notation because it's universal for R objects and clear when your subsetting columns/rows. Also not going to have this bug lol :)

@alexvpickering alexvpickering merged commit 0725b96 into develop May 26, 2021
@alexvpickering alexvpickering deleted the fix-per-sample-config branch May 26, 2021 15:17
kafkasl pushed a commit that referenced this pull request Jun 15, 2021
* fix per sample custom config

* loop through unique samples
kafkasl added a commit that referenced this pull request Jun 17, 2021
* handle NA doublet scores (#61)

* Added no integration (#59)

* Added no integration

* Update qc-runner/src/dataIntegration.r

Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com>

Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com>

* Bump lodash from 4.17.20 to 4.17.21 in /local-runner (#58)

Bumps [lodash](https://github.com/lodash/lodash) from 4.17.20 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](lodash/lodash@4.17.20...4.17.21)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump hosted-git-info from 2.8.8 to 2.8.9 in /local-runner (#64)

Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/npm/hosted-git-info/releases)
- [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md)
- [Commits](npm/hosted-git-info@v2.8.8...v2.8.9)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Na doublet score (#62)

* handle NA doublet scores

* treat NA as doublet_score=0

* Add FastMNN step 6-7 (#60)

* Add FastMNN step 6-7

* Fix dockerfile

* Fix scdata

* change order

* Fix try-catch

* Change value to unisample and fastmnn

* Fix assay

* Add package

* Add dependencies

* Change to pak

* Add active.reduciton and pak

* Add var.explained

* Add comments

* Persist misc

* added improved error handling and traceback (#55)

* added improved error handling and traceback

* removed dump.frames

* no sample output for integration step

* moved state_fail_logic to withCallingHandlers block

* removed intended debug error

* Display the command used to launch docker from lambda (#63)

* Move GEM2S as a step in pipeline (#56)

* refactor wrapper

* Added gem2s functions to pipeline

* Adapted pipeline to new schema for gem2s

* renamed one task

* fixes for gem2s

* renamed qc-runner to pipeline-runner

* fixed tryCatch from rebase

* changed pipeline name to qc

* init upload-to-aws

* init meta_sets

* Added sample sets

* complete upload-to-aws

* add s3/dynamodb functions

* add buckets and tables

* writing input to meta.json file

* changed some pipeline references to gem2s/pipeline

* avoiding creating new containers for gem2s or qc if they already exist

* attaching both pipeline containers output (qc & gem2s) to pipeline process stdout

* just a lonenly runaway letter

* added dependencies

* typo

* make docker build for me

* send dynamodb items to api

* reflect nan bugfix from data-inegest

* temporarily fix for paws.common version 0.3.11

* change SNS messag type

* hotfix

* fix

* fix

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Pol Alvarez <pol.avms@gmail.com>
Co-authored-by: Anugerah Erlaut <aerlaut@live.com>

* Dependency issues and cellsets (#65)

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* Add sns updates when tasks finish (#67)

Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* fix per sample custom config (#66)

* fix per sample custom config

* loop through unique samples

* Fix seurat obj not found (#71)

* Sample ids property sent as list instead of string
* fix seurat_obj not found

Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Xavier Verges <460418+xverges@users.noreply.github.com>

* Handle gem2s response refactoring (#68)

* Add sns updates when tasks finish

* Handle gem2s response refactoring

* Minor renaming

* Fix

* Fix

Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* [BIOMAGE-1016] Sample uuids fix (#72)

* Sample ids property sent as list instead of string

* Fix

* Draft fix for uuids

* Fix

* fix

* fix

* checks

Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* Fix fast mnn (#75)

* Sample ids property sent as list instead of string

* Add FastMNN

* Restore 5_Upload-to-aws

* [BIOMAGE-1040] - Fix metadata structures (#74)

* fix data structures
* fix for empty metadata
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* [BIOMAGE-1009] Fix error sns messages (#73)

* Refactor and add method dedicated to only sending sns error messages

* Add experimentId

* Move classifier to current position in ui

* Move into one single line

Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>
Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* Add experimentID and ingestionDate (#77)

* Add experimentID and ingestionDate

* Update comment

* Init filter stats (#76)

* init calc_filter_stats

* add filter stats for cellSizeDistribution

* return filter stats for all filter steps

* Test

* restore spacing for ease

* revert

* reverting spacing changes

* more de-change

* fix conflict

* check samples

Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>

* move code from data-ingest (#78)

* fix mtcontent plotdata error (#79)

* new flow for releases (#83)

* added deployment for release/hotfixes PRs to master (#85)

* using tokens to allow flow to trigger testing

* Fix doublet settings (#81)

* test doublet settings

* Fix error in doublet scores config

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* building releases/hotfixes on PR instead of push

* Fix determination of HOST_IP on Linux (#91)

* Fix determination of HOST_IP on Linux

* Replace spaces with tab

* Add tty flag to fix local output (#92)

* Update ci-develop.yaml

* Fix default doublet config (#82)

* use seed and fix config storage

* use scDblFinder to set default threshold

* remove gitignore

* remove debug

* change boolean logic

* Rm auto and enabled (#88)

* switch from orig.ident to samples (#95)

* Move to multipart upload (#94)

* Move to multipart upload

* Add print

* Fix progress-bar

* hardcode DEBUG_PATH (#93)

* hardcode DEBUG_PATH

* fix function name

* Remove hardcoded 10x (it is now hardcoded in the ui only) (#96)

Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* re-enable per sample config (#97)

* added r.utils dependency install which is required by seurat-wrappers (#99)

* Deprecated change for ci.yaml

Co-authored-by: Alex Pickering <alexvpickering@gmail.com>
Co-authored-by: ogibson-biomage <76957896+ogibson-biomage@users.noreply.github.com>
Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: seb-mueller <sebm@posteo.de>
Co-authored-by: Xavier Vergés <460418+xverges@users.noreply.github.com>
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Anugerah Erlaut <aerlaut@live.com>
Co-authored-by: Martin Fosco <cosa65@users.noreply.github.com>
Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>
Co-authored-by: Anugerah Erlaut <anugerah@biomage.net>
Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>
Co-authored-by: James Robinson <robioson@pm.me>
ivababukova pushed a commit that referenced this pull request Jul 19, 2021
* add harmony integration (#98)

* add harmony

* refactor

* remove mistakes

* add dispersions for harmony

* Fix p.level (#102)

* Merge master into dev branch (#100)

* handle NA doublet scores (#61)

* Added no integration (#59)

* Added no integration

* Update qc-runner/src/dataIntegration.r

Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com>

Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com>

* Bump lodash from 4.17.20 to 4.17.21 in /local-runner (#58)

Bumps [lodash](https://github.com/lodash/lodash) from 4.17.20 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](lodash/lodash@4.17.20...4.17.21)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump hosted-git-info from 2.8.8 to 2.8.9 in /local-runner (#64)

Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/npm/hosted-git-info/releases)
- [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md)
- [Commits](npm/hosted-git-info@v2.8.8...v2.8.9)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Na doublet score (#62)

* handle NA doublet scores

* treat NA as doublet_score=0

* Add FastMNN step 6-7 (#60)

* Add FastMNN step 6-7

* Fix dockerfile

* Fix scdata

* change order

* Fix try-catch

* Change value to unisample and fastmnn

* Fix assay

* Add package

* Add dependencies

* Change to pak

* Add active.reduciton and pak

* Add var.explained

* Add comments

* Persist misc

* added improved error handling and traceback (#55)

* added improved error handling and traceback

* removed dump.frames

* no sample output for integration step

* moved state_fail_logic to withCallingHandlers block

* removed intended debug error

* Display the command used to launch docker from lambda (#63)

* Move GEM2S as a step in pipeline (#56)

* refactor wrapper

* Added gem2s functions to pipeline

* Adapted pipeline to new schema for gem2s

* renamed one task

* fixes for gem2s

* renamed qc-runner to pipeline-runner

* fixed tryCatch from rebase

* changed pipeline name to qc

* init upload-to-aws

* init meta_sets

* Added sample sets

* complete upload-to-aws

* add s3/dynamodb functions

* add buckets and tables

* writing input to meta.json file

* changed some pipeline references to gem2s/pipeline

* avoiding creating new containers for gem2s or qc if they already exist

* attaching both pipeline containers output (qc & gem2s) to pipeline process stdout

* just a lonenly runaway letter

* added dependencies

* typo

* make docker build for me

* send dynamodb items to api

* reflect nan bugfix from data-inegest

* temporarily fix for paws.common version 0.3.11

* change SNS messag type

* hotfix

* fix

* fix

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Pol Alvarez <pol.avms@gmail.com>
Co-authored-by: Anugerah Erlaut <aerlaut@live.com>

* Dependency issues and cellsets (#65)

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* Add sns updates when tasks finish (#67)

Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* fix per sample custom config (#66)

* fix per sample custom config

* loop through unique samples

* Fix seurat obj not found (#71)

* Sample ids property sent as list instead of string
* fix seurat_obj not found

Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Xavier Verges <460418+xverges@users.noreply.github.com>

* Handle gem2s response refactoring (#68)

* Add sns updates when tasks finish

* Handle gem2s response refactoring

* Minor renaming

* Fix

* Fix

Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* [BIOMAGE-1016] Sample uuids fix (#72)

* Sample ids property sent as list instead of string

* Fix

* Draft fix for uuids

* Fix

* fix

* fix

* checks

Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* Fix fast mnn (#75)

* Sample ids property sent as list instead of string

* Add FastMNN

* Restore 5_Upload-to-aws

* [BIOMAGE-1040] - Fix metadata structures (#74)

* fix data structures
* fix for empty metadata
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* [BIOMAGE-1009] Fix error sns messages (#73)

* Refactor and add method dedicated to only sending sns error messages

* Add experimentId

* Move classifier to current position in ui

* Move into one single line

Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>
Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* Add experimentID and ingestionDate (#77)

* Add experimentID and ingestionDate

* Update comment

* Init filter stats (#76)

* init calc_filter_stats

* add filter stats for cellSizeDistribution

* return filter stats for all filter steps

* Test

* restore spacing for ease

* revert

* reverting spacing changes

* more de-change

* fix conflict

* check samples

Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>

* move code from data-ingest (#78)

* fix mtcontent plotdata error (#79)

* new flow for releases (#83)

* added deployment for release/hotfixes PRs to master (#85)

* using tokens to allow flow to trigger testing

* Fix doublet settings (#81)

* test doublet settings

* Fix error in doublet scores config

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* building releases/hotfixes on PR instead of push

* Fix determination of HOST_IP on Linux (#91)

* Fix determination of HOST_IP on Linux

* Replace spaces with tab

* Add tty flag to fix local output (#92)

* Update ci-develop.yaml

* Fix default doublet config (#82)

* use seed and fix config storage

* use scDblFinder to set default threshold

* remove gitignore

* remove debug

* change boolean logic

* Rm auto and enabled (#88)

* switch from orig.ident to samples (#95)

* Move to multipart upload (#94)

* Move to multipart upload

* Add print

* Fix progress-bar

* hardcode DEBUG_PATH (#93)

* hardcode DEBUG_PATH

* fix function name

* Remove hardcoded 10x (it is now hardcoded in the ui only) (#96)

Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* re-enable per sample config (#97)

* added r.utils dependency install which is required by seurat-wrappers (#99)

* Deprecated change for ci.yaml

Co-authored-by: Alex Pickering <alexvpickering@gmail.com>
Co-authored-by: ogibson-biomage <76957896+ogibson-biomage@users.noreply.github.com>
Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: seb-mueller <sebm@posteo.de>
Co-authored-by: Xavier Vergés <460418+xverges@users.noreply.github.com>
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Anugerah Erlaut <aerlaut@live.com>
Co-authored-by: Martin Fosco <cosa65@users.noreply.github.com>
Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>
Co-authored-by: Anugerah Erlaut <anugerah@biomage.net>
Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>
Co-authored-by: James Robinson <robioson@pm.me>

* done i think (#101)

* Add build as a dependency of run (#103)

Also add new run-only target

* Init R package and test classifier filter (#104)

* init r package

* ignore debug

* remove file names within file

* use styler

* cleanup

* fix Dockerfile

* init filter_emptydrops tests

* fix introduced error for demo image

* add sample aware test

* update README; make task names look distinct

* update README

* update gem2s step messages

* add timer to gem2s steps

* Enable Slack build failure notifications

* Fix defaults (#109)

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* Unlog y values (#112)

* unlog cellsizedistribution plot y values
* switch plot numbers to improve readibility

* Clean pipeline code (#105)

* Clean pipeline code

* Update pipeline-runner/R/gem2s-6-prepare_experiment.R

Co-authored-by: Alex Pickering <alexvpickering@gmail.com>

* Update pipeline-runner/R/gem2s-helpers.R

Co-authored-by: Alex Pickering <alexvpickering@gmail.com>

* Update pipeline-runner/R/gem2s-helpers.R

* Update pipeline-runner/R/gem2s-helpers.R

Co-authored-by: Alex Pickering <alexvpickering@gmail.com>

* fixes

* Update pipeline-runner/R/gem2s-helpers.R

Co-authored-by: Alex Pickering <alexvpickering@gmail.com>

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Alex Pickering <alexvpickering@gmail.com>

* Init filter high mito and filter gene umi outlier tests (#114)

* init filter_gene_umi_outlier tests

* init mito filter tests; fix calc_filter_stats

* Init cellszdoublet tests (#118)

* init filter_gene_umi_outlier tests

* init mito filter tests; fix calc_filter_stats

* Added tests for cellsize and doublets

* export filter_low_cellsize

* simplify parts of filter_low_cellsize

* restyle and update comment for filter_doublets test

* fix filter_doublets documentation

Co-authored-by: Alex Pickering <alexvpickering@gmail.com>
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* Removed embedding (#107)

* Removed embedding

* Update pipeline-runner/R/qc-7-embed_and_cluster.R

* Update pipeline-runner/R/qc-7-embed_and_cluster.R

* Update pipeline-runner/R/qc-7-embed_and_cluster.R

* Fix

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* Fix (#113)

* Fix

* typo

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* init renv (#110)

* try restore remoter docker-compose

* Update pipeline-runner/renv.lock.init

* Update pipeline-runner/renv.lock

* try update all packages

* Fix

* update README; remove R version

* harmony init lockfile

* fix

Co-authored-by: ogibson-biomage <76957896+ogibson-biomage@users.noreply.github.com>
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* Move defaultFilterSettings and missing per sample config additions to the pipeline (#119)

* Add defaultFilterSettings and per sample config for classifier and mito content

* Update gem2s-6-prepare_experiment.R

Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

Co-authored-by: Alex Pickering <alexvpickering@gmail.com>
Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com>
Co-authored-by: ogibson-biomage <76957896+ogibson-biomage@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: seb-mueller <sebm@posteo.de>
Co-authored-by: Xavier Vergés <460418+xverges@users.noreply.github.com>
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Anugerah Erlaut <aerlaut@live.com>
Co-authored-by: Martin Fosco <cosa65@users.noreply.github.com>
Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>
Co-authored-by: Anugerah Erlaut <anugerah@biomage.net>
Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>
Co-authored-by: James Robinson <robioson@pm.me>
Co-authored-by: StefanBabukov <stefanbabukov98@gmail.com>
Co-authored-by: James Robinson <james@biomage.net>
Co-authored-by: Marcell Pek <marcell.pek@marcellp.net>
ivababukova pushed a commit that referenced this pull request Jul 20, 2021
* add harmony integration (#98)

* add harmony

* refactor

* remove mistakes

* add dispersions for harmony

* Fix p.level (#102)

* Merge master into dev branch (#100)

* handle NA doublet scores (#61)

* Added no integration (#59)

* Added no integration

* Update qc-runner/src/dataIntegration.r

Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com>

Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com>

* Bump lodash from 4.17.20 to 4.17.21 in /local-runner (#58)

Bumps [lodash](https://github.com/lodash/lodash) from 4.17.20 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](lodash/lodash@4.17.20...4.17.21)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump hosted-git-info from 2.8.8 to 2.8.9 in /local-runner (#64)

Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/npm/hosted-git-info/releases)
- [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md)
- [Commits](npm/hosted-git-info@v2.8.8...v2.8.9)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Na doublet score (#62)

* handle NA doublet scores

* treat NA as doublet_score=0

* Add FastMNN step 6-7 (#60)

* Add FastMNN step 6-7

* Fix dockerfile

* Fix scdata

* change order

* Fix try-catch

* Change value to unisample and fastmnn

* Fix assay

* Add package

* Add dependencies

* Change to pak

* Add active.reduciton and pak

* Add var.explained

* Add comments

* Persist misc

* added improved error handling and traceback (#55)

* added improved error handling and traceback

* removed dump.frames

* no sample output for integration step

* moved state_fail_logic to withCallingHandlers block

* removed intended debug error

* Display the command used to launch docker from lambda (#63)

* Move GEM2S as a step in pipeline (#56)

* refactor wrapper

* Added gem2s functions to pipeline

* Adapted pipeline to new schema for gem2s

* renamed one task

* fixes for gem2s

* renamed qc-runner to pipeline-runner

* fixed tryCatch from rebase

* changed pipeline name to qc

* init upload-to-aws

* init meta_sets

* Added sample sets

* complete upload-to-aws

* add s3/dynamodb functions

* add buckets and tables

* writing input to meta.json file

* changed some pipeline references to gem2s/pipeline

* avoiding creating new containers for gem2s or qc if they already exist

* attaching both pipeline containers output (qc & gem2s) to pipeline process stdout

* just a lonenly runaway letter

* added dependencies

* typo

* make docker build for me

* send dynamodb items to api

* reflect nan bugfix from data-inegest

* temporarily fix for paws.common version 0.3.11

* change SNS messag type

* hotfix

* fix

* fix

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Pol Alvarez <pol.avms@gmail.com>
Co-authored-by: Anugerah Erlaut <aerlaut@live.com>

* Dependency issues and cellsets (#65)

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* Add sns updates when tasks finish (#67)

Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* fix per sample custom config (#66)

* fix per sample custom config

* loop through unique samples

* Fix seurat obj not found (#71)

* Sample ids property sent as list instead of string
* fix seurat_obj not found

Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Xavier Verges <460418+xverges@users.noreply.github.com>

* Handle gem2s response refactoring (#68)

* Add sns updates when tasks finish

* Handle gem2s response refactoring

* Minor renaming

* Fix

* Fix

Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* [BIOMAGE-1016] Sample uuids fix (#72)

* Sample ids property sent as list instead of string

* Fix

* Draft fix for uuids

* Fix

* fix

* fix

* checks

Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* Fix fast mnn (#75)

* Sample ids property sent as list instead of string

* Add FastMNN

* Restore 5_Upload-to-aws

* [BIOMAGE-1040] - Fix metadata structures (#74)

* fix data structures
* fix for empty metadata
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* [BIOMAGE-1009] Fix error sns messages (#73)

* Refactor and add method dedicated to only sending sns error messages

* Add experimentId

* Move classifier to current position in ui

* Move into one single line

Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>
Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* Add experimentID and ingestionDate (#77)

* Add experimentID and ingestionDate

* Update comment

* Init filter stats (#76)

* init calc_filter_stats

* add filter stats for cellSizeDistribution

* return filter stats for all filter steps

* Test

* restore spacing for ease

* revert

* reverting spacing changes

* more de-change

* fix conflict

* check samples

Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>

* move code from data-ingest (#78)

* fix mtcontent plotdata error (#79)

* new flow for releases (#83)

* added deployment for release/hotfixes PRs to master (#85)

* using tokens to allow flow to trigger testing

* Fix doublet settings (#81)

* test doublet settings

* Fix error in doublet scores config

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* building releases/hotfixes on PR instead of push

* Fix determination of HOST_IP on Linux (#91)

* Fix determination of HOST_IP on Linux

* Replace spaces with tab

* Add tty flag to fix local output (#92)

* Update ci-develop.yaml

* Fix default doublet config (#82)

* use seed and fix config storage

* use scDblFinder to set default threshold

* remove gitignore

* remove debug

* change boolean logic

* Rm auto and enabled (#88)

* switch from orig.ident to samples (#95)

* Move to multipart upload (#94)

* Move to multipart upload

* Add print

* Fix progress-bar

* hardcode DEBUG_PATH (#93)

* hardcode DEBUG_PATH

* fix function name

* Remove hardcoded 10x (it is now hardcoded in the ui only) (#96)

Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* re-enable per sample config (#97)

* added r.utils dependency install which is required by seurat-wrappers (#99)

* Deprecated change for ci.yaml

Co-authored-by: Alex Pickering <alexvpickering@gmail.com>
Co-authored-by: ogibson-biomage <76957896+ogibson-biomage@users.noreply.github.com>
Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: seb-mueller <sebm@posteo.de>
Co-authored-by: Xavier Vergés <460418+xverges@users.noreply.github.com>
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Anugerah Erlaut <aerlaut@live.com>
Co-authored-by: Martin Fosco <cosa65@users.noreply.github.com>
Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>
Co-authored-by: Anugerah Erlaut <anugerah@biomage.net>
Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>
Co-authored-by: James Robinson <robioson@pm.me>

* done i think (#101)

* Add build as a dependency of run (#103)

Also add new run-only target

* Init R package and test classifier filter (#104)

* init r package

* ignore debug

* remove file names within file

* use styler

* cleanup

* fix Dockerfile

* init filter_emptydrops tests

* fix introduced error for demo image

* add sample aware test

* update README; make task names look distinct

* update README

* update gem2s step messages

* add timer to gem2s steps

* Enable Slack build failure notifications

* Fix defaults (#109)

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* Unlog y values (#112)

* unlog cellsizedistribution plot y values
* switch plot numbers to improve readibility

* Clean pipeline code (#105)

* Clean pipeline code

* Update pipeline-runner/R/gem2s-6-prepare_experiment.R

Co-authored-by: Alex Pickering <alexvpickering@gmail.com>

* Update pipeline-runner/R/gem2s-helpers.R

Co-authored-by: Alex Pickering <alexvpickering@gmail.com>

* Update pipeline-runner/R/gem2s-helpers.R

* Update pipeline-runner/R/gem2s-helpers.R

Co-authored-by: Alex Pickering <alexvpickering@gmail.com>

* fixes

* Update pipeline-runner/R/gem2s-helpers.R

Co-authored-by: Alex Pickering <alexvpickering@gmail.com>

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Alex Pickering <alexvpickering@gmail.com>

* Init filter high mito and filter gene umi outlier tests (#114)

* init filter_gene_umi_outlier tests

* init mito filter tests; fix calc_filter_stats

* Init cellszdoublet tests (#118)

* init filter_gene_umi_outlier tests

* init mito filter tests; fix calc_filter_stats

* Added tests for cellsize and doublets

* export filter_low_cellsize

* simplify parts of filter_low_cellsize

* restyle and update comment for filter_doublets test

* fix filter_doublets documentation

Co-authored-by: Alex Pickering <alexvpickering@gmail.com>
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* Removed embedding (#107)

* Removed embedding

* Update pipeline-runner/R/qc-7-embed_and_cluster.R

* Update pipeline-runner/R/qc-7-embed_and_cluster.R

* Update pipeline-runner/R/qc-7-embed_and_cluster.R

* Fix

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* Fix (#113)

* Fix

* typo

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* init renv (#110)

* try restore remoter docker-compose

* Update pipeline-runner/renv.lock.init

* Update pipeline-runner/renv.lock

* try update all packages

* Fix

* update README; remove R version

* harmony init lockfile

* fix

Co-authored-by: ogibson-biomage <76957896+ogibson-biomage@users.noreply.github.com>
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>

* Move defaultFilterSettings and missing per sample config additions to the pipeline (#119)

* Add defaultFilterSettings and per sample config for classifier and mito content

* Update gem2s-6-prepare_experiment.R

Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* init density kneeplot data (#111)

* init density kneeplot data

* update plot

* use unique fdr/ranks

* get_bcranks_plot_data works

* fix

* Fix

* Rollback unnecessary empty plot data addition

Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>

* fix empty guidata case

Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com>
Co-authored-by: Pol Alvarez <pol.avms@gmail.com>
Co-authored-by: ogibson-biomage <76957896+ogibson-biomage@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: seb-mueller <sebm@posteo.de>
Co-authored-by: Xavier Vergés <460418+xverges@users.noreply.github.com>
Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local>
Co-authored-by: Anugerah Erlaut <aerlaut@live.com>
Co-authored-by: Martin Fosco <cosa65@users.noreply.github.com>
Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>
Co-authored-by: Anugerah Erlaut <anugerah@biomage.net>
Co-authored-by: Martin Fosco <martinfosco@192.168.1.7>
Co-authored-by: James Robinson <robioson@pm.me>
Co-authored-by: StefanBabukov <stefanbabukov98@gmail.com>
Co-authored-by: James Robinson <james@biomage.net>
Co-authored-by: Marcell Pek <marcell.pek@marcellp.net>
alexvpickering pushed a commit that referenced this pull request Dec 8, 2022
[BIOMAGE-2253] Subset experiment flag
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants