[BIOMAGE-1009] Fix error sns messages #73
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced May 29, 2021
|
Something important to note here is that the qc pipeline was also not working correctly and displaying error messages, since the error handler in the pipeline didn't send sns messages then the only way to have the api know about this was with a race condition between the pipeline breaking and the last step that didn't fail triggering the pipeline's qc status update (and catching the FAIL on the next step). A simple way to check this with develop branch is adding a stop("ERROR") in any part of classifier (the first step that is run) in the pipeline, since it does not have a previous step then the api never is triggered for an update and the ui doesn't update to show the error (unless we refresh it). These changes don't aim to completely refactor the structure that is being used currently to manage sns messages from the pipeline (because it means more changes and more testing that would take more time when there are more pressing issues) but it does refactor enough so that errors are actually being handled and sent as an sns notification whenever they take place, both in qc and gem2s. |
aerlaut
reviewed
May 31, 2021
there is a try catch block for any error which causes the pipeline (qc or gem2s) to send an error notification to sns. I don't think that the pipeline should do much error handling apart from perhaps something to return more informative error messages (which is not in the scope of this ticket, becuase it would be for the ticket of returning correct error messages down to the api/ui). |
kafkasl
pushed a commit
that referenced
this pull request
Jun 15, 2021
* Refactor and add method dedicated to only sending sns error messages * Add experimentId * Move classifier to current position in ui * Move into one single line Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local>
kafkasl
added a commit
that referenced
this pull request
Jun 17, 2021
* handle NA doublet scores (#61) * Added no integration (#59) * Added no integration * Update qc-runner/src/dataIntegration.r Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com> Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com> * Bump lodash from 4.17.20 to 4.17.21 in /local-runner (#58) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.20 to 4.17.21. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.20...4.17.21) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump hosted-git-info from 2.8.8 to 2.8.9 in /local-runner (#64) Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9. - [Release notes](https://github.com/npm/hosted-git-info/releases) - [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md) - [Commits](npm/hosted-git-info@v2.8.8...v2.8.9) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Na doublet score (#62) * handle NA doublet scores * treat NA as doublet_score=0 * Add FastMNN step 6-7 (#60) * Add FastMNN step 6-7 * Fix dockerfile * Fix scdata * change order * Fix try-catch * Change value to unisample and fastmnn * Fix assay * Add package * Add dependencies * Change to pak * Add active.reduciton and pak * Add var.explained * Add comments * Persist misc * added improved error handling and traceback (#55) * added improved error handling and traceback * removed dump.frames * no sample output for integration step * moved state_fail_logic to withCallingHandlers block * removed intended debug error * Display the command used to launch docker from lambda (#63) * Move GEM2S as a step in pipeline (#56) * refactor wrapper * Added gem2s functions to pipeline * Adapted pipeline to new schema for gem2s * renamed one task * fixes for gem2s * renamed qc-runner to pipeline-runner * fixed tryCatch from rebase * changed pipeline name to qc * init upload-to-aws * init meta_sets * Added sample sets * complete upload-to-aws * add s3/dynamodb functions * add buckets and tables * writing input to meta.json file * changed some pipeline references to gem2s/pipeline * avoiding creating new containers for gem2s or qc if they already exist * attaching both pipeline containers output (qc & gem2s) to pipeline process stdout * just a lonenly runaway letter * added dependencies * typo * make docker build for me * send dynamodb items to api * reflect nan bugfix from data-inegest * temporarily fix for paws.common version 0.3.11 * change SNS messag type * hotfix * fix * fix Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Pol Alvarez <pol.avms@gmail.com> Co-authored-by: Anugerah Erlaut <aerlaut@live.com> * Dependency issues and cellsets (#65) Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * Add sns updates when tasks finish (#67) Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * fix per sample custom config (#66) * fix per sample custom config * loop through unique samples * Fix seurat obj not found (#71) * Sample ids property sent as list instead of string * fix seurat_obj not found Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Xavier Verges <460418+xverges@users.noreply.github.com> * Handle gem2s response refactoring (#68) * Add sns updates when tasks finish * Handle gem2s response refactoring * Minor renaming * Fix * Fix Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * [BIOMAGE-1016] Sample uuids fix (#72) * Sample ids property sent as list instead of string * Fix * Draft fix for uuids * Fix * fix * fix * checks Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * Fix fast mnn (#75) * Sample ids property sent as list instead of string * Add FastMNN * Restore 5_Upload-to-aws * [BIOMAGE-1040] - Fix metadata structures (#74) * fix data structures * fix for empty metadata Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * [BIOMAGE-1009] Fix error sns messages (#73) * Refactor and add method dedicated to only sending sns error messages * Add experimentId * Move classifier to current position in ui * Move into one single line Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * Add experimentID and ingestionDate (#77) * Add experimentID and ingestionDate * Update comment * Init filter stats (#76) * init calc_filter_stats * add filter stats for cellSizeDistribution * return filter stats for all filter steps * Test * restore spacing for ease * revert * reverting spacing changes * more de-change * fix conflict * check samples Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> * move code from data-ingest (#78) * fix mtcontent plotdata error (#79) * new flow for releases (#83) * added deployment for release/hotfixes PRs to master (#85) * using tokens to allow flow to trigger testing * Fix doublet settings (#81) * test doublet settings * Fix error in doublet scores config Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * building releases/hotfixes on PR instead of push * Fix determination of HOST_IP on Linux (#91) * Fix determination of HOST_IP on Linux * Replace spaces with tab * Add tty flag to fix local output (#92) * Update ci-develop.yaml * Fix default doublet config (#82) * use seed and fix config storage * use scDblFinder to set default threshold * remove gitignore * remove debug * change boolean logic * Rm auto and enabled (#88) * switch from orig.ident to samples (#95) * Move to multipart upload (#94) * Move to multipart upload * Add print * Fix progress-bar * hardcode DEBUG_PATH (#93) * hardcode DEBUG_PATH * fix function name * Remove hardcoded 10x (it is now hardcoded in the ui only) (#96) Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * re-enable per sample config (#97) * added r.utils dependency install which is required by seurat-wrappers (#99) * Deprecated change for ci.yaml Co-authored-by: Alex Pickering <alexvpickering@gmail.com> Co-authored-by: ogibson-biomage <76957896+ogibson-biomage@users.noreply.github.com> Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: seb-mueller <sebm@posteo.de> Co-authored-by: Xavier Vergés <460418+xverges@users.noreply.github.com> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Anugerah Erlaut <aerlaut@live.com> Co-authored-by: Martin Fosco <cosa65@users.noreply.github.com> Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> Co-authored-by: Anugerah Erlaut <anugerah@biomage.net> Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> Co-authored-by: James Robinson <robioson@pm.me>
ivababukova
pushed a commit
that referenced
this pull request
Jul 19, 2021
* add harmony integration (#98) * add harmony * refactor * remove mistakes * add dispersions for harmony * Fix p.level (#102) * Merge master into dev branch (#100) * handle NA doublet scores (#61) * Added no integration (#59) * Added no integration * Update qc-runner/src/dataIntegration.r Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com> Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com> * Bump lodash from 4.17.20 to 4.17.21 in /local-runner (#58) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.20 to 4.17.21. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.20...4.17.21) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump hosted-git-info from 2.8.8 to 2.8.9 in /local-runner (#64) Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9. - [Release notes](https://github.com/npm/hosted-git-info/releases) - [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md) - [Commits](npm/hosted-git-info@v2.8.8...v2.8.9) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Na doublet score (#62) * handle NA doublet scores * treat NA as doublet_score=0 * Add FastMNN step 6-7 (#60) * Add FastMNN step 6-7 * Fix dockerfile * Fix scdata * change order * Fix try-catch * Change value to unisample and fastmnn * Fix assay * Add package * Add dependencies * Change to pak * Add active.reduciton and pak * Add var.explained * Add comments * Persist misc * added improved error handling and traceback (#55) * added improved error handling and traceback * removed dump.frames * no sample output for integration step * moved state_fail_logic to withCallingHandlers block * removed intended debug error * Display the command used to launch docker from lambda (#63) * Move GEM2S as a step in pipeline (#56) * refactor wrapper * Added gem2s functions to pipeline * Adapted pipeline to new schema for gem2s * renamed one task * fixes for gem2s * renamed qc-runner to pipeline-runner * fixed tryCatch from rebase * changed pipeline name to qc * init upload-to-aws * init meta_sets * Added sample sets * complete upload-to-aws * add s3/dynamodb functions * add buckets and tables * writing input to meta.json file * changed some pipeline references to gem2s/pipeline * avoiding creating new containers for gem2s or qc if they already exist * attaching both pipeline containers output (qc & gem2s) to pipeline process stdout * just a lonenly runaway letter * added dependencies * typo * make docker build for me * send dynamodb items to api * reflect nan bugfix from data-inegest * temporarily fix for paws.common version 0.3.11 * change SNS messag type * hotfix * fix * fix Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Pol Alvarez <pol.avms@gmail.com> Co-authored-by: Anugerah Erlaut <aerlaut@live.com> * Dependency issues and cellsets (#65) Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * Add sns updates when tasks finish (#67) Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * fix per sample custom config (#66) * fix per sample custom config * loop through unique samples * Fix seurat obj not found (#71) * Sample ids property sent as list instead of string * fix seurat_obj not found Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Xavier Verges <460418+xverges@users.noreply.github.com> * Handle gem2s response refactoring (#68) * Add sns updates when tasks finish * Handle gem2s response refactoring * Minor renaming * Fix * Fix Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * [BIOMAGE-1016] Sample uuids fix (#72) * Sample ids property sent as list instead of string * Fix * Draft fix for uuids * Fix * fix * fix * checks Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * Fix fast mnn (#75) * Sample ids property sent as list instead of string * Add FastMNN * Restore 5_Upload-to-aws * [BIOMAGE-1040] - Fix metadata structures (#74) * fix data structures * fix for empty metadata Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * [BIOMAGE-1009] Fix error sns messages (#73) * Refactor and add method dedicated to only sending sns error messages * Add experimentId * Move classifier to current position in ui * Move into one single line Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * Add experimentID and ingestionDate (#77) * Add experimentID and ingestionDate * Update comment * Init filter stats (#76) * init calc_filter_stats * add filter stats for cellSizeDistribution * return filter stats for all filter steps * Test * restore spacing for ease * revert * reverting spacing changes * more de-change * fix conflict * check samples Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> * move code from data-ingest (#78) * fix mtcontent plotdata error (#79) * new flow for releases (#83) * added deployment for release/hotfixes PRs to master (#85) * using tokens to allow flow to trigger testing * Fix doublet settings (#81) * test doublet settings * Fix error in doublet scores config Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * building releases/hotfixes on PR instead of push * Fix determination of HOST_IP on Linux (#91) * Fix determination of HOST_IP on Linux * Replace spaces with tab * Add tty flag to fix local output (#92) * Update ci-develop.yaml * Fix default doublet config (#82) * use seed and fix config storage * use scDblFinder to set default threshold * remove gitignore * remove debug * change boolean logic * Rm auto and enabled (#88) * switch from orig.ident to samples (#95) * Move to multipart upload (#94) * Move to multipart upload * Add print * Fix progress-bar * hardcode DEBUG_PATH (#93) * hardcode DEBUG_PATH * fix function name * Remove hardcoded 10x (it is now hardcoded in the ui only) (#96) Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * re-enable per sample config (#97) * added r.utils dependency install which is required by seurat-wrappers (#99) * Deprecated change for ci.yaml Co-authored-by: Alex Pickering <alexvpickering@gmail.com> Co-authored-by: ogibson-biomage <76957896+ogibson-biomage@users.noreply.github.com> Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: seb-mueller <sebm@posteo.de> Co-authored-by: Xavier Vergés <460418+xverges@users.noreply.github.com> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Anugerah Erlaut <aerlaut@live.com> Co-authored-by: Martin Fosco <cosa65@users.noreply.github.com> Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> Co-authored-by: Anugerah Erlaut <anugerah@biomage.net> Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> Co-authored-by: James Robinson <robioson@pm.me> * done i think (#101) * Add build as a dependency of run (#103) Also add new run-only target * Init R package and test classifier filter (#104) * init r package * ignore debug * remove file names within file * use styler * cleanup * fix Dockerfile * init filter_emptydrops tests * fix introduced error for demo image * add sample aware test * update README; make task names look distinct * update README * update gem2s step messages * add timer to gem2s steps * Enable Slack build failure notifications * Fix defaults (#109) Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * Unlog y values (#112) * unlog cellsizedistribution plot y values * switch plot numbers to improve readibility * Clean pipeline code (#105) * Clean pipeline code * Update pipeline-runner/R/gem2s-6-prepare_experiment.R Co-authored-by: Alex Pickering <alexvpickering@gmail.com> * Update pipeline-runner/R/gem2s-helpers.R Co-authored-by: Alex Pickering <alexvpickering@gmail.com> * Update pipeline-runner/R/gem2s-helpers.R * Update pipeline-runner/R/gem2s-helpers.R Co-authored-by: Alex Pickering <alexvpickering@gmail.com> * fixes * Update pipeline-runner/R/gem2s-helpers.R Co-authored-by: Alex Pickering <alexvpickering@gmail.com> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Alex Pickering <alexvpickering@gmail.com> * Init filter high mito and filter gene umi outlier tests (#114) * init filter_gene_umi_outlier tests * init mito filter tests; fix calc_filter_stats * Init cellszdoublet tests (#118) * init filter_gene_umi_outlier tests * init mito filter tests; fix calc_filter_stats * Added tests for cellsize and doublets * export filter_low_cellsize * simplify parts of filter_low_cellsize * restyle and update comment for filter_doublets test * fix filter_doublets documentation Co-authored-by: Alex Pickering <alexvpickering@gmail.com> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * Removed embedding (#107) * Removed embedding * Update pipeline-runner/R/qc-7-embed_and_cluster.R * Update pipeline-runner/R/qc-7-embed_and_cluster.R * Update pipeline-runner/R/qc-7-embed_and_cluster.R * Fix Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * Fix (#113) * Fix * typo Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * init renv (#110) * try restore remoter docker-compose * Update pipeline-runner/renv.lock.init * Update pipeline-runner/renv.lock * try update all packages * Fix * update README; remove R version * harmony init lockfile * fix Co-authored-by: ogibson-biomage <76957896+ogibson-biomage@users.noreply.github.com> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * Move defaultFilterSettings and missing per sample config additions to the pipeline (#119) * Add defaultFilterSettings and per sample config for classifier and mito content * Update gem2s-6-prepare_experiment.R Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> Co-authored-by: Alex Pickering <alexvpickering@gmail.com> Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com> Co-authored-by: ogibson-biomage <76957896+ogibson-biomage@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: seb-mueller <sebm@posteo.de> Co-authored-by: Xavier Vergés <460418+xverges@users.noreply.github.com> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Anugerah Erlaut <aerlaut@live.com> Co-authored-by: Martin Fosco <cosa65@users.noreply.github.com> Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> Co-authored-by: Anugerah Erlaut <anugerah@biomage.net> Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> Co-authored-by: James Robinson <robioson@pm.me> Co-authored-by: StefanBabukov <stefanbabukov98@gmail.com> Co-authored-by: James Robinson <james@biomage.net> Co-authored-by: Marcell Pek <marcell.pek@marcellp.net>
ivababukova
pushed a commit
that referenced
this pull request
Jul 20, 2021
* add harmony integration (#98) * add harmony * refactor * remove mistakes * add dispersions for harmony * Fix p.level (#102) * Merge master into dev branch (#100) * handle NA doublet scores (#61) * Added no integration (#59) * Added no integration * Update qc-runner/src/dataIntegration.r Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com> Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com> * Bump lodash from 4.17.20 to 4.17.21 in /local-runner (#58) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.20 to 4.17.21. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.20...4.17.21) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump hosted-git-info from 2.8.8 to 2.8.9 in /local-runner (#64) Bumps [hosted-git-info](https://github.com/npm/hosted-git-info) from 2.8.8 to 2.8.9. - [Release notes](https://github.com/npm/hosted-git-info/releases) - [Changelog](https://github.com/npm/hosted-git-info/blob/v2.8.9/CHANGELOG.md) - [Commits](npm/hosted-git-info@v2.8.8...v2.8.9) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Na doublet score (#62) * handle NA doublet scores * treat NA as doublet_score=0 * Add FastMNN step 6-7 (#60) * Add FastMNN step 6-7 * Fix dockerfile * Fix scdata * change order * Fix try-catch * Change value to unisample and fastmnn * Fix assay * Add package * Add dependencies * Change to pak * Add active.reduciton and pak * Add var.explained * Add comments * Persist misc * added improved error handling and traceback (#55) * added improved error handling and traceback * removed dump.frames * no sample output for integration step * moved state_fail_logic to withCallingHandlers block * removed intended debug error * Display the command used to launch docker from lambda (#63) * Move GEM2S as a step in pipeline (#56) * refactor wrapper * Added gem2s functions to pipeline * Adapted pipeline to new schema for gem2s * renamed one task * fixes for gem2s * renamed qc-runner to pipeline-runner * fixed tryCatch from rebase * changed pipeline name to qc * init upload-to-aws * init meta_sets * Added sample sets * complete upload-to-aws * add s3/dynamodb functions * add buckets and tables * writing input to meta.json file * changed some pipeline references to gem2s/pipeline * avoiding creating new containers for gem2s or qc if they already exist * attaching both pipeline containers output (qc & gem2s) to pipeline process stdout * just a lonenly runaway letter * added dependencies * typo * make docker build for me * send dynamodb items to api * reflect nan bugfix from data-inegest * temporarily fix for paws.common version 0.3.11 * change SNS messag type * hotfix * fix * fix Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Pol Alvarez <pol.avms@gmail.com> Co-authored-by: Anugerah Erlaut <aerlaut@live.com> * Dependency issues and cellsets (#65) Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * Add sns updates when tasks finish (#67) Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * fix per sample custom config (#66) * fix per sample custom config * loop through unique samples * Fix seurat obj not found (#71) * Sample ids property sent as list instead of string * fix seurat_obj not found Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Xavier Verges <460418+xverges@users.noreply.github.com> * Handle gem2s response refactoring (#68) * Add sns updates when tasks finish * Handle gem2s response refactoring * Minor renaming * Fix * Fix Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * [BIOMAGE-1016] Sample uuids fix (#72) * Sample ids property sent as list instead of string * Fix * Draft fix for uuids * Fix * fix * fix * checks Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * Fix fast mnn (#75) * Sample ids property sent as list instead of string * Add FastMNN * Restore 5_Upload-to-aws * [BIOMAGE-1040] - Fix metadata structures (#74) * fix data structures * fix for empty metadata Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * [BIOMAGE-1009] Fix error sns messages (#73) * Refactor and add method dedicated to only sending sns error messages * Add experimentId * Move classifier to current position in ui * Move into one single line Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * Add experimentID and ingestionDate (#77) * Add experimentID and ingestionDate * Update comment * Init filter stats (#76) * init calc_filter_stats * add filter stats for cellSizeDistribution * return filter stats for all filter steps * Test * restore spacing for ease * revert * reverting spacing changes * more de-change * fix conflict * check samples Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> * move code from data-ingest (#78) * fix mtcontent plotdata error (#79) * new flow for releases (#83) * added deployment for release/hotfixes PRs to master (#85) * using tokens to allow flow to trigger testing * Fix doublet settings (#81) * test doublet settings * Fix error in doublet scores config Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * building releases/hotfixes on PR instead of push * Fix determination of HOST_IP on Linux (#91) * Fix determination of HOST_IP on Linux * Replace spaces with tab * Add tty flag to fix local output (#92) * Update ci-develop.yaml * Fix default doublet config (#82) * use seed and fix config storage * use scDblFinder to set default threshold * remove gitignore * remove debug * change boolean logic * Rm auto and enabled (#88) * switch from orig.ident to samples (#95) * Move to multipart upload (#94) * Move to multipart upload * Add print * Fix progress-bar * hardcode DEBUG_PATH (#93) * hardcode DEBUG_PATH * fix function name * Remove hardcoded 10x (it is now hardcoded in the ui only) (#96) Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * re-enable per sample config (#97) * added r.utils dependency install which is required by seurat-wrappers (#99) * Deprecated change for ci.yaml Co-authored-by: Alex Pickering <alexvpickering@gmail.com> Co-authored-by: ogibson-biomage <76957896+ogibson-biomage@users.noreply.github.com> Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: seb-mueller <sebm@posteo.de> Co-authored-by: Xavier Vergés <460418+xverges@users.noreply.github.com> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Anugerah Erlaut <aerlaut@live.com> Co-authored-by: Martin Fosco <cosa65@users.noreply.github.com> Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> Co-authored-by: Anugerah Erlaut <anugerah@biomage.net> Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> Co-authored-by: James Robinson <robioson@pm.me> * done i think (#101) * Add build as a dependency of run (#103) Also add new run-only target * Init R package and test classifier filter (#104) * init r package * ignore debug * remove file names within file * use styler * cleanup * fix Dockerfile * init filter_emptydrops tests * fix introduced error for demo image * add sample aware test * update README; make task names look distinct * update README * update gem2s step messages * add timer to gem2s steps * Enable Slack build failure notifications * Fix defaults (#109) Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * Unlog y values (#112) * unlog cellsizedistribution plot y values * switch plot numbers to improve readibility * Clean pipeline code (#105) * Clean pipeline code * Update pipeline-runner/R/gem2s-6-prepare_experiment.R Co-authored-by: Alex Pickering <alexvpickering@gmail.com> * Update pipeline-runner/R/gem2s-helpers.R Co-authored-by: Alex Pickering <alexvpickering@gmail.com> * Update pipeline-runner/R/gem2s-helpers.R * Update pipeline-runner/R/gem2s-helpers.R Co-authored-by: Alex Pickering <alexvpickering@gmail.com> * fixes * Update pipeline-runner/R/gem2s-helpers.R Co-authored-by: Alex Pickering <alexvpickering@gmail.com> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Alex Pickering <alexvpickering@gmail.com> * Init filter high mito and filter gene umi outlier tests (#114) * init filter_gene_umi_outlier tests * init mito filter tests; fix calc_filter_stats * Init cellszdoublet tests (#118) * init filter_gene_umi_outlier tests * init mito filter tests; fix calc_filter_stats * Added tests for cellsize and doublets * export filter_low_cellsize * simplify parts of filter_low_cellsize * restyle and update comment for filter_doublets test * fix filter_doublets documentation Co-authored-by: Alex Pickering <alexvpickering@gmail.com> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * Removed embedding (#107) * Removed embedding * Update pipeline-runner/R/qc-7-embed_and_cluster.R * Update pipeline-runner/R/qc-7-embed_and_cluster.R * Update pipeline-runner/R/qc-7-embed_and_cluster.R * Fix Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * Fix (#113) * Fix * typo Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * init renv (#110) * try restore remoter docker-compose * Update pipeline-runner/renv.lock.init * Update pipeline-runner/renv.lock * try update all packages * Fix * update README; remove R version * harmony init lockfile * fix Co-authored-by: ogibson-biomage <76957896+ogibson-biomage@users.noreply.github.com> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> * Move defaultFilterSettings and missing per sample config additions to the pipeline (#119) * Add defaultFilterSettings and per sample config for classifier and mito content * Update gem2s-6-prepare_experiment.R Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * init density kneeplot data (#111) * init density kneeplot data * update plot * use unique fdr/ranks * get_bcranks_plot_data works * fix * Fix * Rollback unnecessary empty plot data addition Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> * fix empty guidata case Co-authored-by: Juanlu <76958039+JuanluOnieva-biomage@users.noreply.github.com> Co-authored-by: Pol Alvarez <pol.avms@gmail.com> Co-authored-by: ogibson-biomage <76957896+ogibson-biomage@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: seb-mueller <sebm@posteo.de> Co-authored-by: Xavier Vergés <460418+xverges@users.noreply.github.com> Co-authored-by: Oliver Gibson <olivergibson@MacBook-Pro-de-Oliver.local> Co-authored-by: Anugerah Erlaut <aerlaut@live.com> Co-authored-by: Martin Fosco <cosa65@users.noreply.github.com> Co-authored-by: Martin Fosco <martinfosco@Martins-MacBook-Pro.local> Co-authored-by: Anugerah Erlaut <anugerah@biomage.net> Co-authored-by: Martin Fosco <martinfosco@192.168.1.7> Co-authored-by: James Robinson <robioson@pm.me> Co-authored-by: StefanBabukov <stefanbabukov98@gmail.com> Co-authored-by: James Robinson <james@biomage.net> Co-authored-by: Marcell Pek <marcell.pek@marcellp.net>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background
Link to issue
https://biomage.atlassian.net/browse/BIOMAGE-1009
Link to staging deployment URL
https://ui-martinfosco-ui289-api136-p.scp-staging.biomage.net/
Links to any Pull Requests related to this
hms-dbmi-cellenics/ui#289
hms-dbmi-cellenics/api#136
Anything else the reviewers should know about the changes here
Changes
Code changes
Definition of DONE
Your changes will be ready for merging after each of the steps below have been completed:
Testing
To set up easy local testing with inframock, follow the instructions here: https://github.com/biomage-ltd/inframock
To deploy to the staging environment, follow the instructions here: https://github.com/biomage-ltd/biomage-utils
Documentation updates
Is all relevant documentation updated to reflect the proposed changes in this PR?
Approvers
Just before merging:
unstagescript in here: https://github.com/biomage-ltd/biomage-utils is executed. This script cleans up your deployment to stagingOptional