Frozen data version 1 #63

gwaybio · 2021-03-30T23:35:49Z

I update pycytominer and add associated fixes as described in #62

TODO

Check example batch 2 data
Run full pipeline for both batches, add profiles
Rerun cytominer vs. pycytominer comparison notebook
Rerun consensus signature notebook - note that I will no longer need to recode dose
Rerun spherize notebook

In the next PR, I will migrate from git lfs to dvc

also adding recoded dose to all annotated profiles

and run black

gwaybio · 2021-04-20T20:20:32Z

@shntnu - this is good to go. Sorry for the HUGE amounts of files (most are just profiles).

Please pay extra attention to any updated documentation. Any code changes will require a complete rerun (which I'll only do if absolutely necessary). If necessary, I can address #65 simultaneously.

The next step will be to update to dvc!

shntnu · 2021-04-20T23:21:02Z

consensus/scripts/nbconverted/build-consensus-signatures.py

 ]

 # Output option
 float_format = "%5g"
-compression = "gzip"
+compression_options = {"method": "gzip", "mtime": 1}


consensus/scripts/nbconverted/build-consensus-signatures.py

shntnu · 2021-04-20T23:23:48Z

profiles/README.md

-python profiling_pipeline.py --batch "2017_12_05_Batch2" --plate_prefix "BR" --well_col "Metadata_Well" --plate_col "Metadata_Plate" --extract_cell_line
+```bash
+# Make sure you are in the profiles/ directory
+./run.sh


shntnu · 2021-04-20T23:26:38Z

spherized_profiles/scripts/nbconverted/spherize-batch-effects.py

+        if batch == "2017_12_05_Batch2":
+            spherize_df = (
+                profile_df
+                .groupby("Metadata_cell_line")


No need to edit but pretty sure you don't need this logic (grouping by cell line is trivially valid in batch 1 as well)

Regarding spherizing overall, we will make a lot of changes over time which are likely to improve the quality:

Also group by timepoint and not just cell line because if we don't we might be effectively factoring out subspaces we care about

Figure out epsilon

Drop outliers prior to spherizing

Also group by timepoint and not just cell line because if we don't we might be effectively factoring out subspaces we care about

nice catch, i'll update this

Figure out epsilon

I think i'll skip this one for data freeze version 1. It seems to far off in the future to wait on.

Drop outliers prior to spherizing

Yep, we already do this. We're discussing in #65 (comment) and once we decide there, I'll run this notebook again with all the suggested changes (I'll also add drop outliers to consensus signature generation notebook)

Great! I agree you should skip epslion optimization, and it's great you're updating 1. as well

And great that the outlier features will be gone! For dropping outlier samples, we need new functionality in pycytominer, right? See cytomining/pycytominer#140

pycytominer currently has a very crude outlier removal strategy (here), which can be specified as an operation in feature_select().

It should easily handle Michael's features as defined in #65, but you're right, the method needs to be improved in the future. Thanks for opening that pycytominer issue!

IIUC, it needs to be a different operation altogether (a row filter, not a column filter; drop_outlier_features is the latter), but we can discuss this in cytomining/pycytominer#140

All set here from my end.

shntnu · 2021-04-20T23:31:24Z

Please pay extra attention to any updated documentation.

I focused on only .py and .md

I didn't see any documentation changes other than the README.md

Did I miss any documentation?

Any code changes will require a complete rerun (which I'll only do if absolutely necessary). If necessary, I can address #65 simultaneously.

No need to do #65 except perhaps this thing you suggested:

I can add a prominent note to make sure these are dropped in all downstream analyses in a README in #63

gwaybio · 2021-04-21T20:06:50Z

Did I miss any documentation?

Nope, I think you got it all. Thank you!

I can make all of these changes, and we should be good to merge soon

adding dose info to profile readme, adding outlier feature drop to consensus readme

spherize based on both time and cell line, and perform blocklist feature selection beforehand

gwaybio · 2021-04-28T18:35:43Z

Alright @shntnu - this is ready for your eyes again. Here is what changed:

Adding note to how we decided to remove features 8a794f9
Rearranging documentation and minor fixes 6adb1c3
Adding the new blocklist features 510fe92
Using the new blocklist feature file in feature selection 2201875
Adding the updated consensus files 49ec1c7
Fixing sphering (sphering based on time as well and dropping new blocklist features) 7c96161
Adding spherized profiles 3a47efd
Adding a UMAP visualization of spherized profiles acb3e89

shntnu

Yikes, I totally missed the notification 23 days ago, sorry!

Everything looks good. Thank you for documenting everything point-by-point.

Just one q:

Adding the new blocklist features 510fe92

Do you want to update this as well? https://figshare.com/articles/dataset/Blacklist_Features_-_Cell_Profiler/10255811

(but no need to wait for that of course)

gwaybio · 2021-05-21T20:13:38Z

Do you want to update this as well? https://figshare.com/articles/dataset/Blacklist_Features_-_Cell_Profiler/10255811

I don't think so... Although i do think that we want to update this figshare document to include other version-specific CellProfiler blocklists. At the very least, much more thought needs to go into updating it (much more thought for me at least!). As a separate but related note: I really want to do a deep dive into CellProfiler features... i think its the first step to understanding generic morphology features, which we'll want to annotate with more interpretable biology. It'll also help us with interpreting DeepProfiler features in the future.

shntnu · 2021-05-21T20:19:06Z

As a separate but related note: I really want to do a deep dive into CellProfiler features... i think its the first step to understanding generic morphology features, which we'll want to annotate with more interpretable biology. It'll also help us with interpreting DeepProfiler features in the future.

There are two resources that I can think of that will be relevant for this effort

A well-documented readme https://github.com/carpenterlab/2016_bray_natprot/wiki/What-do-Cell-Painting-features-mean%3F
An incomplete notebook
https://github.com/cytomining/cytominergallery/blob/105be87878d13024ef283e284cf085351b498883/notebooks/empty_readouts.Rmd (knit here https://rpubs.com/shantanu/cp_feature_stats)

gwaybio added 3 commits March 30, 2021 23:23

upgrade pycytominer and pandas

bd91993

update profiling pipeline for upgraded pycytominer

3d25221

also adding recoded dose to all annotated profiles

add dose utility file

59ec5a5

gwaybio marked this pull request as draft March 30, 2021 23:35

gwaybio mentioned this pull request Mar 30, 2021

Should we reprocess all profiles before frozen data release? #62

Closed

2 tasks

gwaybio added 10 commits April 1, 2021 21:11

fix compression in aggregate

377f113

add cyto_utils import

be4f3f7

and run black

add an execution script

999276d

remove instructions from duplicate sources

79b32ae

reprocess batch 2 data

e2ecf84

reprocess batch 1 data

6b03638

rerun comparison module with reprocessed profiles

46c51e8

update consensus signatures for data freeze

7121f04

update spherized data freeze

260dfb1

update per cell line spherize

e17a47c

gwaybio mentioned this pull request Apr 14, 2021

Spherized data UMAP figures #64

Closed

gwaybio marked this pull request as ready for review April 20, 2021 20:14

gwaybio changed the title ~~Frozen data~~ Frozen data version 1 Apr 20, 2021

gwaybio requested a review from shntnu April 20, 2021 20:14

This was referenced Apr 20, 2021

Dropping outlier features #65

Closed

Add --no-name gzip flag to compression file output #50

Closed

shntnu reviewed Apr 20, 2021

View reviewed changes

consensus/scripts/nbconverted/build-consensus-signatures.py Show resolved Hide resolved

shntnu reviewed Apr 20, 2021

View reviewed changes

gwaybio added 2 commits April 21, 2021 16:31

add link to drop outlier features in broadinstitute#65

8a794f9

update documentation

6adb1c3

adding dose info to profile readme, adding outlier feature drop to consensus readme

gwaybio added 7 commits April 26, 2021 14:58

add additional blocklist features

510fe92

remove outlier features using a file

2201875

add consensus profiles

49ec1c7

fix spherize two ways:

7c96161

spherize based on both time and cell line, and perform blocklist feature selection beforehand

add updated spherized profiles

3a47efd

add umap visualization of spherized data

acb3e89

track large umap pdfs

ad71b3c

gwaybio requested a review from shntnu May 19, 2021 16:02

shntnu approved these changes May 21, 2021

View reviewed changes

gwaybio merged commit 60d85f6 into broadinstitute:master May 21, 2021

gwaybio deleted the data-freeze branch May 21, 2021 20:13

This was referenced May 21, 2021

Second batch of lincs data #57

Closed

Consensus profiles are missing MOA and target information #49

Closed

gwaybio mentioned this pull request May 31, 2021

Adding profiles to dvc #66

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Frozen data version 1 #63

Frozen data version 1 #63

gwaybio commented Mar 30, 2021 •

edited

Loading

gwaybio commented Apr 20, 2021

shntnu Apr 20, 2021

shntnu Apr 20, 2021

shntnu Apr 20, 2021

shntnu Apr 20, 2021

gwaybio Apr 21, 2021

shntnu Apr 21, 2021

shntnu Apr 21, 2021

gwaybio Apr 21, 2021

shntnu Apr 21, 2021

shntnu commented Apr 20, 2021

gwaybio commented Apr 21, 2021

gwaybio commented Apr 28, 2021

shntnu left a comment

gwaybio commented May 21, 2021

shntnu commented May 21, 2021

Frozen data version 1 #63

Frozen data version 1 #63

Conversation

gwaybio commented Mar 30, 2021 • edited Loading

TODO

gwaybio commented Apr 20, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shntnu commented Apr 20, 2021

gwaybio commented Apr 21, 2021

gwaybio commented Apr 28, 2021

shntnu left a comment

Choose a reason for hiding this comment

gwaybio commented May 21, 2021

shntnu commented May 21, 2021

gwaybio commented Mar 30, 2021 •

edited

Loading