Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run VistoSeg for 2720 Ant (round 4) #40

Closed
lcolladotor opened this issue Dec 1, 2021 · 4 comments
Closed

Run VistoSeg for 2720 Ant (round 4) #40

lcolladotor opened this issue Dec 1, 2021 · 4 comments
Assignees
@madhavitippani @heenadivecha

Comments

@lcolladotor
Copy link
Member

Once #39 is done, we can proceed with this.
@lcolladotor lcolladotor mentioned this issue Dec 1, 2021
Closed
@lcolladotor lcolladotor changed the title Run VistoSeg for 2720 Ant Run VistoSeg for 2720 Ant (round 2) Dec 1, 2021
@lcolladotor lcolladotor mentioned this issue Dec 1, 2021
Closed
@abspangler13 abspangler13 changed the title Run VistoSeg for 2720 Ant (round 2) Run VistoSeg for 2720 Ant (round 4) Dec 6, 2021
@lcolladotor
Copy link
Member Author

@heenadivecha I see that now /dcs04/lieber/lcolladotor/spatialDLPFC_LIBD4035/spatialDLPFC/processed-data/NextSeq/Round4/DLPFC_Br2720_ant_2 has the necessary SpaceRanger outputs to run VistoSeg on this re-run.

This was referenced Dec 8, 2021
Closed
Closed
lcolladotor added a commit that referenced this issue Dec 14, 2021
@lcolladotor @heenadivecha @abspangler13
Complete array job script for re-running countNuclei
91bdf5f 
Missing sample 28 which is the one from #40

Co-authored-by: Heena Divecha <heenadivecha@gmail.com>
Co-authored-by: Abby Spangler <aspangle@gmail.com>
@lcolladotor
Copy link
Member Author

At this point we are missing /dcs04/lieber/lcolladotor/spatialDLPFC_LIBD4035/spatialDLPFC/raw-data/Images/round4/V10B01-002_D1_Br2720_ant_DLPFC_nuclei.mat in order to edit

spatialDLPFC/code/VistoSeg/code/countNuclei_array.sh

Line 9 in 7b40abc

#$ -t 29-30
to be #$ -t 28 in order to run this script for this missing sample.

@lcolladotor
Copy link
Member Author

Heena told me that we are struggling with this mat file due to a Matlab issue that Maddy has been trying to get support from Matlab to help resolve it.

lcolladotor added a commit that referenced this issue Dec 15, 2021
@lcolladotor
Finished re-running VistoSeg countNuclei on 29 samples. Close #48
cf8430a 
Note that #40 is still open though.

Here's some R code for checking the difference between these new 29 files
and the older ones. Basically, 26/29 are identical in terms of the cell
counts, which is excellent news (3 old files didn't exist!). 6 of the 26 had different imagerow/imagecol
information, which is likely due to the changes in the input images. So, well,
re-making these files was still worth it from that sense, to make sure that
they are as accurate as possible.

```R
library("here")
library("glue")

sample_id <- dir(here("processed-data/rerun_spaceranger"))
file_new <- "~/Desktop/tissue_spot_counts.csv"

for(i in sample_id) {
    message("**** Checking sample ", i, " ****")
    if(file.exists(file_new)) unlink(file_new)

    file_old <- here(glue("processed-data/rerun_spaceranger/{i}/outs/spatial/tissue_spot_counts.csv"))
    if(!file.exists(file_old)) {
        message("no old file, skipping")
        next()
    }
    old <- read.csv(file_old)

    Sys.sleep(3)

    system(glue("scp e:/dcs04/lieber/lcolladotor/spatialDLPFC_LIBD4035/spatialDLPFC/processed-data/rerun_spaceranger/{i}/outs/spatial/tissue_spot_counts.csv ~/Desktop/"), ignore.stdout = TRUE, ignore.stderr = TRUE)
    if(!file.exists(file_new)) {
        message("no new file, skipping")
        next()
    }
    new <- read.csv("~/Desktop/tissue_spot_counts.csv")
    if(!identical(old, new)) {
        warning("Old and new spot counts not matching for sample ", i, "!", call. = FALSE)
        print(mapply(identical, old, new))
    }
}
```

I ran the above R code with a local clone of 7b40abc on my laptop,
which compared those files against the newest ones from JHPCE.

Here's the output:

```R
**** Checking sample DLPFC_Br2720_ant_2 ****
no old file, skipping
**** Checking sample DLPFC_Br2720_mid_manual_alignment ****
**** Checking sample DLPFC_Br2720_post_extra_reads ****
no old file, skipping
**** Checking sample DLPFC_Br2743_ant_manual_alignment ****
**** Checking sample DLPFC_Br2743_mid_manual_alignment_extra_reads ****
**** Checking sample DLPFC_Br2743_post_manual_alignment ****
 barcode   tissue      row      col imagerow imagecol    count
    TRUE     TRUE     TRUE     TRUE    FALSE    FALSE     TRUE
**** Checking sample DLPFC_Br3942_ant_manual_alignment ****
**** Checking sample DLPFC_Br3942_mid_manual_alignment ****
 barcode   tissue      row      col imagerow imagecol    count
    TRUE     TRUE     TRUE     TRUE    FALSE    FALSE     TRUE
**** Checking sample DLPFC_Br3942_post_manual_alignment ****
 barcode   tissue      row      col imagerow imagecol    count
    TRUE     TRUE     TRUE     TRUE    FALSE    FALSE     TRUE
**** Checking sample DLPFC_Br6423_ant_manual_alignment_extra_reads ****
**** Checking sample DLPFC_Br6423_mid_manual_alignment ****
 barcode   tissue      row      col imagerow imagecol    count
    TRUE     TRUE     TRUE     TRUE    FALSE    FALSE     TRUE
**** Checking sample DLPFC_Br6423_post_extra_reads ****
no old file, skipping
**** Checking sample DLPFC_Br6432_ant_2 ****
**** Checking sample DLPFC_Br6432_mid_manual_alignment ****
**** Checking sample DLPFC_Br6432_post_manual_alignment ****
**** Checking sample DLPFC_Br6471_ant_manual_alignment_all ****
**** Checking sample DLPFC_Br6471_mid_manual_alignment_all ****
**** Checking sample DLPFC_Br6471_post_manual_alignment_all ****
**** Checking sample DLPFC_Br6522_ant_manual_alignment_all ****
**** Checking sample DLPFC_Br6522_mid_manual_alignment_all ****
**** Checking sample DLPFC_Br6522_post_manual_alignment_all ****
**** Checking sample DLPFC_Br8325_ant_manual_alignment_all ****
**** Checking sample DLPFC_Br8325_mid_2 ****
**** Checking sample DLPFC_Br8325_post_manual_alignment_all ****
**** Checking sample DLPFC_Br8492_ant_manual_alignment ****
 barcode   tissue      row      col imagerow imagecol    count
    TRUE     TRUE     TRUE     TRUE    FALSE    FALSE     TRUE
**** Checking sample DLPFC_Br8492_mid_manual_alignment_extra_reads ****
**** Checking sample DLPFC_Br8492_post_manual_alignment ****
 barcode   tissue      row      col imagerow imagecol    count
    TRUE     TRUE     TRUE     TRUE    FALSE    FALSE     TRUE
**** Checking sample DLPFC_Br8667_ant_extra_reads ****
**** Checking sample DLPFC_Br8667_mid_manual_alignment_all ****
**** Checking sample DLPFC_Br8667_post_manual_alignment_all ****
Warning messages:
1: Old and new spot counts not matching for sample DLPFC_Br2743_post_manual_alignment!
2: Old and new spot counts not matching for sample DLPFC_Br3942_mid_manual_alignment!
3: Old and new spot counts not matching for sample DLPFC_Br3942_post_manual_alignment!
4: Old and new spot counts not matching for sample DLPFC_Br6423_mid_manual_alignment!
5: Old and new spot counts not matching for sample DLPFC_Br8492_ant_manual_alignment!
6: Old and new spot counts not matching for sample DLPFC_Br8492_post_manual_alignment!
```

Here are the new 29 files on JHPCE:

```bash
$ ls -lh processed-data/rerun_spaceranger/*/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 199K Dec 14 17:21 processed-data/rerun_spaceranger/DLPFC_Br2720_mid_manual_alignment/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 199K Dec 14 17:37 processed-data/rerun_spaceranger/DLPFC_Br2720_post_extra_reads/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 199K Dec 14 16:27 processed-data/rerun_spaceranger/DLPFC_Br2743_ant_manual_alignment/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 199K Dec 14 16:17 processed-data/rerun_spaceranger/DLPFC_Br2743_mid_manual_alignment_extra_reads/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 199K Dec 14 16:39 processed-data/rerun_spaceranger/DLPFC_Br2743_post_manual_alignment/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 198K Dec 14 17:47 processed-data/rerun_spaceranger/DLPFC_Br3942_ant_manual_alignment/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 199K Dec 14 18:27 processed-data/rerun_spaceranger/DLPFC_Br3942_mid_manual_alignment/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 199K Dec 14 19:40 processed-data/rerun_spaceranger/DLPFC_Br3942_post_manual_alignment/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 198K Dec 14 19:17 processed-data/rerun_spaceranger/DLPFC_Br6423_ant_manual_alignment_extra_reads/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 198K Dec 14 18:27 processed-data/rerun_spaceranger/DLPFC_Br6423_mid_manual_alignment/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 199K Dec 14 19:28 processed-data/rerun_spaceranger/DLPFC_Br6423_post_extra_reads/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 199K Dec 14 19:56 processed-data/rerun_spaceranger/DLPFC_Br6432_ant_2/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 198K Dec 14 16:30 processed-data/rerun_spaceranger/DLPFC_Br6432_mid_manual_alignment/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 198K Dec 14 17:42 processed-data/rerun_spaceranger/DLPFC_Br6432_post_manual_alignment/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 199K Dec 14 18:18 processed-data/rerun_spaceranger/DLPFC_Br6471_ant_manual_alignment_all/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 200K Dec 14 16:47 processed-data/rerun_spaceranger/DLPFC_Br6471_mid_manual_alignment_all/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 200K Dec 14 17:11 processed-data/rerun_spaceranger/DLPFC_Br6471_post_manual_alignment_all/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 198K Dec 14 17:30 processed-data/rerun_spaceranger/DLPFC_Br6522_ant_manual_alignment_all/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 202K Dec 14 19:12 processed-data/rerun_spaceranger/DLPFC_Br6522_mid_manual_alignment_all/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 200K Dec 14 18:39 processed-data/rerun_spaceranger/DLPFC_Br6522_post_manual_alignment_all/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 198K Dec 14 17:00 processed-data/rerun_spaceranger/DLPFC_Br8325_ant_manual_alignment_all/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 199K Dec 14 16:33 processed-data/rerun_spaceranger/DLPFC_Br8325_mid_2/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 199K Dec 14 17:37 processed-data/rerun_spaceranger/DLPFC_Br8325_post_manual_alignment_all/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 198K Dec 14 18:53 processed-data/rerun_spaceranger/DLPFC_Br8492_ant_manual_alignment/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 198K Dec 14 18:40 processed-data/rerun_spaceranger/DLPFC_Br8492_mid_manual_alignment_extra_reads/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 199K Dec 14 20:21 processed-data/rerun_spaceranger/DLPFC_Br8492_post_manual_alignment/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 198K Dec 14 18:12 processed-data/rerun_spaceranger/DLPFC_Br8667_ant_extra_reads/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 199K Dec 14 17:53 processed-data/rerun_spaceranger/DLPFC_Br8667_mid_manual_alignment_all/outs/spatial/tissue_spot_counts.csv
-rw-rw---- 1 lcollado lieber_lcolladotor 199K Dec 14 17:56 processed-data/rerun_spaceranger/DLPFC_Br8667_post_manual_alignment_all/outs/spatial/tissue_spot_counts.csv
$ ls -lh processed-data/rerun_spaceranger/*/outs/spatial/tissue_spot_counts.csv | wc -l
29
```
@madhavitippani
Copy link
Contributor

madhavitippani commented Dec 15, 2021 •
edited

I think the issue is sometimes MATLAB is not able to use all the memory available on the node and that's why it crashes or gets stuck. I was able to run it fine on a windows system with 160GB RAM (interactively). I tried running it interactively on JHPCE also, but it gave weird results. I also ran it on my MAC with 16GB RAM, but this sample needed ~30GB or even more, so it threw an error (Requested 1260933427x3x1 (28.2GB) array exceeds maximum array size preference (16.0GB). This might cause MATLAB to become unresponsive.). This sample was taking more memory than any other sample in that round, and JHPCE was not able to provide that to MATLAB somehow?

@heenadivecha I ran VNS and saved the outputs to the same folder /dcs04/lieber/lcolladotor/spatialDLPFC_LIBD4035/spatialDLPFC/raw-data/Images/round4. Cluster5 seems to be the one corresponding to nuclei.

lcolladotor added a commit that referenced this issue Dec 17, 2021
@lcolladotor
Add file related to #40
8bcfd71
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@madhavitippani madhavitippani

@heenadivecha heenadivecha

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lcolladotor @madhavitippani @heenadivecha