Upload example of transforming raw MERFISH -> spaceTx formatted data #478

dganguli · 2018-08-23T23:31:40Z

Remaining work includes:

Running this tool against all raw data
Uploading formatted data to cloud
Using experiment API against this dataset to test parsing larger datasets

dganguli · 2018-08-23T23:32:40Z

@joshmoore -- take a look at the raw tiff files on s3. I suspect there's metadata in them that can be used to obviate my explicit map. also curious what you think! will work on an nd2 example shortly.

codecov-io · 2018-08-23T23:34:02Z

Codecov Report

Merging #478 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #478   +/-   ##
=======================================
  Coverage   84.56%   84.56%           
=======================================
  Files          73       73           
  Lines        2812     2812           
=======================================
  Hits         2378     2378           
  Misses        434      434

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3e29d5b...bc2fc48. Read the comment docs.

joshmoore · 2018-08-24T10:47:19Z

Ran get_merfish*py
Renamed for x in hybridization-fov*tiff; do mv $x ${x//H/T}; done as a workaround
Created a file with the contents hybridization-fov_000-Z0-T<0-7>-C<0-1>.tiff file & loaded in Bio-Formats

i.e. if the input files had been named as above with the one extra file, the FOV writer would have generated the same output (assuming my code is correct, which I'll now verify...)

You stopped at 16 (154 MB) of the 496 (71 GB) TIFFs to just get this working ASAP? (Or is something else going on? S3 didn't contain an XLS for cross-checking.)

I suspect there's metadata in them that can be used to obviate my explicit map.

I did some digging, @dganguli, but didn't find any metadata. Do you know anything about the provenance of these files?

joshmoore · 2018-08-24T11:53:29Z

With the last commit on spacetx-fov-writer and ignoring file names and checksums, here's the diff:

$ diff -w <(grep -vE "(file|sha256)" /tmp/merfish-deep/hybridization-fov_000.json) <(grep -vE "(file|sha256)" /tmp/merfish-josh/hybridization-fov_000.json)
14c14,16
<     "extras": {},
---
>     "extras": {
>       "OME": "hybridization-fov_000.companion.ome"
>     },
438c440
<     "version": "0.0.0"
---
>     "version": "1.0.0"

ambrosejcarr · 2018-08-24T15:53:07Z

@joshmoore You work fast. 😮

ambrosejcarr

Would love a bit of clarification on a comment, otherwise this looks good. Thanks @dganguli !

ambrosejcarr · 2018-08-24T15:57:24Z

examples/get_merfish_u20s_data.py

+class MERFISHTile(FetchedTile):
+    def __init__(self, file_path, hyb, ch):
+        self.file_path = file_path
+        # how to index tiles from indices into multi-page tiff


I want to make sure I'm clear on what this means:
# maps information on the acquisition channel and round to an index that identifies the correct page in the original multi-page tiff file

so each fov_00x.tiff loads into numpy as a 18X2048X2048 array. that first axis corresponds to some round and color channel. the map dictates how to index into that first axis based on the round/channel you want.

16 rounds + dapi + ___ ?

failed attempt at a membrane stain

joshmoore · 2018-08-24T16:28:40Z

You work fast.

You guys have to sleep at some point!

ambrosejcarr · 2018-08-24T16:30:59Z

You guys have to sleep at some point!

Says who? :-)

dganguli · 2018-08-24T19:26:33Z

I did some digging, @dganguli, but didn't find any metadata. Do you know anything about the provenance of these files?

Really? Nothing fun embedded in the tiffs? I do not know anything about the provenance of these files.

dganguli · 2018-08-24T19:27:17Z

You stopped at 16 (154 MB) of the 496 (71 GB) TIFFs to just get this working ASAP? (Or is something else going on? S3 didn't contain an XLS for cross-checking.)

Oops the 16 is a bug. Fixing. I'll also add the XLS to s3.

dganguli · 2018-08-24T19:30:32Z

@joshmoore : s3://czi.starfish.data.public/browse/raw/20180820/merfish_u2os/data_organization.xlsx

is the xls file to check against.

joshmoore · 2018-08-24T20:02:02Z

is the xls file to check against.

Cheers.

joshmoore · 2018-08-27T09:06:51Z

Really? Nothing fun embedded in the tiffs? I do not know anything about the provenance of these files.

$ tiffinfo ...
TIFF Directory at offset 0x9000d14 (150998292)
  Image Width: 2048 Image Length: 2048
  Bits/Sample: 16
  Compression Scheme: None
  Photometric Interpretation: min-is-black
  Planar Configuration: single image plane
  Software: MATLAB

My guess is going to be that the generating MATLAB script didn't store anything.

For those following along, here's the contents of the spreadsheet:

In [8]: print(df.to_string())
         0                             1                                                  2          3             4      5      6                            7                                                  8                     9              10             11
0   bitName                     imageType                                        imageRegExp  bitNumber  imagingRound  color  frame            fiducialImageType                                     fiducialRegExp  fiducialImagingRound  fiducialFrame  fiducialColor
1    RS0015      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...          1             1    650      2     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     1              3            561
2    RS0083      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...          2             1    750      1     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     1              3            561
3    RS0095      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...          3             2    750      1     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     2              3            561
4    RS0109      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...          4             2    650      2     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     2              3            561
5    RS0175      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...          5             3    650      2     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     3              3            561
6    RS0237      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...          6             3    750      1     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     3              3            561
7    RS0247      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...          7             4    650      2     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     4              3            561
8    RS0255      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...          8             4    750      1     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     4              3            561
9    RS0307      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...          9             5    750      1     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     5              3            561
10   RS0332      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...         10             5    650      2     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     5              3            561
11   RS0343      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...         11             6    650      2     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     6              3            561
12   RS0384      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...         12             6    750      1     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     6              3            561
13   RS0406      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...         13             7    650      2     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     7              3            561
14   RS0451      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...         14             7    750      1     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     7              3            561
15   RS0468      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...         15             8    750      1     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     8              3            561
16   RS0548      Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...         16             8    650      2     Conventional_750_650_560  (?<imageType>\w+)_(?<fov>[0-9]+)_(?<imagingRou...                     8              3            561
17      WGA  CostainImages_1_488_405_fast                   (?<imageType>\w+)_(?<fov>[0-9]+)         17            -1    488      1  CostainImages_2_750_650_560                   (?<imageType>\w+)_(?<fov>[0-9]+)                    -1              3            561
18     DAPI  CostainImages_1_488_405_fast                   (?<imageType>\w+)_(?<fov>[0-9]+)         18            -1    405      2  CostainImages_2_750_650_560                   (?<imageType>\w+)_(?<fov>[0-9]+)                    -1              3            561

joshmoore · 2018-08-27T14:40:41Z

With:

Docker & bump to slicedimage 0.0.3 sptx-format#19
Use the value of the contents map sptx-format#20
Manual creation of a (fake) codebook
Manual addition of "fov" values:

$ grep '"fov": 0' /data/merfish-2018082-a/*.json
/data/merfish-2018082-a/hybridization-fov_000.json:    "fov": 0,
/data/merfish-2018082-a/nuclei-fov_000.json:    "fov": 0

I was able to validate the output of this PR:

$ docker run --rm -v /data/merfish-2018082-a/:/data sptx-format --experiment-json /data/experiment.json && echo yay
yay

joshmoore · 2018-08-30T11:40:15Z

@dganguli @ttung : fovs provisionally added in spacetx/slicedimage#55

see: spacetx/starfish#478

dganguli added 3 commits August 23, 2018 16:25

get_merfish_u2os initial commit

861a517

remove property annotation

ef44dba

remove print statements

f8714da

dganguli requested review from ambrosejcarr and ttung August 23, 2018 23:31

joshmoore mentioned this pull request Aug 24, 2018

Docker & bump to slicedimage 0.0.3 spacetx/sptx-format#19

Merged

4 tasks

ambrosejcarr approved these changes Aug 24, 2018

View reviewed changes

set num_fov to 496

bc2fc48

dganguli merged commit afec66d into master Aug 24, 2018

This was referenced Sep 3, 2018

Upload raw MERFISH data to browse folder #415

Closed

Raw Data Available #414

Closed

Use FOV builder to generate synthetic datasets for ingestion #418

Closed

Synthetic dataset examples in spaceTx format #355

Closed

ttung deleted the dg-merfish-u20s-public branch September 27, 2018 18:10

joshmoore added a commit to spacetx/spacetx-writer that referenced this pull request Apr 2, 2019

Minor fixes to reduce diff to merfish

79cbfe7

see: spacetx/starfish#478

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upload example of transforming raw MERFISH -> spaceTx formatted data #478

Upload example of transforming raw MERFISH -> spaceTx formatted data #478

dganguli commented Aug 23, 2018

dganguli commented Aug 23, 2018

codecov-io commented Aug 23, 2018 •

edited

joshmoore commented Aug 24, 2018 •

edited

joshmoore commented Aug 24, 2018

ambrosejcarr commented Aug 24, 2018

ambrosejcarr left a comment

ambrosejcarr Aug 24, 2018

dganguli Aug 24, 2018

ambrosejcarr Aug 24, 2018

dganguli Aug 24, 2018

ambrosejcarr Aug 24, 2018

joshmoore commented Aug 24, 2018

ambrosejcarr commented Aug 24, 2018 •

edited

dganguli commented Aug 24, 2018

dganguli commented Aug 24, 2018

dganguli commented Aug 24, 2018

joshmoore commented Aug 24, 2018

joshmoore commented Aug 27, 2018

joshmoore commented Aug 27, 2018 •

edited

joshmoore commented Aug 30, 2018

Upload example of transforming raw MERFISH -> spaceTx formatted data #478

Upload example of transforming raw MERFISH -> spaceTx formatted data #478

Conversation

dganguli commented Aug 23, 2018

dganguli commented Aug 23, 2018

codecov-io commented Aug 23, 2018 • edited

Codecov Report

joshmoore commented Aug 24, 2018 • edited

joshmoore commented Aug 24, 2018

ambrosejcarr commented Aug 24, 2018

ambrosejcarr left a comment

Choose a reason for hiding this comment

ambrosejcarr Aug 24, 2018

Choose a reason for hiding this comment

dganguli Aug 24, 2018

Choose a reason for hiding this comment

ambrosejcarr Aug 24, 2018

Choose a reason for hiding this comment

dganguli Aug 24, 2018

Choose a reason for hiding this comment

ambrosejcarr Aug 24, 2018

Choose a reason for hiding this comment

joshmoore commented Aug 24, 2018

ambrosejcarr commented Aug 24, 2018 • edited

dganguli commented Aug 24, 2018

dganguli commented Aug 24, 2018

dganguli commented Aug 24, 2018

joshmoore commented Aug 24, 2018

joshmoore commented Aug 27, 2018

joshmoore commented Aug 27, 2018 • edited

joshmoore commented Aug 30, 2018

codecov-io commented Aug 23, 2018 •

edited

joshmoore commented Aug 24, 2018 •

edited

ambrosejcarr commented Aug 24, 2018 •

edited

joshmoore commented Aug 27, 2018 •

edited