Add example of raw->spacetx format conversion for ISS Breast Data #464

dganguli · 2018-08-22T19:50:17Z

This PR:

Introduces get_iss_breast_data.py, which converts raw data from s3://czi.starfish.data.public/browse/raw/20180820/iss_breast/ to spaceTx formatted data. This will be our first real multi-fov dataset.
re-names things. image -> tile, image_fetcher -> tile_fetcher

I tested this PR in a notebook locally. I downloaded the 1 FOV from he breast data formatter and ran the ISS breast notebook against it. It worked.

Remaining work for @ambrosejcarr :

Run this tool.
Upload formatted data to s3 browsable.
Copy codebook.json from 1 FOV dataset over to latter bucket.

Remaining work for @ttung :

Get cracking on experiment API so that...

Remaining work for Deep:

Use experiment API to analyze this entire dataset (blocked by @ttung )
Generate copy number plot for call 2 (high priority @ttung )
Start spec'ing out what multi-fov experiment processing will require (blocked by @ttung )

codecov-io · 2018-08-22T20:16:26Z

Codecov Report

❗ No coverage uploaded for pull request base (master@1b4469c). Click here to learn what that means.
The diff coverage is 81.25%.

@@            Coverage Diff            @@
##             master     #464   +/-   ##
=========================================
  Coverage          ?   85.17%           
=========================================
  Files             ?       73           
  Lines             ?     3009           
  Branches          ?        0           
=========================================
  Hits              ?     2563           
  Misses            ?      446           
  Partials          ?        0

Impacted Files	Coverage Δ
starfish/experiment/builder/__init__.py	`92% <71.42%> (ø)`
starfish/experiment/builder/imagedata.py	`64.7% <88.88%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1b4469c...4fcea96. Read the comment docs.

ttung · 2018-08-23T04:44:37Z

examples/get_iss_breast_data.py

+        return dict(zip(range(4), hyb_str))
+
+    def get_tile(self, fov: int, hyb: int, ch: int, z: int) -> FetchedTile:
+        filename = 'slideA_' + str(fov + 1) + '_' + \


string concatenation is a huge mess and very hard to read. consider switching to f-style python strings.

ttung · 2018-08-23T04:45:06Z

examples/get_iss_breast_data.py

+
+    @property
+    def ch_dict(self):
+        ch_str = ['Cy3 5', 'Cy3', 'Cy5', 'FITC']


not sure why this is better than just a literal dict.

ttung · 2018-08-23T04:45:33Z

examples/get_iss_breast_data.py

+
+    @property
+    def hyb_dict(self):
+        hyb_str = ['1st', '2nd', '3rd', '4th']


ummmm, dict(enumerate(hyb_str))?

ttung · 2018-08-23T04:46:00Z

examples/get_iss_breast_data.py

+        elif self.aux_type == 'dots':
+            filename = 'slideA_' + str(fov + 1) + '_DO_' + 'Cy3.TIF'
+        else:
+            msg = 'invalid aux type: {}'.format(self.aux_type)


ahhh, string concatenation!

ttung · 2018-08-23T04:46:37Z

examples/get_iss_breast_data.py

+
+
+def format_data(input_dir, output_dir):
+    if not input_dir.endswith("/"):


these should not be necessary.

lol i copied this from your code in get_iss.py. i can nuke it.

ttung · 2018-08-23T04:47:19Z

examples/get_iss_breast_data.py

+    if not output_dir.endswith("/"):
+        output_dir += "/"
+
+    def add_codebook(experiment_json_doc):


hmmmm, i think we need a better way to build codebooks programmatically....

ttung · 2018-08-23T04:48:24Z

examples/get_iss_breast_data.py

+    parser.add_argument("output_dir", type=FsExistsType())
+
+    args = parser.parse_args()
+    s3_bucket = "s3://czi.starfish.data.public/browse/raw/20180820/iss_breast/"


i don't get this. who's going to do this?

ambrose is going to do this. josh is going to read this code. this is also an example to help collaborators understand how to convert their data.

I think this message is confusing. i would just have the --help output tell people where to get the data.

Rationale: the only time this message prints is if the command is running, and to do that, you kind of need the data already.

ttung · 2018-08-23T07:56:00Z

Is it possible to generate a full experiment manifest from this?

ambrosejcarr · 2018-08-23T13:02:21Z

@ttung I think this is a WIP that I'm supposed to take over. I was looking through the code yesterday and your question was my main question: "how does one pass a directory with multiple fields of view and expect them to be parsed separately? Do I write os.path.walk() into the TileFetcher?

A toy example of that would be very helpful, if you know how it's supposed to function. If that functionality is not supported, that would also be great to know, as I will try to add it.

When I'm finished using this tool to upload the s3 data, I will add a README to it.

dganguli · 2018-08-23T18:13:04Z

@ttung what do you mean is it possible to generate a full experiment manifest from this. That is the entire point. Unless you have a different definition of experiment manifest than I do? The problem, is that we don't have a way of loading this full experiment manifest. That's why we need the multi-fov API ASAP!

@ambrosejcarr parsing images that live in separate directories is easy; however, that's super unimportant to worry about now. regardless, the way you would handle it, is in TileFetcher.get_tile(...). You would implement the logic needed os.path.join self.input_dir to the right thing based on fov_num, hyb, ch, etc. a more interesting question, is what if you'd like build_experiment_json to write hybridization_fov_xxx.jsons to different directories.

…cks'

dganguli · 2018-08-23T18:25:50Z

@ttung i addressed all ur comments in the PR. thx!

ambrosejcarr · 2018-08-23T19:04:58Z

@dganguli Ok, so if I wanted to use this tool as is, I should put all the existing images that need to be formatted into the same directory, and it will write a series of hybridization.json files into a single output directory?

I was for some reason stuck on the prior that each FOV would live in its own directory and couldn't reconcile how this would work in that case. I think I'm straightened out.

dganguli · 2018-08-23T20:43:50Z

@ttung can u approve and merge? i'd like to get the corresponding merfish work done today as well.

ttung · 2018-08-23T20:45:24Z

examples/get_iss_breast_data.py

+        crp = img[40:1084, 20:1410]
+        return crp
+
+    def tile_data_handle(self) -> IO:


i think this is supposed to be a @property. it worked before due to a fluke, but the new slicedimage api will require it.

you can make the change now or i can make it when i land the slicedimage change.

ttung · 2018-08-23T20:46:31Z

examples/get_iss_breast_data.py

+    parser.add_argument("output_dir", type=FsExistsType())
+
+    args = parser.parse_args()
+    s3_bucket = "s3://czi.starfish.data.public/browse/raw/20180820/iss_breast/"


I think this message is confusing. i would just have the --help output tell people where to get the data.

Rationale: the only time this message prints is if the command is running, and to do that, you kind of need the data already.

dganguli added 8 commits August 20, 2018 13:58

ISS breast data initial commit

a91add6

get_iss_breast_data works

efe7b3b

whitespace ftw

7baa884

add codebook post-processing hook

3ce1fbe

Merge branch 'master' into dg-iss-breast-public

bd704d6

re-factored Image->Tile and organized code

c1a105a

remove references to hybridizations

c2ef75b

use tile instead of image

2767d55

dganguli requested review from ttung, ambrosejcarr and shanaxel42 August 22, 2018 19:50

fix import order

fb49d1c

ttung requested changes Aug 23, 2018

View reviewed changes

remove str concatenation, better dict making, remove dir.endswith che…

00d6385

…cks'

ttung approved these changes Aug 23, 2018

View reviewed changes

formatted help string, added missing property annotation

4fcea96

dganguli merged commit 3e29d5b into master Aug 23, 2018

This was referenced Sep 3, 2018

Upload ISS breast data to s3 browse directory #416

Closed

Raw Data Available #414

Closed

Use FOV builder to generate synthetic datasets for ingestion #418

Closed

Synthetic dataset examples in spaceTx format #355

Closed

ttung deleted the dg-iss-breast-public branch September 27, 2018 18:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add example of raw->spacetx format conversion for ISS Breast Data #464

Add example of raw->spacetx format conversion for ISS Breast Data #464

dganguli commented Aug 22, 2018

codecov-io commented Aug 22, 2018 •

edited

ttung Aug 23, 2018

ttung Aug 23, 2018

dganguli Aug 23, 2018

ttung Aug 23, 2018

ttung Aug 23, 2018

ttung Aug 23, 2018

dganguli Aug 23, 2018

ttung Aug 23, 2018

ttung Aug 23, 2018

dganguli Aug 23, 2018

ttung Aug 23, 2018

dganguli Aug 23, 2018

ttung commented Aug 23, 2018

ambrosejcarr commented Aug 23, 2018 •

edited

dganguli commented Aug 23, 2018

dganguli commented Aug 23, 2018

ambrosejcarr commented Aug 23, 2018

dganguli commented Aug 23, 2018

ttung Aug 23, 2018

dganguli Aug 23, 2018

ttung Aug 23, 2018



		def format_data(input_dir, output_dir):
		if not input_dir.endswith("/"):

Add example of raw->spacetx format conversion for ISS Breast Data #464

Add example of raw->spacetx format conversion for ISS Breast Data #464

Conversation

dganguli commented Aug 22, 2018

codecov-io commented Aug 22, 2018 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ttung commented Aug 23, 2018

ambrosejcarr commented Aug 23, 2018 • edited

dganguli commented Aug 23, 2018

dganguli commented Aug 23, 2018

ambrosejcarr commented Aug 23, 2018

dganguli commented Aug 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Aug 22, 2018 •

edited

ambrosejcarr commented Aug 23, 2018 •

edited