Robustspotdetection #34

jennyfolkesson · 2020-04-08T04:56:16Z

I believe this should be stable now, but @bryantChhun should have a final say before this is merged. I wanted to start the code review at least.

One question - it looks like a lot of images are displayed now as you're running, and that makes the run pretty slow, plus you have all these plots popping up. What about changing this to write only?

…ntensity_fix

fixed background correction adjusted debug reporting to display coordinates on cropped image

Bryant od calc in icp

mattersoflight · 2020-04-08T13:41:40Z

I suggest @smguo merge his recent changes that improve the estimation of OD into this branch, and then we test it against March 25 and Apr 4 datasets.

@jennyfolkesson these plots are to catch edge cases as the pipeline is developed, but they are all invoked if debug is enabled. If there are any plots that are outside if debug: flag, we should move them there.

bryantChhun · 2020-04-08T21:10:45Z

There is a minor issue with the OD calculation -- both the cropped image and the background need to be on range 0-1, float64, not inverted

I will submit a PR to robostspotdetection

fixed and verified OD calculation

# Conflicts: # array_analyzer/workflows/interpolation_wf.py # run_array_analyzer.py

Spot intensity fix fit

bryantChhun · 2020-04-21T18:06:58Z

I'll work on merging this into master. There was a bit of module-movement, so I'll need to refactor a little, then retest across test data using interp/fit/well.

Will create a new branch for those changes and submit a PR here.

jennyfolkesson · 2020-04-21T18:11:09Z

I didn't rebase robustspotdetection wrt master recently, do you want me to do that?

…

On Tue, Apr 21, 2020 at 11:07 AM Bryant ***@***.***> wrote: I'll work on merging this into master. There was a bit of module-movement, so I'll need to refactor a little, then retest across test data using interp/fit/well. Will create a new branch for those changes and submit a PR here. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABT3MZOKP26SRJ4WLYNEKU3RNXOFFANCNFSM4MDUHJPA> .

…into robustspotdetection fixed bug in well_wf.py that prevented method='well_crop' from outputting debug images # Conflicts: # array_analyzer/extract/image_parser.py

clearer variable naming

Robustspotdetection for master

bryantChhun

I'm ok to merge this PR as is

bryantChhun · 2020-04-13T16:50:50Z

array_analyzer/workflows/registration_workflow.py

+                output_name,
+                from_source=True,
+            )
+


it looks like the plots for the background images have been removed and are not present in the "debug_images" module. Plotting background images were exactly what told me there was an issue with scaling and bitdepth. Will this become a test instead?

I agree with @bryantChhun. I think it would be better to keep some of the useful debug plots or move them to "debug_images" in case we need to look at them in the future. We can speed up the processing by running in regular mode instead of debug mode if the speed is the concern here.

smguo · 2020-04-22T23:09:20Z

https://github.com/czbiohub/serology-COVID19/blob/ed5783b449aae66758edf701958f2d779bc5aaed/array_analyzer/transform/point_registration.py#L31

https://github.com/czbiohub/serology-COVID19/blob/ed5783b449aae66758edf701958f2d779bc5aaed/array_analyzer/extract/image_parser.py#L310
Are icp and find_fiducials_markers just old version of particle_filter? Is so, can we remove them?

mattersoflight

I tested the branch with last two datasets (blinded flu plate and covid concentration test plate) for accuracy of spot detection and optical density estimation. They both look great. Ready to merge with debug plots for background added back to the pipeline.

smguo

@jennyfolkesson The PR looks great. I only have some minor suggestions. Feel free to merge when you think it's ready.

run_array_analyzer.py

array_analyzer/workflows/registration_workflow.py

smguo · 2020-04-11T01:41:18Z

array_analyzer/workflows/registration_workflow.py

+            dtype=object,
+        )
+
+        nbr_grid_rows, nbr_grid_cols = props_array.shape


Are nbr_grid_rows, nbr_grid_cols same as params['rows'], params['columns']?

Well, look at that! I didn't pay much attention to params I guess... :-)

smguo · 2020-04-11T01:49:27Z

run_array_analyzer.py

+        output_dir=args.output,
+        method=args.method,
+        debug=args.debug,
+    )


smguo · 2020-04-22T21:37:12Z

array_analyzer/workflows/registration_workflow.py

+                output_name,
+                from_source=True,
+            )
+


I agree with @bryantChhun. I think it would be better to keep some of the useful debug plots or move them to "debug_images" in case we need to look at them in the future. We can speed up the processing by running in regular mode instead of debug mode if the speed is the concern here.

smguo · 2020-04-22T22:35:34Z

array_analyzer/extract/img_processing.py

+
+    im_norm = cv.GaussianBlur(im, (blur_sigma, blur_sigma), 0)
+    # Black top hat filter, turns spots bright
+    im_norm = black_tophat(im_norm, size=(tophat_size, tophat_size))


Just curious about the purpose of adding black tophat. Is blob detector better at detecting bright v.s. dark spots?

No, the blob detector just needs a flag to know if the spots are bright or dark.
The black tophat is for detecting dark spots whereas a regular tophat would detect bright spots.

smguo · 2020-04-22T22:46:28Z

array_analyzer/transform/point_registration.py

+            trans_coords = trans_coords[0].astype(np.float32)
+            # Find nearest spots
+            ret, results, neighbors, dist = knn.findNearest(trans_coords, 1)
+            if remove_outlier:


It looks like the the outlier (fiducial that is not detected) will be removed starting at the first iteration. I'm wondering if it's possible that the wrong fiducial will be removed if there happens to be a spot close to the missing fiducial in the initial grid.

With enough stochasticity it doesn't seem to be a problem. All the other spots need to be in agreement with the grid so if there's one spot that's not quite on the grid that will still be the one excluded would be my guess.

I see. I guess the particles with the missing fiducial distance removed will have higher weights than the wrong solution, so these particles will eventually dominate through resampling. Very cool!

smguo · 2020-04-22T22:52:00Z

array_analyzer/transform/point_registration.py

+            ret, results, neighbors, dist = knn.findNearest(trans_coords, 1)
+            if remove_outlier:
+                # Remove worst fitted spot
+                remove_spot = np.where(np.squeeze(dist) != np.amax(dist))[0]


Is this line to get the Boolean indices of the distances to keep? if so wouldn't dist_ids = (dist != np.amax(dist)) do the same?

Much better, thanks (what was I thinking...).

smguo · 2020-04-22T22:54:01Z

array_analyzer/transform/point_registration.py

+            if remove_outlier:
+                # Remove worst fitted spot
+                remove_spot = np.where(np.squeeze(dist) != np.amax(dist))[0]
+                dist = dist[remove_spot]


Is remove_spot actually indices of the distances to keep? Maybe change the name?

I agree, that was a bad name. I removed that variable altogether.

bryantChhun

...

jennyfolkesson and others added 13 commits April 6, 2020 17:31

adding particle filter code

d14a6d9

include OD calculations (background estimation is not functional)

979241e

functioning background estimation during icp

3008612

adding particle filter workflow

40f1595

merged with bryants branch

040d1e9

added together workflows

98ebf1e

switched back to mean spot intensity

e31ee6e

Merge remote-tracking branch 'origin/robustspotdetection' into spot_i…

91ca926

…ntensity_fix

fixed function call from new location

a1b91ac

fixed background correction adjusted debug reporting to display coordinates on cropped image

revert back to un-normalized image.

90e20cb

Merge pull request #32 from czbiohub/bryant_OD_calc_in_ICP

4e6ad30

Bryant od calc in icp

switched back to mean spot intensity

26a185d

tested registration

d0c991e

jennyfolkesson requested review from mattersoflight, smguo and bryantChhun April 8, 2020 04:56

fixed and verified OD calculation

84dc141

mattersoflight mentioned this pull request Apr 8, 2020

Write integration tests #39

Closed

bryantChhun mentioned this pull request Apr 9, 2020

ICP method does not write OD files #28

Closed

bryantChhun and others added 9 commits April 9, 2020 10:23

fixed crop scaling to tolerate any bitdepth image

2210ecc

Output median spot intensity to be consistent with Scienion output

8c75fab

removed a few more hardcoded 8-bit scaling factors

58a70e1

switched default image scaling back

942acef

Merge pull request #38 from czbiohub/bryant_OD_calc_in_ICP

d99ed8c

fixed and verified OD calculation

adding particle filter code

4a2bee2

adding particle filter workflow

edd590d

include OD calculations (background estimation is not functional)

1415da7

functioning background estimation during icp

2f5174d

jennyfolkesson and others added 13 commits April 10, 2020 15:30

bryant comments

db98841

debugging registration fails

5499fc4

tuned blob detector

735c893

Merge branch 'spot_intensity_fix' into spot_intensity_fix_fit

8ca04c4

# Conflicts: # array_analyzer/workflows/interpolation_wf.py # run_array_analyzer.py

added the function to extract spot intensities within masks

8a34c4f

debugged intensity difference from Scienion

bc05e45

incorporated Jenny's and Bryant's comments

1ce7930

Merge pull request #45 from czbiohub/spot_intensity_fix_fit

4a401bf

Spot intensity fix fit

updated debug plots

b459393

added top hat filter to spot detection

a2305a0

added outlier removal to particle filter

b357bc0

starting cleanup

aa44bfe

code cleanup

5322d0e

bryantChhun and others added 4 commits April 21, 2020 15:01

Merge branch 'master' of https://github.com/czbiohub/serology-COVID19 …

3ef6f2b

…into robustspotdetection fixed bug in well_wf.py that prevented method='well_crop' from outputting debug images # Conflicts: # array_analyzer/extract/image_parser.py

Deleted commented code

fb20234

docstrings

3120443

clearer variable naming

Merge pull request #49 from czbiohub/robustspotdetection_for_master

ed5783b

Robustspotdetection for master

bryantChhun reviewed Apr 22, 2020

View reviewed changes

mattersoflight approved these changes Apr 22, 2020

View reviewed changes

smguo approved these changes Apr 22, 2020

View reviewed changes

pr feedback

79f9667

jennyfolkesson requested a review from bryantChhun April 23, 2020 00:45

jennyfolkesson merged commit a7700dc into master Apr 23, 2020

bryantChhun reviewed Apr 23, 2020

View reviewed changes

jennyfolkesson deleted the robustspotdetection branch April 23, 2020 17:27

This was referenced Apr 24, 2020

Tune get_spot_intensity #48

Closed

Streamline spot detection and OD estimation #23

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Robustspotdetection #34

Robustspotdetection #34

jennyfolkesson commented Apr 8, 2020

mattersoflight commented Apr 8, 2020 •

edited

bryantChhun commented Apr 8, 2020

bryantChhun commented Apr 21, 2020

jennyfolkesson commented Apr 21, 2020 via email

bryantChhun left a comment

bryantChhun Apr 13, 2020

smguo Apr 22, 2020

smguo commented Apr 22, 2020

mattersoflight left a comment

smguo left a comment

smguo Apr 11, 2020

jennyfolkesson Apr 22, 2020

smguo Apr 11, 2020

smguo Apr 22, 2020

smguo Apr 22, 2020

jennyfolkesson Apr 22, 2020

smguo Apr 22, 2020

jennyfolkesson Apr 23, 2020

smguo Apr 23, 2020 •

edited

smguo Apr 22, 2020 •

edited

jennyfolkesson Apr 23, 2020

smguo Apr 22, 2020 •

edited

jennyfolkesson Apr 23, 2020

bryantChhun left a comment

Robustspotdetection #34

Robustspotdetection #34

Conversation

jennyfolkesson commented Apr 8, 2020

mattersoflight commented Apr 8, 2020 • edited

bryantChhun commented Apr 8, 2020

bryantChhun commented Apr 21, 2020

jennyfolkesson commented Apr 21, 2020 via email

bryantChhun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smguo commented Apr 22, 2020

mattersoflight left a comment

Choose a reason for hiding this comment

smguo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smguo Apr 23, 2020 • edited

Choose a reason for hiding this comment

smguo Apr 22, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smguo Apr 22, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryantChhun left a comment

Choose a reason for hiding this comment

mattersoflight commented Apr 8, 2020 •

edited

smguo Apr 23, 2020 •

edited

smguo Apr 22, 2020 •

edited

smguo Apr 22, 2020 •

edited