Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for collapsing unique light paths #3261

Merged
merged 19 commits into from
Jan 29, 2015
Merged

Support for collapsing unique light paths #3261

merged 19 commits into from
Jan 29, 2015

Conversation

chris-allan
Copy link
Member

Logic and unit test cases for collapsing unique light paths. Test cases were also expanded to cover all other existing collapse logic. Some existing, but old and non-functional, test cases have been ported to TestNG and the entire suite of ome.formats.* unit tests added to the server unit test suite.

Due to the amount of white space changes that were required, adding ?w=1 to the end of the GitHub "Files Changed" page may be beneficial during review.

Manual testing has been performed:

  • Imported one or more Plates with LightPath metadata populated (Flex in particular is useful here)
  • Confirmed that LogicalChannel and LightPath reduction has taken place
  • Confirmed that the Plates can be deleted
  • Confirmed that the Plates can be be moved between groups
  • Testing also took place in the presence of a backported version of fix #11779: graph operations use "compiler" rather than "rollback" strategy #3228

LightPath and LogicalChannel reduction can be confirmed by examining the import log for a sufficiently small number. For example:

2014-11-07 11:15:50,320 INFO  [          ome.formats.OMEROMetadataStore] (      main) Unique objective settings: 1
2014-11-07 11:15:50,324 INFO  [          ome.formats.OMEROMetadataStore] (      main) Unique light settings: 1
2014-11-07 11:15:50,324 INFO  [          ome.formats.OMEROMetadataStore] (      main) Unique detector settings: 1
2014-11-07 11:15:50,324 INFO  [          ome.formats.OMEROMetadataStore] (      main) Unique light paths: 2
2014-11-07 11:15:50,324 INFO  [          ome.formats.OMEROMetadataStore] (      main) Unique logical channels: 2

Example datasets contained 12 fields, 384 wells and 2 channels resulting in 9216 unique logical channels per Plate. Completely broken on import. With a 2GB heap, takes ~4.5 hours just to get to pixel data processing:

...
2014-11-05 06:12:13,751 INFO  [          ome.formats.OMEROMetadataStore] (.Server-10) Unique objective settings: 1
2014-11-05 06:12:13,751 INFO  [          ome.formats.OMEROMetadataStore] (.Server-10) Unique light settings: 2
2014-11-05 06:12:13,751 INFO  [          ome.formats.OMEROMetadataStore] (.Server-10) Unique detector settings: 0
2014-11-05 06:12:13,751 INFO  [          ome.formats.OMEROMetadataStore] (.Server-10) Unique logical channels: 9216
...

Reduced version with just one Well's worth of .flex files:

...
2014-11-07 05:43:23,329 INFO  [          ome.formats.OMEROMetadataStore] (erver-2027) Unique objective settings: 1
2014-11-07 05:43:23,329 INFO  [          ome.formats.OMEROMetadataStore] (erver-2027) Unique light settings: 2
2014-11-07 05:43:23,329 INFO  [          ome.formats.OMEROMetadataStore] (erver-2027) Unique detector settings: 0
2014-11-07 05:43:23,329 INFO  [          ome.formats.OMEROMetadataStore] (erver-2027) Unique logical channels: 24
...

After reduction with just one Well's worth of .flex files:

...
2014-11-07 05:52:44,408 INFO  [          ome.formats.OMEROMetadataStore] (l.Server-3) Unique objective settings: 1
2014-11-07 05:52:44,408 INFO  [          ome.formats.OMEROMetadataStore] (l.Server-3) Unique light settings: 2
2014-11-07 05:52:44,408 INFO  [          ome.formats.OMEROMetadataStore] (l.Server-3) Unique detector settings: 0
2014-11-07 05:52:44,408 INFO  [          ome.formats.OMEROMetadataStore] (l.Server-3) Unique light paths: 2
2014-11-07 05:52:44,408 INFO  [          ome.formats.OMEROMetadataStore] (l.Server-3) Unique logical channels: 2
...

Total time to import with a 2GB heap is now under <60 minutes.

 _____________________ 
< I love light paths! >
 --------------------- 
        \   ^__^
         \  (**)\_______
            (__)\       )\/\
             U  ||----w |
                ||     ||

emilroz and others added 17 commits November 6, 2014 16:36
@emilroz
Adjust test to asses correct linkage to LightPath and Instrument
fe9d9e2
@chris-allan
Port MultiplePlateTest from JUnit to TestNG
5c79f1c
@chris-allan
Fix GenericReferenceTest logic errors
275d302
@chris-allan
Tabs to spaces and remove trailing white space
90ad267
@chris-allan
Port GenericReferenceTest from JUnit to TestNG
dfe95ea
@chris-allan
Fix DetectorSettingsDetectorTest logic errors
d9751f1
@chris-allan
Tabs to spaces and remove trailing white space
5625fac
@chris-allan
Port DetectorSettingsDetectorTest from JUnit to TestNG
6e2d47d
@chris-allan
Add ome.formats.* to the server unit test suite
818e348
@emilroz
Make checkAndCollapseGraph() public for testing
2b01c19
@emilroz
Test case for checkAndCollapseGraph()
1498f33
@emilroz
Reverse assertions to be inline with testNG
5d08849
@emilroz
Expand test case to cover ObjectiveSettings
163fb83
@emilroz
Expand test case to cover LightSettings
354f2f1
@emilroz
Expand test case to cover DetectorSettings
5c6cbce
@emilroz
Add getUniqueLightPath() method to optimise performance
e367839 
This new method will reduce the number of LightPath objects
and by extension LogicalChannel objects when importing large
plates with these metadata populated.
@chris-allan
Add the collapsing of LightPaths to the method Javadoc
48837fb
@jburel jburel mentioned this pull request Dec 9, 2014
Merged
@jburel
Copy link
Member

jburel commented Dec 9, 2014

--rebased-to #3264

@jburel jburel added the dev_5_0 label Dec 9, 2014
@joshmoore
Copy link
Member

I assume the same changes are needed here as #3264 (comment) ?

@jburel
Copy link
Member

jburel commented Jan 23, 2015

@joshmoore: It dropped of my radar. I will add it to my todo list.

@jburel
Copy link
Member

jburel commented Jan 23, 2015

PR now opened.

@chris-allan
Copy link
Member Author

Now synchronised with work from @jburel.

@qidane
Copy link
Contributor

qidane commented Jan 29, 2015

Code looks like it doing the right thing with the LightPaths, little less sure with the Channel/LogicalChannel bits as the schema only had Channel, but if the tests are green I would assume you are good to go.

@joshmoore
Copy link
Member

Thanks, all.

joshmoore added a commit that referenced this pull request Jan 29, 2015
@joshmoore
Merge pull request #3261 from chris-allan/unique-light-paths
f67edac 
Support for collapsing unique light paths
@joshmoore joshmoore merged commit f67edac into ome:dev_5_0 Jan 29, 2015
@sbesson sbesson added this to the 5.0.7 milestone Feb 4, 2015
@sbesson sbesson mentioned this pull request Jun 23, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
dev_5_0
Projects
None yet
Milestone
5.0.7
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@chris-allan @jburel @joshmoore @qidane @sbesson @emilroz