Support OMERO.table StringColumn

[will-moore]

From https://www.openmicroscopy.org/qa2/qa/feedback/21658/ and https://www.openmicroscopy.org/qa2/qa/feedback/21733/

If a user has an OMERO.table with StringColumns they get listed in the "Add Table data..." menu but when chosen, the loading of data fails because histogram can't be created (error below).

Seems that when loading filters (includes ALL columns from a table) we only try to get numpy min/max/histogram for DoubleColumn and LongColumn (https://github.com/ome/omero-parade/blob/master/omero_parade/table_filters/omero_filters.py#L74)

However, for loading Table data, we don't test column type, so numpy.histogram, numpy.amin, numpy.amax are tried for e.g. StringColumn which fails.
If I comment-out these lines in views.py get_data() then it returns just the values dict which is displayed in the OMERO.parade table OK.
Not sure why we need to calculate histogram, min and max since they're not used by the UI in any way?

For StringColumns that actually contain Long/Float values, I wrote a script to convert columns to LongColumn or DoubleColumn in a new Table:
https://gitlab.com/openmicroscopy/incubator/python-scripts/blob/master/omero_table_stringcol_to_doublecol.py
but this doesn't help if the values are actually Strings.

File "/home/omero_user/omeroenv2/local/lib/python2.7/site-packages/omero_parade/views.py", line 177, in get_data
histogram, bin_edges = numpy.histogram(values, bins=bins)
File "/home/omero_user/omeroenv2/local/lib/python2.7/site-packages/numpy/lib/histograms.py", line 676, in histogram
bin_edges, uniform_bins = _get_bin_edges(a, bins, range, weights)
File "/home/omero_user/omeroenv2/local/lib/python2.7/site-packages/numpy/lib/histograms.py", line 299, in _get_bin_edges
first_edge, last_edge = _get_outer_edges(a, range)
File "/home/omero_user/omeroenv2/local/lib/python2.7/site-packages/numpy/lib/histograms.py", line 250, in _get_outer_edges
first_edge, last_edge = a.min(), a.max()
File "/home/omero_user/omeroenv2/local/lib/python2.7/site-packages/numpy/core/_methods.py", line 32, in _amin
return umr_minimum(a, axis, None, out, keepdims, initial)
TypeError: cannot perform reduce with flexible type
path:/parade/data/VGFibGVfU3BvdCBDZWxsIENvdW50/,

[dsudar]

Thanks @will-moore . I will try the script but already noticed that for my situation I'll have to make a few mods so it can handle tables that are attached at the Screen (not Plate) level.

The script will indeed serve as a patch but indeed best if the parade functionality can handle StringColumns "natively" since that is what "omero metadata populate" generates, right?
Side note: I just noticed on PyPI that there is a omero-metadata CLI plugin version 0.2.2. Should I start using that instead of the older built-in version that is enabled with OMERO_DEV_PLUGINS? Does that new version perchance create DoubleColumn, maybe? If not, maybe it should?

[will-moore]

Hi Damir,
To update for a Screen, just replace Plate with Screen on these lines

link = omero.model.PlateAnnotationLinkI()
link.setParent(omero.model.PlateI(plate_id, False))

Yes, please use the latest omero-metadata plugin.
Looking at the source code https://github.com/ome/omero-metadata/blob/master/src/populate_metadata.py#L76 it seems to support creation of different column types with --columns but haven't worked out how to run it yet.

[will-moore]

Hi Damir, with @joshmoore and @sbesson's help I managed to use the --columns argument. It's not supported via the CLI, so you have to run the populate_metadata.py directly...

With a sample csv that looks like:

Well, Drug, Concentration, Cell Count, Percent Mitotic
A1, DMSO, 10.1, 10, 25.4
A2, DMSO, 0.1, 1000, 2.54
A3, DMSO, 5.5, 550, 4
B1, Monastrol, 12.3, 50, 44.43
B2, Drug3, 1.1, 1, 15.4
B3, Water, 20.1, 44, 12.3
C1, Acid, 15.1, 20, 25.4
C2, DrugX, 4.44, 10, 35.4
C3, XYZ, 0.001, 303, 45.4

Columns are Well, String, Double, Long, Double well,s,d,l,d
I ran:

$ pip install omero-metadata

$ python /usr/local/lib/python2.7/site-packages/populate_metadata.py -s localhost -p 4064 -u username -w password --columns well,s,d,l,d Plate:7967 data_for_plate_7967.csv

This allowed me to load the Concentration, Cell Count, Percent Mitotic columns in OMERO.parade "Add table data..." (but not the Drug column because of the numpy histogram() fail as mentioned above).
Hope that helps.

[dsudar]

Hi all, thanks for looking into this and for having the solution already under development. This indeed looks to go into a very usable direction. I'll let you know how I get on with it. A couple quick comments: 1) for annotating an entire screen's worth, is there another argument for the --columns option called "plate" that works just like the "well" argument? 2) may I suggest a slightly different approach where the column "keys" are encoded in the .csv file itself? E.g. as a 2nd row? Cheers, Damir

…

On 8/22/2018 4:18, Will Moore wrote: Hi Damir, with @joshmoore <https://github.com/joshmoore> and @sbesson <https://github.com/sbesson>'s help I managed to use the |--columns| argument. It's not supported via the CLI, so you have to run the |populate_metadata.py| directly... With a sample csv that looks like: |Well, Drug, Concentration, Cell Count, Percent Mitotic A1, DMSO, 10.1, 10, 25.4 A2, DMSO, 0.1, 1000, 2.54 A3, DMSO, 5.5, 550, 4 B1, Monastrol, 12.3, 50, 44.43 B2, Drug3, 1.1, 1, 15.4 B3, Water, 20.1, 44, 12.3 C1, Acid, 15.1, 20, 25.4 C2, DrugX, 4.44, 10, 35.4 C3, XYZ, 0.001, 303, 45.4 | Columns are Well, String, Double, Long, Double |well,s,d,l,d| I ran: |$ pip install omero-metadata $ python /usr/local/lib/python2.7/site-packages/populate_metadata.py -s localhost -p 4064 -u username -w password --columns well,s,d,l,d Plate:7967 data_for_plate_7967.csv | This allowed me to load the Concentration, Cell Count, Percent Mitotic columns in OMERO.parade "Add table data..." (but not the Drug column because of the numpy histogram() fail as mentioned above). Hope that helps. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#47 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJvaur90sy_aY5sLbyUKYux7M9lho-geks5uTT4igaJpZM4V9x8U>.

-- Damir Sudar - Affiliate Scientist Lawrence Berkeley Natl Laboratory / MBIB One Cyclotron Road, MS 977, Berkeley, CA 94720, USA E: DSudar@lbl.gov W: http://biosciences.lbl.gov/profiles/damir-sudar-2/ Visiting Scientist, Oregon Health & Science University

[will-moore]

Hi Damir, yes there is a "plate" argument - See https://github.com/ome/omero-metadata/blob/master/src/populate_metadata.py#L76
or

$ python /usr/local/lib/python2.7/site-packages/populate_metadata.py --help

[dsudar]

Hi Will, I finally found some time to modify my workflow to use the --column functionality in populate_metadata.py and indeed, that now works fine to create table columns that Parade likes. Thanks to you, Sebastien, and Josh. To learn how to use Parade, I watched the Glencoe webinar again and was wondering about a few of the functions that Emil was demo'ing that do not appear to be available or behave differently in the open source Parade version 0.1.2: 1) Emil showed a SETTINGS "button" at the top that allowed switching between heatmaps, thumbnails, and empty boxes. 2) Interestingly, Emil was able to add using Add table data... a column (Table_Drug) that appears to have string values. 3) He also showed selection of multiple plates by selecting a Screen. I see there's a PR for this from Chris. Will that make it in soon? Is there an idea when/if these capabilities will appear in Parade open-source or are some of them limited to Glencoe/OMERO-Plus? I'll now ask some of my colleagues to try using Parade so hopefully we'll have more feedback soon. Cheers, - Damir

…

On 8/23/2018 1:29, Will Moore wrote: Hi Damir, yes there is a "plate" argument - See https://github.com/ome/omero-metadata/blob/master/src/populate_metadata.py#L76 or |$ python /usr/local/lib/python2.7/site-packages/populate_metadata.py --help | — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#47 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJvausEQoot968qgyD7tbTKab-Usik12ks5uTmfkgaJpZM4V9x8U>.

-- Damir Sudar - Affiliate Scientist Visiting Scientist, Oregon Health & Science University

[chris-allan]

@dsudar: All of the functionality @emilroz demonstrated in our webinar is included in #39. If you were to merge that PR with master you would have the exact code that was used. There are several limitations and it is the focus of ongoing work.

/cc @jburel

[dsudar]

Thanks @chris-allan . I'll first try to do that on a test instance. Meanwhile, for the StringColumn issue, I have been trying the work-around where I am coping those metadata columns in my Table on which I want to filter to Key-Value pairs (using "omero metadata populate"). That works for what I need except that the match it uses in Add Filter... is a match-anything so when trying to filter on AAA, it will also pick up AAABBB. Any way to request an exact match?
Since this discussion is possibly of interest to others, should it be continued on the Forum?
Thanks, Damir

[will-moore]

This issue reported again at https://www.openmicroscopy.org/qa2/qa/feedback/27391/ and https://www.openmicroscopy.org/qa2/qa/feedback/27396/