Idr prototype

[will-moore]

Update to Gallery to allow searching & filtering of top-level Projects & Screens and a nicer
layout showing 'categories' of interest, selected by querying Map-Annotations.
See ome/design#100

Can be tested at http://web-dev-merge.openmicroscopy.org/gallery/

For screenshots etc see the design issue link above.

TODO:

• Fix Key-Value auto-complete for /cells and /tissues (broken since I started using config of 'categories' to define studies in "tissue").
  • The cells super_category is defined as "other", meaning it includes everything that is not in tissue category. Q: Is this strategy the way that we want to define 'cells' vv 'tissue'??
• Check that all URLs generated in JS are based on 'base_url'
• Move hard-coded FILTER_KEYS to config
• Same for mapr config - list subset of mapr keys to appear in the UI
• Config for title "Welcome to IDR"
• Customise UI: Header Icon, other html? links etc? See Customise CSS, header and footer e.g. for IDR #26
• Browser support? Not IE 10 unless we transpile JavaScript. Need build step, e.g. webpack See Build step #27
• Fix listing of multiple Screens/Projects for a single idr00xx Study, and do this in a non IDR-specific way? See Group Screens & Projects? #28
• Don't try render_image/ID/ for Big Images. Maybe load sizeX and sizeY when we get study/image/ JSON?
• Don't use /idr/ directory for static files & templates
• Handle duplicate keys e.g. FRAP in https://idr.openmicroscopy.org/webclient/?show=project-753 for auto-complete (and filtering).
• Show "Most Recent" category. Update category labels as suggested by Frances"
• Auto-complete show on input focus()
• Search by idr00xx number
• Error handling on all fetch() calls.
• Allow 'Enter' to goto /search without having to pick auto-complete option
• Always show links to "Cells" and "Tissue" in the header
• If filter gives 0 results, say "Try your search on ***-IDR?"
• Case sensitive option for IDR?
• Show auto-complete options for MAPR without user needing to type, e.g. for Cell Lines (as in idr mapr)
• Front page of /cells = "Welcome to Cell-IDR" and of /tissue = "Welcome to Tissue-IDR".
• Show spinner when during auto-complete.
• Study Title config to use Map-Annotation for Study/Publication Title

Need Help:

• Config web-dev-merge to allow to connect to IDR "public", "public"
• What "categories" do we want? Does current query support what we need? Needs curation.
• Use 'live' mapr config loaded from IDR. First requires Add /api/config/ to return mapr_settings.CONFIG as JSON omero-mapr#46 to be merged and deployed somewhere. If not IDR then need a temp config to point elsewhere.
• Also need to deploy, test and merge Support ?show=screen-123 for mapr jstree omero-mapr#45 so we can link to Study within mapr.
• Test with various configs. At least: current working configs vv None, but more is better.
• Move configs from default values in gallery_settings.py to deployment CI (and write docs)

[will-moore]

Summary of Study Types: count (NB: duplicate studies e.g. idr0038 are counted multiple times).

3D-tracking of tagged chromatin loci: 1
4D timelapse of in vivo cells: 1
5D light sheet and confocal microscopy imaging of mouse zygotes: 1
Confocal laser scanning microscopy of cytoskeletal systems: 1
Quantitative protein localization using FCS-calibrated 3D time-lapse imaging: 1
high content analysis of cells: 1
high content screen: 37
high resolution electron microscopy: 1
histology: 2
imaging mass cytometry: 1
in situ hybridization: 1
light sheet fluorescence microscopy of TARDIS registered embryos: 1
light sheet fluorescence microscopy of organoids: 4
light sheet microscopy of zebrafish tailbud: 1
microscopy assay: 4
protein localization: 1
protein localization using 3D-SIM: 1
protein localization using dSTORM: 1
timelapse microscopy: 1

[will-moore]

Some potential categories: IDs are idr numbers.

"time-lapse": [41, 27, 26, 40],
"lightsheet": [51, 45,  44, 38],
"protein localization": [43, 41, 23, 21, 19],
"histology": [42, 18, 43],
"yeast": [3, 4, 7, 11, 27, 40, 47],
"human nuclear stain": [16, 9, 6, 17, 30, 36, 10]

[chris-allan]

Chrome on Windows 10. Just over 1080p browser window size. Default font scaling.

Few pieces of feedback for the landing page:

1. Getting a series of JavaScript errors in the console on initial page load:

model.js:312 Uncaught (in promise) TypeError: Cannot read property 'json' of undefined
    at model.js:312
(anonymous) @ model.js:312
Promise.then (async)
loadImage @ model.js:313
(anonymous) @ categories.js:223
loadStudyThumbnails @ categories.js:217
render @ categories.js:173
(anonymous) @ categories.js:264
(anonymous) @ model.js:191
Promise.then (async)
loadStudiesMapAnnotations @ model.js:162
(anonymous) @ model.js:144
Promise.then (async)
loadStudies @ model.js:124
(anonymous) @ categories.js:259
model.js:305 No Dataset in Project! {meta: {…}, data: Array(0)}
model.js:312 Uncaught (in promise) TypeError: Cannot read property 'json' of undefined
    at model.js:312

2. I've got a pretty fast connection, am ~20ms from idr.openmicroscopy.org and the page takes over 10 seconds to fully load. Might be an idea to batch thumbnail calls via get_thumbnails. The page makes 50+ individual requests for thumbnails at the moment and this will only get worse over time.
3. Not confining the author lists and not having some line breaks in the descriptions creates some pretty wacky visualization conditions:
   
   
4. The scaling of the thumbnail strips is creating some weird excessive whitespace in the horizontal direction. You can scroll well beyond any thumbnails:
   
5. Yeast Studies scrollbar/header overlap:
   

And for search:

1. If idr0018 is matched by a search, such as "microscopy", it is missing some of its metadata:
   
2. Feels kind of weird to have a search results grid that is only two studies wide
   

[will-moore]

@chris-allan Thanks for the feedback. I've fixed the console errors and search results is now 3 columns wide. The other css fixes I have put on hold since the whole page is likely to get a facelift and I was already spending too much time trying to design and fix those.

[chris-allan]

👍

Already looking pretty solid and adept at what it's trying to do. Is your github.io up to date with the latest code? Just noticed there are still console errors there.

[will-moore]

@sbesson @francesw As noted by @chris-allan, idr0018 is missing Publication Title both in the Project Description and in the linked Map Annotation. I'm getting this from the Description to show in the UI like this:

  let titleRe = /Publication Title\n(.+)[\n]?/
  let match = titleRe.exec(desc);
  let title = match ? match[1] : '';

Do we want to add some title to this study to "fix" the UI?

[will-moore]

@chris-allan I hadn't pushed the console error fixes to github.io. But did it just now.

[chris-allan]

Great. Definitely no error messages now.

[will-moore]

To test last commit (batch loading of thumbnails), first login to "idr" as "public/pubilc" at http://web-dev-merge.openmicroscopy.org/webclient/login/ then go to:
http://web-dev-merge.openmicroscopy.org/gallery/
You should see thumbnails loading much faster than before. Instead of making 2 JSON calls to get the image ID then the loading of the thumb itself (3 calls for each image) we now load the Image IDs and thumbnails for 10 studies/thumbnails at once. (30 x fewer calls).
URLs like http://web-dev-merge.openmicroscopy.org/gallery/api/thumbnails/?project=51&project=101&project=402&screen=1603&screen=1602&screen=1601&screen=1551&screen=1501&screen=201&screen=202 are used to get thumbnails AND image IDs (needed for the viewer links) in one shot.

[chris-allan]

Looking now at the aggregate changes in urls.py and views.py I can see us slowly going down the road of adding some pretty fundamental API to an OMERO.web plugin. Representative thumbnails for containers is something that has been in the background for a long time. Glencoe did something similar for the JCB DataViewer in what we called the “featured” image via a specifically namespaced BooleanAnnotation and a fallback to the first image if missing.

I’d like to propose that we iterate on this API using https://github.com/glencoesoftware/omero-ms-thumbnail. Doing so is very much in the spirit of trying to avoid the deployment idiosyncrasies mentioned above, reinforces a separation of concerns, and also helps avoid the need for a full OMERO.web deployment being required to contribute to this plugin.

/cc @kkoz, @emilroz

[sbesson]

Re-tested the latest version of this code on the deployed CI this morning. Most of the bug fixes discussed above have been addressed. I will capture some errors as issues.

As noted in #25 (comment), the size of this PR is starting to be big enough that reviewing the code and the comments becomes quickly impractical. In line with various conversations, I think we all agree that we have now reached the end of the first phase which was dedicated to building this UI.

Proposing to tag this initial effort as 3.2.0a1 and start working on the next steps in smaller steps, releasing the app in milestones as needed:

1. make the changes to deploy this application into a testing resource in preparation of production deployment. This might require a couple of additional cleanup PRs to remove hard-coded configuration, enable thumbnail viewing without the need to log in etc.
2. iterate over the UI issues (CSS & other bugs) as well as the metadata that should be added to the studies to make the UI functional
3. once deployed, schedule some performance testing of the new UI
4. investigate the migration of the required thumbnail API calls to a micro-service as suggested in Idr prototype #25 (comment). This might also be highlighted by the load testing.