Webgateway api

[will-moore]

This was previously opened as #4357 - restarting work after discussion in meeting yesterday see notes (internal only).

[will-moore]

First question is how much to duplicate the projection-backed API to provide enhanced performance over the omero_marshal-backed API. Need some performance assessment to see what the difference is:
In 45a61a1 I printed the duration of marshal_datasets (uses projection queries) and the equivalent function omero_marshal_datasets that uses omero_marshal.
Also added support for optionally counting images in Datasets.

    # Limit: 200 datasets
0.156903982162 Projection childCount
0.0777089595795 Projection
0.183522939682 omero-marshal childCount
0.111650943756 omero-marshal
    # repeat.... 200 datasets
0.154545068741 Projection childCount
0.0822401046753 Projection
0.176666021347 omero-marshal childCount
0.108825922012 omero-marshal

    # Limit: 1000 datasets
0.457422018051 Projection childCount
0.312170982361 Projection
0.593158006668 omero-marshal childCount
0.515108108521 omero-marshal
    # repeat.... 1000 datasets
0.405097961426 Projection childCount
0.30159497261 Projection
0.581645965576 omero-marshal childCount
0.455047130585 omero-marshal

    # No limits: 5147 datasets
1.85328197479 Projection childCount
1.4323759079 Projection
2.89162492752 omero-marshal childCount
2.60826516151 omero-marshal
     # repeat... 5147 datasets
1.67262816429 Projection childCount
1.56610679626 Projection
2.93430304527 omero-marshal childCount
2.65302586555 omero-marshal

[will-moore]

Closing till after 5.2.3. release...

[will-moore]

@jburel, @chris-allan - Probably time to restart discussions on the json api.
After our last discussion, I did some performance testing on omero_marshal vv projection-based queries and found that even up to 1000 datasets, omero_marshal is not much slower.
So we can probably use omero_marshal for most api calls.
We need to extend coverage of omero_marshal to all model objects - started looking at Image encoder at ome/omero-marshal#10

Also need to 'normalise' the experimenters and groups in the json we return, instead of including them fully nested in the objects themselfves E.g.

{
    datasets: [],
    experimenters: [],
    experimenterGroups: []
}

[will-moore]

Travis is failing with a lot of

  File "/home/travis/build/openmicroscopy/openmicroscopy/components/tools/OmeroWeb/omeroweb/webgateway/api_marshal.py", line 32, in <module>
    from omero_marshal import get_encoder
ImportError: No module named omero_marshal

@sbesson @chris-allan Presumably we have to specify this dependency in the Travis config somewhere?

[sbesson]

@will-moore: the Python dependencies are currently managed via the requirements file in Travis or in real-life deployments - see https://github.com/openmicroscopy/openmicroscopy/blob/v5.2.2/.travis.yml#L38. Adding omero-marshal to these dependencies should be straightforward. I also think the new Web deployment job handles dependency installation? /cc @aleksandra-tarkowska What requires more work will be to deal with coupled PRs (i.e. omero-marshal + openmicroscopy).

[will-moore]

@sbesson Seeing as omero-marshal is not pip installable, do we just add commands to download and update $PYTHONPATH?

[jburel]

Let's discuss that tomorrow at the web meeting since this will have to be done outside the standard ci infrastructure. This should follow the same approach as the one used recently for other major changes e.g. regions, ice 3.6 support etc.
This effort will have to be done in isolation so it does not affect the rest.

[chris-allan]

It is pip installable it's just not on PyPy, for the latest release version:

pip install https://github.com/openmicroscopy/omero-marshal/tarball/v0.3.0#egg=omero-marshal-0.3.0

For the HEAD of master:

pip install https://github.com/openmicroscopy/omero-marshal/tarball/master

Until we do put it on PyPy these specs can be put in requirements.txt or setup.py very safely. We use them extensively for PathViewer.

[jburel]

since ome/omero-marshal#10 is focusing on image, we should start with image (only) in the webgateway so it is easier to follow i.e. start a new branch with the relevant commits and keep the work done here a reference point.