Add client support for an OMERO auto-connection file on public websites

[manics]

Background

For a long time the IDR has recommended the use of a custom connection method instead of the standard OMERO.py client or gateway methods. This was originally done to allow load-balancing since the OMERO binary protocol does not allow session pinning on a front end proxy:

• https://github.com/IDR/deployment/blob/IDR-0.8.2/ansible/idr-haproxy.yml#L3-L10

We can't do proper session pinning with the OMERO binary protocol because
there's no way to insert or extract an identifier, so instead we assign
separate front-end ports for each back-end server and rely on the client
to choose a random port for each connection.

To support autoconfiguration of clients we also create a json config
file with a list of connections parameters that clients should randomly
choose from.

This custom connection method fetches a json configuration file from the IDR website if a base hostname is provided. For example, conn = connection('idr.openmicroscopy.org', ...) will attempt to download a file connection/omero-client.json from the host using https, in this case https://idr.openmicroscopy.org/connection/omero-client.json

This connection file effectively allows custom configuration to be pushed to the clients- normally this would require code changes in the client to override the defaults, or a knowledgable user who knows and remembers to pass additional configuration to their client.

Most recently this is being used to provide a default username and password to connect to the IDR:

• Include public user/pass in omero-client.json IDR/deployment#246

Future benefits to us or others include:

• Provide an easy to remember hostname e.g. example.org for users to configure their clients, whilst deploying OMERO.server on a different address- there is no requirement for connection/omero-client.json to reside on the same host or domain as OMERO.server
• Easily switch the public credentials if necessary for any reason without actively notifying clients
• Provide a standard way for other public resources to provide public OMERO API access
• Ability to force all clients to connect using an alternative transport such as wss (websockets)

Proposal

• Add support for https://HOSTNAME/connection/omero-client.json to OMERO.py clients and BlitzGateway.
• Optionally add support to clients in other languages.

Alternative: .well-known

.well-known is standard for defining well-known locations on a web server providing metadata:

• https://en.wikipedia.org/wiki/List_of_/.well-known/_services_offered_by_webservers

We could consider using something like https://HOSTNAME/.well-known/omero/client.json instead.

[joshmoore]

• I didn't know of .well-known but following some form of convention makes sense.
• I'm certainly for also adding to the Java Gateway. Unsure if it's worth adding to C++ at the moment. cc: @dominikl
• One thing it's perhaps pre-designing for is: what happens when/if login moves to being HTTP based rather than Ice based?

[manics]

If we move to http login I think this would still work, since client.json is a set of overrides, so the following are all possible

• omero-client.json refers to the same host and port
• omero-client.json is empty [] so nothing is overridden
• client gets a 404 in response to the omero-client.json request and proceeds with a normal login

[joshmoore]

Hmmm...but "port" probably doesn't make any sense via HTTP. I wonder then if it shouldn't be a different endpoint:

• /$prefix/ice-client.json
• /$prefix/http-client.json

though maybe in a future HTTP world this all just goes away.

[manics]

If BlitzGateway is updated to match idr.connection it means handling a one-argument form differently from the multi-argument form:

• Currently BlitzGateway('idr.openmicroscopy.org', <other parameters ...>) connects to idr.openmicroscopy.org:4064
• The single argument form BlitzGateway('idr.openmicroscopy.org') would fetch omero-client.json. Is this OK?

[jburel]

what about the websocket case which is now critical for many notebooks/data access?
everything included in omero-client.json?
i.e.

[
"--Ice.Default.Router=OMERO.Glacier2/router:wss -p 443 -h idr.openmicroscopy.org -r /omero-ws",
"--omero.user=xxx",
"--omero.pass=xxx"
]

[manics]

Yes, that should also work if we want to switch everyone to using websockets. IIRC we've tested that before.

[joshmoore]

#106 (comment) Is this OK?

I guess since this optionally gathers the json login info and then falls back to previous behavior (?) it would be ok. Not sure if it wouldn't be safer to have a separate class to do the next-gen login. (Another option might be something like BlitzGateway(omero.client(host))...)

[manics]

Existing behaviour 😀

>>> from omero.gateway import BlitzGateway
>>> c = BlitzGateway('idr.openmicroscopy.org')
Traceback (most recent call last):
  File "/venv/lib/python3.8/site-packages/omero/clients.py", line 297, in _initData
    self.__ic = Ice.initialize(id)
  File "/venv/lib/python3.8/site-packages/Ice.py", line 705, in initialize
    communicator = IcePy.Communicator(args, data)
Ice.EndpointParseException: exception ::Ice::EndpointParseException
{
    str = unrecognized argument `not' in endpoint `ssl -p 4064 -h <"omero.host" not set>'
}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/venv/lib/python3.8/site-packages/omero/gateway/__init__.py", line 1594, in __init__
    self._resetOmeroClient()
  File "/venv/lib/python3.8/site-packages/omero/gateway/__init__.py", line 2144, in _resetOmeroClient
    self.c = omero.client(
  File "/venv/lib/python3.8/site-packages/omero/__init__.py", line 67, in client
    return omero.clients.BaseClient(*args, **kwargs)
  File "/venv/lib/python3.8/site-packages/omero/clients.py", line 170, in __init__
    self._initData(id)
  File "/venv/lib/python3.8/site-packages/omero/clients.py", line 301, in _initData
    raise omero.ClientError(msg)
omero.ClientError: No host specified. Use omero.client(HOSTNAME), ICE_CONFIG, or similar.

[jburel]

That's the problem I mentioned on Monday
and we adjust the notebook accordingly before switching to websockets

[manics]

@jburel Could you explain a bit more? I don't see how it's related to websockets.

[jburel]

it is not. I first had to adjust the way to connect then we started to use binder for most notebooks so I had to adjust anyway.