Web API versioning

[will-moore]

We recently discussed this issue, which is kinda complex and really needs some decisions made.

API versions

Currently we have version in api url E.g. /api/v0.1/m/projects/ but this is only useful if we're supporting more than 1 version at a time.
So, do we want to do that, and if so, how?

When we last discussed this, I think the consensus was moving towards "Only support 1 version at a time".
This will really simplify our lives (and be much less confusing for users).
If users really need access to multiple versions at a time, they can choose to deploy 2 separate versions of the app, under different url prefixes of their choosing.

Any objections to this approach?

OMERO_marshal versions

Currently, the web API will be affected by:

• Changes to OMERO mode, Omeroweb/api/ app & BlitzGateway (whether to split these is another discussion)
• Version of omero_marshal

Do we specify a fixed version of omero_marshal for each release of the OMERO? Currently this is fixed at 0.4.1 in requirements-common.txt
Or do we allow users to install latest omero_marshal and say in our API docs that the json you get will depend on your omero_marshal version?

Decoupling

If we decouple OMERO, OMERO.web, BlitzGateway, api app etc, then these questions get amplified by many factors! Since we are currently moving more of the web api logic into BlitzGateway, these are becoming less decoupled. I don't see us resolving these issues in time for 5.3.0 since the priority has to be on building the api itself.

cc @chris-allan @joshmoore @aleksandra-tarkowska @jburel @emilroz

[joshmoore]

If users really need access to multiple versions at a time, they can choose to deploy 2 separate versions of the app, under different url prefixes of their choosing.

Will this really work? I assume there will be clashes in the library names unless the primary version appears in the name of all the related code (omero_api_v1, omero_marshal_v1, ...) If not, it sounds more like there would be the need to deploy and manage multiple web servers.

BlitzGateway (whether to split these is another discussion)

As an aside, having all of omero-api depend on all BlitzGateway seems cumbersome, which is mostly just a vote for the follow-up splitting discussion.

Changes to OMERO model...

Can we hide changes to OMERO model behind the omero_marshal version, which brings me to the point:

Or do we allow users to install latest omero_marshal and say in our API docs that the json you get will depend on your omero_marshal version?

If possible, I'd think pinning the api to a major marshal version would be appropriate.

[manics]

How will this appear to non-omero clients using the API? If the API version isn't in the URL is there an easy way for them to detect it?

[sbesson]

Seconding #64 (comment), the strategy for omero-marshal has been to handle new/breaking schema changes as major version increments (currently minor version increments as we are sub 1.0.0) - see e.g. https://github.com/openmicroscopy/omero-marshal/milestone/3?closed=1.

I would also vote for pinning the dependency of omero-marshal in the API like we do for all our dependencies. If a minor/major release of omero-marshal modifies the content/layout of the JSON, we should clearly document the changes and upgrade process for consumers of API and also expose the changes via a minor/major version release of the API component anyways.

[chris-allan]

Agreeing with the above comments. I don't think not having the major version in the URL is a usability improvement. Personally I'd also prefer v1, v2, v3, etc. and not a decimal version number that needs to be parsed but that's stylistic really.

Yes, we should pin the omero-marshal dependency major version. As @sbesson has already mentioned, it has been a deliberate goal to avoid changing the content or layout of marshaled dictionaries and resultant JSON except in major version.

In an ideal world, omero-marshal should also be isolating us from OMERO model changes. No doubt there are exceptions to this but they should be minor and we are working very hard to minimise them. That was one of its original design goals. If there is interest, for instance, in supporting a previous schemat then we only need to perform the implementation.

Regarding BlitzGateway coupling this is certainly a concern but not one we are about to fix for 5.3.0. I had previously considered effectively "banning" BlitzGateway usage in the web API. However, that becomes unwieldy quite quickly and results in a significant amount of needless code duplication. I'm of the opinion we should be using this opportunity to improve the BlitzGateway as it pertains to the web API. This has the added value of forcing us to fix longstanding issues with it. Furthermore, I would argue that strategy has been relatively successful already considering the past and current bugfix, simplification and uniformity PRs targeted to BlitzGateway during the web API work.

[atarkowska]

If users really need access to multiple versions at a time, they can choose to deploy 2 separate versions of the app, under different url prefixes of their choosing.

Will this really work? I assume there will be clashes in the library names unless the primary version appears in the name of all the related code (omero_api_v1, omero_marshal_v1, ...) If not, it sounds more like there would be the need to deploy and manage multiple web servers.

I think this is what deploy 2 separate versions of the app means. On the last meeting it was proposed to deploy multiple web instances and maintain API version independent of the api django app by SCRIPT_NAME which is omero.web.prefix. So if someone would like to deploy OMERO.web 5.3 another version of omero_marshal it would be :

http://host/v1/api/projects/
http://host/v2/api/projects/

it seams to be more like others do rather then hidden in the middle of URL

Maybe using prefix is not the most intuitive way, but we could just have omero.web.api.version which would be merged with prefix. This value could also be set directly from omero_marshal package.

What is exactly the use-case where someone would like to run multiple versions of API at the same time?

[atarkowska]

How will this appear to non-omero clients using the API? If the API version isn't in the URL is there an easy way for them to detect it?

I think https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/tools/OmeroWeb/omeroweb/api/views.py#L64 will give you version including absolute urls

[manics]

What is exactly the use-case where someone would like to run multiple versions of API at the same time?

Public data repositories such as the IDR

[atarkowska]

@manics, sure but more specific and realistic one?

[manics]

Isn't that realistic enough?

[chris-allan]

Just speculative but since we have already seen something similar with PathViewer and the need to support multiple server versions I don't think it's a stretch to expect that at some point a deployment may need to run two versions of the API to support clients or plugins that work exclusively with one version of the API or another.

The lion share of web APIs that I have interacted with have the version number after the /api (ex. /api/v1/foo). That said, we should probably not be using other people's public APIs as the sole justification for our design decisions. Their requirements are often fundamentally different than ours. GitHub is obviously a notable exception to this. They have many reasons for this and are even in the process of elevating their GraphQL work which is actually quite similar to the trifecta of IQuery, HQL and OMERO model objects. You can read whatever you want into that.

[chris-allan]

A good blog post, and further interesting set of references, in case anyone really has the patience to read up more on the topic and get an idea of some of the alternatives:

• https://www.troyhunt.com/your-api-versioning-is-wrong-which-is/

[atarkowska]

I think my question was about concrete examples of how and where we could consider deploying multiple versions of API within single instance of OMERO.web? What is possible now and what may be possible in the future? (that is brainstorming, isn't it?) My impression is that we strictly connect versioning of API with schema, although data itself may also be versioned using the same version of software.

[manics]

Can the web-api be run on it's own (just django, no OMERO.web)?

[will-moore]

@manics No, /api app uses a lot of OMERO.web for login, connection, decorators etc.