Formulate a long-term HTTP client plan

[jmakeig]

We use Jersey currently for HTTP communication with the MarkLogic REST API. However, the long-term viability of Jersey is uncertain. (TODO: enumerate existing issues.) Let’s put together a plan for a robust HTTP client that will support our current and future needs.

[sammefford]

This is related / overlapping #65 "Upgrade or remove Jersey"

[sammefford]

With the upcoming HTTP/2, some core paradigms may change. Any plans we make should account for that fact.

One example of the impact of HTTP/2 push: DMSDK uses pagination and checks each returned batch to see if it's a full batch, then if it is it requests the next batch presuming there are more matches. With HTTP/2 push it might be better to ask the server to push all matches to us since the server knows when there are no more matches.

[jmakeig]

With the upcoming HTTP/2…

There are no immediate plans to implement HTTP/2 in MarkLogic’s HTTP server. However, it’s something we will be assessing for a future release. The hypothesis is the HTTP/2 will provide better performance and conformance, especially for workloads like the Java Client API.

[sammefford]

Thank you, Justin, for clarifying.

[sammefford]

I've done a lot of evaluation of many HTTP libraries. The most popular libraries: Apache HTTPClient and URLConnection are both in "alpha" or "incubator" status on complete overhauls due to two upcoming changes: HTTP/2, and reactive-style (async) development. While either choice has current stable releases that would serve well our current needs, we'd most likely want to throw them out before too long to take advantage of their new incarnations.

OkHttp appears to be mature, stable, fast, supported, and it appears it will support all our non-trivial requirements, which include:

• auth types: basic, digest, Kerberos, Certificate-based auth (SSL client cert)
• multipart attachments
• support all REST request types (including PATCH, DELETE, etc.)
• SSL
• pool/reuse HTTP connections
• chunked encoding (streaming) requests
• retry when appropriate
• gzip
• 100/continue

The current plan is to implement #65 using OkHttp. That's slated to happen in 4.0.3.

[sammefford]

@jmakeig referenced "clever dependency isolation" tricks in #737, and I thought a summary of discussion about those belongs here. The reported problem is that Jersey 2.x conflicts with Jersey 1.x when both are on the classpath (#282). Jersey 1.x code is very out of date, which is why we initiated an upgrade task. Unfortunately, Jersey 2.x had a bug which made it unusable for us.

As a possible "path of least resistance" we discussed the possibility of "clever dependency isolation". I believe the options are:

1. use OSGI or parent-first classloaders to allow peaceful jar co-existance within a JVM
2. repackage our Jersey so it doesn't conflict

The problem with (1) is it's not an option in all scenarios, and requires advanced configuration. We'd rather not require people jump through hoops to avoid this conflict.

The problem with (2) is that we'd then need a repository and distribution channel for that code. And we'd be on the hook to maintain that code.

After discussion we decided that given these challenges the above plan to upgrade to a new HTTP library would is "the path of least resistance".