Stress tests fail because OkHttp Digest auth doesn't support multi-threaded Java

[sammefford]

As reported in rburgst/okhttp-digest#33 , we encountered an issue when we ran with OkHttp in our stress tests. I see most tests fail with error messages like:

WARNING: previous digest authentication with same nonce failed, returning null
com.marklogic.client.FailedRequestException: Local message: write failed: Unauthorized. Server Message: Unauthorized
        at com.marklogic.client.impl.OkHttpServices.putPostDocumentImpl(OkHttpServices.java:1466)
        at com.marklogic.client.impl.OkHttpServices.putDocument(OkHttpServices.java:1117)
        at com.marklogic.client.impl.DocumentManagerImpl.write(DocumentManagerImpl.java:836)
        at com.marklogic.client.impl.DocumentManagerImpl.write(DocumentManagerImpl.java:708)
        at com.marklogic.client.impl.DocumentManagerImpl.write(DocumentManagerImpl.java:652)
        at test.stress.JavaAPISession.putDocument(JavaAPISession.java:220)
        at test.stress.RestQueryTester.runTest(RestQueryTester.java:133)
        at test.stress.StressTest.run(StressTest.java:378)

This should definitely not be happening, because the user is authorized and most requests succeed. To have some intermittent failures with the error "FailedRequestException ... Unauthorized" means the same user/password combo is failing intermittently. As reported in the issue linked above, I believe this is because the tests are running multi-threaded, and DigestAuthenticator isn't designed for that. We must either get a fix from the developer or fix this ourselves as most of our clients deploy our API in a multi-threaded environment.

[sammefford]

So far the developer of the library has been very responsive. He has not, however, committed to making the fix.

[sammefford]

I made the fix, and submitted a pull request: rburgst/okhttp-digest#35 . Hopefully he'll accept it. I re-ran the stress tests today with the fix in place and the issue is now gone. I see a couple other issues, but I believe they're unrelated.

In order to get this to work I had to:

• git clone git@github.com:sammefford/okhttp-digest.git
• cd okhttp-digest
• add two lines to build.gradle:
  • apply plugin: 'maven'
  • version = "1.12.1"
• gradle install
  • this creates ~/.m2/repository/com/burgstaller/okhttp-digest/1.12.1/okhttp-digest-1.12.1.jar
• cp ~/.m2/repository/com/burgstaller/okhttp-digest/1.12.1/okhttp-digest-1.12.1.jar $QA_HOME/lib/stress/lib/javaclientapi/
  • also copy all the below okhttp jars from similar locations in ~/.m2/repository
• change stress/widowmaker/run-rest.sh to use the following jars instead of jersey* and httpclient* and httpcore* and mimepull* and commons-logging*
  • $QA_HOME/lib/stress/lib/javaclientapi/okhttp-3.6.0.jar
  • $QA_HOME/lib/stress/lib/javaclientapi/logging-interceptor-3.6.0.jar
  • $QA_HOME/lib/stress/lib/javaclientapi/okio-1.11.0.jar
  • $QA_HOME/lib/stress/lib/javaclientapi/okhttp-digest-1.12.1.jar
  • $QA_HOME/lib/stress/lib/javaclientapi/javax.mail-1.5.6.jar
  • $QA_HOME/lib/stress/lib/javaclientapi/activation-1.1.jar
  • $QA_HOME/lib/stress/lib/javaclientapi/javax.ws.rs-api-2.0.1.jar
• nohup sh $QA_HOME/stress/widowmaker/run-rest.sh > /space/run-rest.log 2>&1 &

[sammefford]

This process is obviously a stop-gap. Before we can release we'll need the library author to accept my PR and push a new release to jcenter so we can depend on that. Or if that doesn't happen quickly enough we'll need to decide if we push our own release of it and depend on that.

[sammefford]

The fix I submitted was accepted and pushed to jcenter as okhttp-digest version 1.14. So I've update our pom.xml to use that.