Database clear after one or multiple ServerEvaluationCall eval on SSL enabled REST server has issues. #446
Description
In the QA functional tests, whenever the tests are running on SSL enabled REST server, it has been observed that the database clear operation times out, when there are one or multiple ServerEvaluationCall eval() methods used (in a loop of 300+).
The test inserts one document (binary of size 4MB) and then loop the eval query for 330 times and verifies that it should not be stuck. The code pasted below has the loop counter set to 1.
The test on teardown clears the database and release the client. But the clear database operation times out and throws an exception. This is the exception from teardown.
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:149)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:110)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:260)
at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:252)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:281)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:247)
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:219)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:298)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:633)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:454)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
at com.marklogic.client.functionaltest.ConnectedRESTQA.clearDB(ConnectedRESTQA.java:845)
at com.marklogic.client.functionaltest.ConnectedRESTQA.clearDB(ConnectedRESTQA.java:2096)
at com.marklogic.client.functionaltest.TestEvalwithRunTimeDBnTransactions.tearDown(TestEvalwithRunTimeDBnTransactions.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Issue seen: 1) Subsequent QA Junit tests fail with SSLPeerUnverifiedException when trying to write documents on the same SSL enabled REST server. Here is a stack trace of a test that was run after the ServerEvaluationCall eval() test.
- On doing a manual clear of the database using the MarkLogic Admin GUI, the clear database operation takes about 5 minutes to clear the one document.
Snippet of the trace from other failed test is below.
com.sun.jersey.api.client.ClientHandlerException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:184)
at com.marklogic.client.impl.DigestChallengeFilter.handle(DigestChallengeFilter.java:34)
at com.sun.jersey.api.client.filter.HTTPDigestAuthFilter.handle(HTTPDigestAuthFilter.java:493)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.head(WebResource.java:176)
at com.marklogic.client.impl.JerseyServices.makeFirstRequest(JerseyServices.java:497)
at com.marklogic.client.impl.JerseyServices.putPostDocumentImpl(JerseyServices.java:1303)
at com.marklogic.client.impl.JerseyServices.putDocument(JerseyServices.java:1183)
at com.marklogic.client.impl.DocumentManagerImpl.write(DocumentManagerImpl.java:913)
at com.marklogic.client.impl.DocumentManagerImpl.write(DocumentManagerImpl.java:751)
at com.marklogic.client.impl.DocumentManagerImpl.write(DocumentManagerImpl.java:681)
at com.marklogic.client.functionaltest.BasicJavaClientREST.writeDocumentUsingInputStreamHandle(BasicJavaClientREST.java:142)
at com.marklogic.client.functionaltest.TestAggregates.testTuplesAggregates(TestAggregates.java:222)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(Unknown Source)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:390)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:561)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:776)
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:170)
... 38 more
Note : This issue does not happen when REST server is not SSL enabled.
Steps used to reproduce:
- Create a SSL enabled REST server.
- Create a role "test-eval" with Previlages ("xdbc:eval", "xdbc:eval-in","xdmp:eval-in","any-uri","xdbc:invoke")
- Create a user "user1" having "test-eval","rest-admin","rest-writer","rest-reader" roles.
- Get a DatabaseClient instance that has the SSLContext parameter set for the user1.
- Run the following test. The binary file is available from Git. Note: I have tested with loop counter values of 330, 100, 10 and even 1. The issue remains the same. Do not clear the database contents from Java client API. Use Admin GUI to clear database contents
@Test
public void test1MultipleEvalQueries() throws KeyManagementException, NoSuchAlgorithmException, Exception {
GenericDocumentManager docMgr = client.newDocumentManager();
File file1= null;
FileInputStream fis=null;
try {
file1 = new File("src/test/java/com/marklogic/client/functionaltest/data/Sega-4MB.jpg");
fis = new FileInputStream(file1);
InputStreamHandle handle1 = new InputStreamHandle(fis);
handle1.setFormat(Format.BINARY);
docMgr.write("/binary4mbdoc",handle1);
String query = "declare variable $myInteger as xs:integer external;"
+ "(fn:doc()/binary(),$myInteger,xdmp:database-name(xdmp:database()))";
long sizeOfBinary =docMgr.read("/binary4mbdoc",new InputStreamHandle()).getByteLength();
for(int i=0 ; i<=1;i++) {
ServerEvaluationCall evl= client.newServerEval().xquery(query);
evl.addVariable("myInteger", (int)i);
EvalResultIterator evr = evl.eval();
while(evr.hasNext()) {
EvalResult er =evr.next();
if(er.getType().equals(Type.INTEGER)) {
assertEquals("itration number",i,er.getNumber().intValue());
} else if(er.getType().equals(Type.BINARY)) {
FileHandle readHandle1 = new FileHandle();
assertEquals("size of the binary ",er.get(readHandle1).get().length(),sizeOfBinary);
} else if(er.getType().equals(Type.STRING)) {
assertEquals("database name ","TestEvalXqueryWithTransDB",er.getString());
} else {
fail("Getting incorrect type");
}
}
}
}
catch(Exception e) {
throw e;
}
finally {
fis.close();
}
}
- Now using the Admin GUI clear the database contents .The test used Apache HttpClient to clear the database. Admin GUI takes 5 minutes to clear the contents (1 document).
Code used to clear the DB from JUnit teardown. Uses dbName as database name to be cleared.
// In case of SSL use 8002 port to clear DB contents.
DefaultHttpClient client = new DefaultHttpClient();
client.getCredentialsProvider().setCredentials(
new AuthScope("localhost", 8002),
new UsernamePasswordCredentials("admin", "admin"));
HttpGet getrequest = new HttpGet("http://localhost:8002/manage/v2/servers/"+getRestAppServerName()+"/properties?group-id=Default&format=json");
HttpResponse response1 = client.execute(getrequest);
jsonstream = response1.getEntity().getContent();
JsonNode jnode= new ObjectMapper().readTree(jsonstream);
String dbName = jnode.get("content-database").asText();
System.out.println("App Server's content database properties value from ClearDB is :"+ dbName);
ObjectMapper mapper = new ObjectMapper();
ObjectNode mainNode = mapper.createObjectNode();
mainNode.put("operation", "clear-database");
HttpPost post = new HttpPost("http://localhost:8002"+ "/manage/v2/databases/" + dbName);
post.addHeader("Content-type", "application/json");
post.setEntity(new StringEntity(mainNode.toString()));
HttpResponse response = client.execute(post);
HttpEntity respEntity = response.getEntity();
if( response.getStatusLine().getStatusCode() == 400) {
System.out.println("Database contents cleared");
}
else if (respEntity != null) {
// EntityUtils to get the response content
String content = EntityUtils.toString(respEntity);
System.out.println(content);
}
else {
System.out.println("No Proper Response from clearDB in SSL.");
}
- Run a different test with another database and DatabaseClient instance on the same SSL REST server. Write a document into this new database
We see PeerUnverifiedException while attempting to write.