Buffer limit IOException thrown with validate-refs

[jordanpadams]

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

When I did validate-refs test:

validate-3.4.0-SNAPSHOT/bin/validate-refs --auth-opensearch config.properties urn:nasa:pds:orex.ovirs:data_calibrated::11.0

where config.properties points to https://search-en-prod-di7dor7quy7qwv3husi2wt5tde.us-west-2.es.amazonaws.com/_search

I get an IOException:

14:18:12.928 [main] FATAL gov.nasa.pds.validate.ri.OpensearchDocument - Error reading from URL: https://search-en-prod-di7dor7quy7qwv3husi2wt5tde.us-west-2.es.amazonaws.com/_search
java.io.IOException: entity content is too long [196292016] for the configured buffer limit [104857600]
	at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:958) ~[opensearch-rest-client-2.5.0.jar:2.5.0]
	at org.opensearch.client.RestClient.performRequest(RestClient.java:332) ~[opensearch-rest-client-2.5.0.jar:2.5.0]
	at org.opensearch.client.RestClient.performRequest(RestClient.java:320) ~[opensearch-rest-client-2.5.0.jar:2.5.0]
	at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1911) ~[opensearch-rest-high-level-client-2.5.0.jar:2.5.0]
	at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1877) ~[opensearch-rest-high-level-client-2.5.0.jar:2.5.0]
	at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1845) ~[opensearch-rest-high-level-client-2.5.0.jar:2.5.0]
	at org.opensearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1088) ~[opensearch-rest-high-level-client-2.5.0.jar:2.5.0]
	at gov.nasa.pds.validate.ri.OpensearchDocument.search(OpensearchDocument.java:152) ~[validate-3.4.0-SNAPSHOT.jar:?]
	at gov.nasa.pds.validate.ri.OpensearchDocument.load_refs(OpensearchDocument.java:114) [validate-3.4.0-SNAPSHOT.jar:?]
	at gov.nasa.pds.validate.ri.OpensearchDocument.getReferencesOf(OpensearchDocument.java:189) [validate-3.4.0-SNAPSHOT.jar:?]
	at gov.nasa.pds.validate.ri.Cylinder.run(Cylinder.java:47) [validate-3.4.0-SNAPSHOT.jar:?]
	at gov.nasa.pds.validate.ri.Engine.processQueueUntilEmpty(Engine.java:72) [validate-3.4.0-SNAPSHOT.jar:?]
	at gov.nasa.pds.validate.ri.CommandLineInterface.process(CommandLineInterface.java:121) [validate-3.4.0-SNAPSHOT.jar:?]
	at gov.nasa.pds.validate.ReferenceIntegrityMain.main(ReferenceIntegrityMain.java:14) [validate-3.4.0-SNAPSHOT.jar:?]
Caused by: org.apache.http.ContentTooLongException: entity content is too long [196292016] for the configured buffer limit [104857600]
	at org.opensearch.client.HeapBufferedAsyncResponseConsumer.onEntityEnclosed(HeapBufferedAsyncResponseConsumer.java:92) ~[opensearch-rest-client-2.5.0.jar:2.5.0]
	at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.responseReceived(AbstractAsyncResponseConsumer.java:137) ~[httpcore-nio-4.4.15.jar:4.4.15]
	at org.apache.http.impl.nio.client.MainClientExec.responseReceived(MainClientExec.java:315) ~[httpasyncclient-4.1.5.jar:4.1.5]
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseReceived(DefaultClientExchangeHandlerImpl.java:152) ~[httpasyncclient-4.1.5.jar:4.1.5]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.responseReceived(HttpAsyncRequestExecutor.java:315) ~[httpcore-nio-4.4.15.jar:4.4.15]
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:255) ~[httpcore-nio-4.4.15.jar:4.4.15]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87) ~[httpasyncclient-4.1.5.jar:4.1.5]
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40) ~[httpasyncclient-4.1.5.jar:4.1.5]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) ~[httpcore-nio-4.4.15.jar:4.4.15]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.15.jar:4.4.15]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.15.jar:4.4.15]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.15.jar:4.4.15]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.15.jar:4.4.15]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.15.jar:4.4.15]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.15.jar:4.4.15]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "java.util.Collection.toArray()" because "c" is null
	at java.base/java.util.ArrayList.addAll(ArrayList.java:670)
	at gov.nasa.pds.validate.ri.OpensearchDocument.getReferencesOf(OpensearchDocument.java:190)
	at gov.nasa.pds.validate.ri.Cylinder.run(Cylinder.java:47)
	at gov.nasa.pds.validate.ri.Engine.processQueueUntilEmpty(Engine.java:72)
	at gov.nasa.pds.validate.ri.CommandLineInterface.process(CommandLineInterface.java:121)
	at gov.nasa.pds.validate.ReferenceIntegrityMain.main(ReferenceIntegrityMain.java:14)

🕵️ Expected behavior

I expected the tool to work

📜 To Reproduce

validate-3.4.0-SNAPSHOT/bin/validate-refs --auth-opensearch config.properties urn:nasa:pds:orex.ovirs:data_calibrated::11.0

🖥 Environment Info

Mac OSx
Java openJDK17

📚 Version of Software Used

Version 3.4.0-SNAPSHOT
Release Date: 2023-11-02 14:14:31

🩺 Test Data / Additional context

No response

🦄 Related requirements

No response

⚙️ Engineering Details

No response

[al-niessner]

See bottom of this thread: https://stackoverflow.com/questions/51020646/org-apache-http-contenttoolongexception-entity-content-is-too-long-105539255

[miguelp1986]

@jordanpadams I cannot access https://search-en-prod-di7dor7quy7qwv3husi2wt5tde.us-west-2.es.amazonaws.com/_search with the credentials you provided me in the past. Should I get my own credentials to test this?

[viviant100]

@miguelp1986 you have your own credentials as well (set up Oct 2023). I just sent you your username just in case.

[miguelp1986]

@miguelp1986 you have your own credentials as well (set up Oct 2023). I just sent you your username just in case.

@viviant100 I just found my saved credentials lol. Thank you for sending them anyway.

[miguelp1986]

@jordanpadams @al-niessner I'm getting Could not connect warnings when running the command from this issue:

validate-refs --auth-opensearch ~/Documents/PDS/validate_test_files/748/config.properties --threads 15 urn:nasa:pds:orex.ovirs:data_calibrated::11.0 --verbose

12:33:43.563 [main] WARN  gov.nasa.pds.validate.ri.CommandLineInterface - Using Registry OpenSearch Database to check references.
12:33:43.606 [main] INFO  gov.nasa.pds.validate.ri.CommandLineInterface - Starting the duplicate filename in FileArea checks.
12:33:43.607 [main] INFO  gov.nasa.pds.validate.ri.CommandLineInterface - Starting the reference integrity checks.
12:33:43.607 [main] INFO  gov.nasa.pds.validate.ri.Engine - Have 1 lidvids to check with 0 threads to get them done.
12:33:44.266 [Thread-1] INFO  gov.nasa.pds.validate.ri.Cylinder - The lidvid 'urn:nasa:pds:orex.ovirs:data_calibrated::11.0' is of type: Product_Collection
13:15:59.912 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
13:32:45.136 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
13:39:45.542 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
13:41:49.967 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
13:42:39.280 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
13:44:24.478 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
13:48:36.499 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
13:49:02.710 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
13:50:30.204 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
13:51:50.140 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
13:53:18.767 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
13:57:57.426 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
13:59:54.804 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
14:01:01.853 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
14:01:38.417 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
14:06:18.799 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
14:06:38.925 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1
14:09:18.077 [Thread-1] WARN  gov.nasa.pds.validate.ri.OpensearchDocument - Could not connect but trying again: 1

I am able to run validate-refs for other issues successfully. I've tried using https://search-en-prod-di7dor7quy7qwv3husi2wt5tde.us-west-2.es.amazonaws.com:443 and https://search-en-prod-di7dor7quy7qwv3husi2wt5tde.us-west-2.es.amazonaws.com:443/_search and I'm getting the same result.