Odinson server cuts connection after 75 seconds

[mcshlain]

Describe the bug
When executing a long query (against the backend API) the connection is cut by the server.
The connection always terminates at the 1 minute, 15 seconds mark. This makes it hard to work with odinson against very large corpora.

To Reproduce
Any query that doesn't return in 75 seconds.

Expected behavior
I would expect the connection to stay alive until results are returned.

Screenshots/Errors
Two output of curl and the timing with date (IP parts replaced by X).

date && curl -v -X GET "http://X.X.X.X:9000/api/search?odinsonQuery=%5B%5D%20%3Edobj%20%5B%5D" -H "accept: application/json" || date
Sun Sep  8 15:23:56 IDT 2019
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying X.X.X.X...
* TCP_NODELAY set
* Connected to X.X.X.X (X.X.X.X) port 9000 (#0)
> GET /api/search?odinsonQuery=%5B%5D%20%3Edobj%20%5B%5D HTTP/1.1
> Host: X.X.X.X:9000
> User-Agent: curl/7.58.0
> accept: application/json
> 
* Recv failure: Connection reset by peer
* stopped the pause stream!
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
Sun Sep  8 15:25:11 IDT 2019

date && curl -v -X GET "http://X.X.X.X:9000/api/search?odinsonQuery=%5B%5D%20%3Edobj%20%5B%5D" -H "accept: application/json" || date
Sun Sep  8 15:30:00 IDT 2019
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying X.X.X.X...
* TCP_NODELAY set
* Connected to X.X.X.X (X.X.X.X) port 9000 (#0)
> GET /api/search?odinsonQuery=%5B%5D%20%3Edobj%20%5B%5D HTTP/1.1
> Host: X.X.X.X:9000
> User-Agent: curl/7.58.0
> accept: application/json
> 
* Recv failure: Connection reset by peer
* stopped the pause stream!
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
Sun Sep  8 15:31:16 IDT 2019

[myedibleenso]

@mcshlain, can you please try running the issue35 branch branch to confirm that those changes take care of timeout you reported?

[mcshlain]

Will do.

[mcshlain]

The branch did not solve the issue, I'm still getting the connection cut after 75 seconds:

'''
$ date > time.it && curl -X GET "http://localhost:9000/api/search?odinsonQuery=%5B%5D%20%3Cnsubj%20%5B%5D" -H "accept: application/json" || date >> time.it
$ cat time.it
Mon Sep 23 09:12:49 IDT 2019
Mon Sep 23 09:14:04 IDT 2019
'''

[mcshlain]

I've encountered this:
https://www.playframework.com/documentation/2.6.x/SettingsAkkaHttp
There is an option there called idleTimeout:

      # The idle timeout for an open connection after which it will be closed
      # Set to null or "infinite" to disable the timeout, but notice that this
      # is not encouraged since timeout are important mechanisms to protect your
      # servers from malicious attacks or programming mistakes.
      idleTimeout = 75 seconds

which is set to exactly 75 seconds, I tried playing with it at some point but couldn't make it work, but I have no experience with the play framework so I might have not been setting it correctly.

[myedibleenso]

Thanks for testing it out.

Would you mind trying once more? I think I've figured out the problem. It seems that in development mode the HTTP server starts before the config is available, so this sort of setting has to be specified in the project's build.sbt.

[mcshlain]

The new change worked, the query I used returned after 750 seconds (so it did not got cut after 75).

[myedibleenso]

Curious to hear what sort of query and corpus you're using, but glad this last change took care of the timeout.

[mcshlain]

We're using the full wikipedia corpus (the index is 100GB) and I tested this with a [] <nsubj [] query. This is not the query that is actually interesting but it recreated the problem we encountered with other queries.

[myedibleenso]

Now that's a real index! You're probably already aware of this, but if you're not using the UI (or at least not the detailed parse view), you can set this config property to false to substantially reduce the size of the index:

• odinson/extra/src/main/resources/application.conf

  Line 21 in 8fd6ebb

  storeSentenceJson = true

[mcshlain]

thanks, I don't actually remember if we used this option or not when creating the index.