Unfinished requests in Tomcat's AJP connector

[gomezgoiri]

When I check the status of Tomcat, I see a section called ajp-bio-8009 which has some old requests in the Service stage:

S   672034 ms   0 KB    0 KB    137.108.118.197 137.108.118.197 forge.kmi.open.ac.uk    GET /manager/status?org.apache.catalina.filters.CSRF_NONCE=25744C78960A454C9C3AAE5F0E7F8A93 HTTP/1.1
S   51216220 ms 0 KB    0 KB    46.208.21.235   46.208.21.235   forge.kmi.open.ac.uk    GET /api/v1/sessions/tVkHHrQlRg2XpcZnvumz7A--/network HTTP/1.1
S   6087074 ms  0 KB    0 KB    137.108.118.197 137.108.118.197 forge.kmi.open.ac.uk    GET /api/management/backend.html HTTP/1.1
S   6107918 ms  0 KB    0 KB    137.108.118.197 137.108.118.197 forge.kmi.open.ac.uk    GET /api/ HTTP/1.1
S   616515 ms   0 KB    0 KB    137.108.118.197 137.108.118.197 forge.kmi.open.ac.uk    GET /manager/status?org.apache.catalina.filters.CSRF_NONCE=25744C78960A454C9C3AAE5F0E7F8A93 HTTP/1.1
S   51218377 ms 0 KB    0 KB    46.208.21.235   46.208.21.235   forge.kmi.open.ac.uk    GET /api/v1/sessions/tVkHHrQlRg2XpcZnvumz7A--/network HTTP/1.1
S   51222753 ms 0 KB    0 KB    46.208.21.235   46.208.21.235   forge.kmi.open.ac.uk    GET /api/v1/sessions/tVkHHrQlRg2XpcZnvumz7A--/network HTTP/1.1

I created this requests at some point in the past, but even if they entered into an erroneous state (I suspect of 500 Internal Server Error I got from Tomcat), they should have been destroyed by Tomcat as they are not used anymore.

Note: This issue is more a deployment problem than an issue directly related with the API development. However, I will put here as it has to with the API performance.

[gomezgoiri]

The AJP connector is probably linked to the Apache reverse proxy used.

Therefore, I have checked in the Apache reverse proxy's logs and I have seen the following related error lines:

./error_log_2016-01.log:[Wed Jan 20 22:26:20.394599 2016] [proxy_ajp:error] [pid 19228] [client 46.208.21.235:51720] AH00992: ajp_read_header: ajp_ilink_receive failed, referer: http://forge.kmi.open.ac.uk/pt/app/default.html?session=tVkHHrQlRg2XpcZnvumz7A--
./error_log_2016-01.log:[Wed Jan 20 22:26:20.394630 2016] [proxy_ajp:error] [pid 19228] (70007)The timeout specified has expired: [client 46.208.21.235:51720] AH00878: read response failed from [::1]:8009 (localhost), referer: http://forge.kmi.open.ac.uk/pt/app/default.html?session=tVkHHrQlRg2XpcZnvumz7A--

Both error lines have appeared 1160 times since March 09, 2015.

The error and its solution are perfectly described here.

[gomezgoiri]

Solution adopted (suggested by systems), set connectionTimeout (it was unset):

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" connectionTimeout="20000" />

After restarting it, no more threads appear indefinitely in the Service stage.

Now once a request finishes they pass to the Keepalive stage and then they disappear. Note that a browser tab might be opening several requests.