Che Server rejects JSON RPC messages which may lead to losing events

[sleshchenko]

Description

Che Server rejects JSON RPC messages which may lead to losing events.
There is a ci-test job that failed because of slow tests execution.
#11920 (comment)

Che Server Logs contain a lot of logged wsagent log events like the following:

2018-11-15 08:47:48,833[stProcessor-285]  [ERROR] [l.ErrorInstallerLogEventLogger 43]   - Installer `org.eclipse.che.ws-agent` error from machine=`dev-machine` owner=`1cf07453-4b2a-4226-8ca0-c22189c5da34` env=`workspacejm96ne` workspace=`workspaceii5wyhr367a3cmz3` text=`  "params": {` time=`2018-11-15T08:47:48.827536783Z`
2018-11-15 08:47:48,833[stProcessor-360]  [ERROR] [l.ErrorInstallerLogEventLogger 43]   - Installer `org.eclipse.che.ws-agent` error from machine=`dev-machine` owner=`1cf07453-4b2a-4226-8ca0-c22189c5da34` env=`workspacejm96ne` workspace=`workspaceii5wyhr367a3cmz3` text=`    "uri": "file:///projects/console-cpp-simple/iseven.cpp"` time=`2018-11-15T08:47:48.827616797Z`
2018-11-15 08:47:48,834[stProcessor-354]  [ERROR] [l.ErrorInstallerLogEventLogger 43]   - Installer `org.eclipse.che.ws-agent` error from machine=`dev-machine` owner=`1cf07453-4b2a-4226-8ca0-c22189c5da34` env=`workspacejm96ne` workspace=`workspaceii5wyhr367a3cmz3` text=`    "uri": "file:///projects/console-cpp-simple/iseven.cpp"` time=`2018-11-15T08:47:48.804885696Z`

After that there are errors that JSON RPC messages are dropped:

2018-11-15 08:49:19,326[nio-8080-exec-1]  [WARN ] [j.i.ServerSideRequestProcessor 62]   - Message org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable@5e59b14 rejected for execution
2018-11-15 08:49:19,326[nio-8080-exec-1]  [WARN ] [j.i.ServerSideRequestProcessor 62]   - Message org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable@25c49e90 rejected for execution

It happes because JSON RPC Thead Executor is full.

Except logs, there are bootstrapper/plugin broker events which MUST NOT be rejected because it will lead to workspace start failure.

The idea of how to solve this issue is to limit the number of logs which are propagated by bootstrapper like 10 per second or maybe sent them in batch.

Reproduction Steps

OS and version:

Diagnostics:
Full Che Servers Log available here https://ci.codenvycorp.com/job/che-pullrequests-test-ocp/1179/Selenium_20tests_20report/che_server_logs.zip