forbidden requests cause elasticsearch to fullGC

[boicehuang]

Hello.
I installed readonlyrest-1.16.14_es5.6.4.zip. When requests that have no permission to the cluster were forbidden by the plugins, I noticed that they remained in the cluster by using task management API, and the number of them increases when more HTTP requests are forbidden. After a few days, i found that there were millions of tasks remained in memory and causing continuous full GC.

• command:

curl "localhost:9200/_tasks?pretty&filter_path=nodes.*.tasks.*.action" | grep action | sort | uniq -c

• result:

236 "action" : "cluster:monitor/health"
8191197 "action" : "cluster:monitor/main"

• log:
  [2017-12-28T12:14:44,526][WARN ][o.e.m.j.JvmGcMonitorService] [1498146454000002009] [gc][old][240145][905] duration [16.8s], collections [1]/[17s], total [16.8s]/[2.1h], memory [29.1gb]->[12.1gb]/[29.9gb], all_pools {[young] [416.5mb]->[3.8mb]/[665.6mb]}{[survivor] [83.1mb]->[0b]/[83.1mb]}{[old] [28.6gb]->[12.1gb]/[29.1gb]}

  It is easy to make this reproduce. All you have to do is sending forbidden requests to cluster continuously and check tasks of the cluster.

[sscarduzio]

Hi @viewsite thanks for reporting this. It's a regression, as we found this issue before, months ago before a couple of major refactoring sessions which must have reintroduced this behaviour.

Just to let you know I'm right on this at the moment. Will update soon with progress. 👍

[sscarduzio]

@viewsite this is now fixed, will release a new version this weekend!

[boicehuang]

Great!