New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove `fuzzy_like_this` query #10391

Merged
merged 1 commit into from Apr 10, 2015

Conversation

Projects
None yet
6 participants
@jpountz
Contributor

jpountz commented Apr 2, 2015

The fuzzy-like-this query builds very expensive queries and only serves esoteric
use-cases.

@jpountz

This comment has been minimized.

Show comment
Hide comment
@jpountz

jpountz Apr 2, 2015

Contributor

The plan is to mark it as deprecated in 1.x.

Contributor

jpountz commented Apr 2, 2015

The plan is to mark it as deprecated in 1.x.

@s1monw

This comment has been minimized.

Show comment
Hide comment
@s1monw

s1monw Apr 2, 2015

Contributor

+1

Contributor

s1monw commented Apr 2, 2015

+1

@s1monw s1monw self-assigned this Apr 2, 2015

@s1monw

This comment has been minimized.

Show comment
Hide comment
@s1monw

s1monw Apr 2, 2015

Contributor

LGTM yet the only thing that I am always wondering about is what are we doing if somebody has this thing stuck in the the translog as a delete by query? It's not sovleable to be honest unless we say f*** it

Contributor

s1monw commented Apr 2, 2015

LGTM yet the only thing that I am always wondering about is what are we doing if somebody has this thing stuck in the the translog as a delete by query? It's not sovleable to be honest unless we say f*** it

@jpountz

This comment has been minimized.

Show comment
Hide comment
@jpountz

jpountz Apr 2, 2015

Contributor

It's a valid concern. However I don't think this query is used much for delete by query, so few users would be impacted, and if they are it would still be possible to downgrade, restore a snapshot, flush and upgrade again? I'm personally more concerned about the percolator, but this is what it is: if we can't have such breaks on rarely used queries then we have no opportunities to clean things up?

Contributor

jpountz commented Apr 2, 2015

It's a valid concern. However I don't think this query is used much for delete by query, so few users would be impacted, and if they are it would still be possible to downgrade, restore a snapshot, flush and upgrade again? I'm personally more concerned about the percolator, but this is what it is: if we can't have such breaks on rarely used queries then we have no opportunities to clean things up?

Queries: Remove fuzzy-like-this support.
The fuzzy-like-this query builds very expensive queries and only serves esoteric
use-cases.

jpountz added a commit that referenced this pull request Apr 10, 2015

Merge pull request #10391 from jpountz/fix/remove_flt
Queries: Remove fuzzy-like-this support.

Close #10391

@jpountz jpountz merged commit bad6656 into elastic:master Apr 10, 2015

1 check passed

CLA Commit author is a member of Elasticsearch
Details

@jpountz jpountz removed the review label Apr 10, 2015

@markharwood

This comment has been minimized.

Show comment
Hide comment
@markharwood

markharwood Apr 13, 2015

Contributor

only serves esoteric use-cases.

One non-esoteric use case is as a work-around to Lucene's fuzzy bug which I never managed to get acknowledged as an issue.
This came up recently with one of our users.

Agreed FLT is not a long-term solution but it was always a work-around I'd reach for when hitting this issue with Lucene.

Maybe the removal of this functionality should coincide with a proper fix for core Lucene?

Contributor

markharwood commented Apr 13, 2015

only serves esoteric use-cases.

One non-esoteric use case is as a work-around to Lucene's fuzzy bug which I never managed to get acknowledged as an issue.
This came up recently with one of our users.

Agreed FLT is not a long-term solution but it was always a work-around I'd reach for when hitting this issue with Lucene.

Maybe the removal of this functionality should coincide with a proper fix for core Lucene?

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley Apr 13, 2015

Member

@markharwood well volunteered!

Member

clintongormley commented Apr 13, 2015

@markharwood well volunteered!

@markharwood

This comment has been minimized.

Show comment
Hide comment
@markharwood

markharwood May 1, 2015

Contributor

I've taken the guts of our BlendedTermQuery and working it back into Lucene's FuzzyQuery rewrite capabilities.
The use case is not exactly the same - BTQ includes important tie-breaker logic to avoid "favouring the wrong field" when querying >1 field.
FuzzyQuery is single-field only and so we don't have to give terms different DFs - they can all share the same DF and therefore be ranked internally purely on similarity (via the TQ boosts for edit distance). The only DF manipulation required is that they are all rewarded with the highest-DF of any term in the set as this represents the "most likely" interpretation of what all these high-similarity terms are hoped to represent.

I toyed with the idea of retaining some sense of the original terms' DF to blend popularity with similarity scores (to help tune out all those pesky typos). There are 2 issues with this:

  1. It would be hard to blend edit-distance similarity with notions of popularity in a way that scores well for all cases.
  2. It happens too late - I think FQ's term-selection process is pure edit-distance-based and so may have already thrown away popular terms in favour of similar one-off typos by the time rewrite methods get called to tweak any DF settings.
    I think the right answer here is to a add a minDF setting as a filter to FuzzyQuery's term-selection logic to remove the typos.
Contributor

markharwood commented May 1, 2015

I've taken the guts of our BlendedTermQuery and working it back into Lucene's FuzzyQuery rewrite capabilities.
The use case is not exactly the same - BTQ includes important tie-breaker logic to avoid "favouring the wrong field" when querying >1 field.
FuzzyQuery is single-field only and so we don't have to give terms different DFs - they can all share the same DF and therefore be ranked internally purely on similarity (via the TQ boosts for edit distance). The only DF manipulation required is that they are all rewarded with the highest-DF of any term in the set as this represents the "most likely" interpretation of what all these high-similarity terms are hoped to represent.

I toyed with the idea of retaining some sense of the original terms' DF to blend popularity with similarity scores (to help tune out all those pesky typos). There are 2 issues with this:

  1. It would be hard to blend edit-distance similarity with notions of popularity in a way that scores well for all cases.
  2. It happens too late - I think FQ's term-selection process is pure edit-distance-based and so may have already thrown away popular terms in favour of similar one-off typos by the time rewrite methods get called to tweak any DF settings.
    I think the right answer here is to a add a minDF setting as a filter to FuzzyQuery's term-selection logic to remove the typos.
@cedric-marcone

This comment has been minimized.

Show comment
Hide comment
@cedric-marcone

cedric-marcone May 7, 2015

At least, in the documentation, you should provide some migration tips for the users in the "esoteric" clan.

cedric-marcone commented May 7, 2015

At least, in the documentation, you should provide some migration tips for the users in the "esoteric" clan.

@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil May 7, 2015

We stumbled on the thread mentionned by @markharwood a while ago when encountering the exact same issue. We also decided to use FLT to work around #9103.
We have been monitoring #9103 ever since we went with this workaround, as far as I can tell it is not even considered for inclusion in 2.0

My current understanding is that removing FLT without fixing #9103 would effectively render serious uses of fuzzy queries impossible as they would favor very rare fuzzy matches way over frequent exact matches. Unfortunately we lack the detailed knowledge to be able to contribute a pull request.

jeantil commented May 7, 2015

We stumbled on the thread mentionned by @markharwood a while ago when encountering the exact same issue. We also decided to use FLT to work around #9103.
We have been monitoring #9103 ever since we went with this workaround, as far as I can tell it is not even considered for inclusion in 2.0

My current understanding is that removing FLT without fixing #9103 would effectively render serious uses of fuzzy queries impossible as they would favor very rare fuzzy matches way over frequent exact matches. Unfortunately we lack the detailed knowledge to be able to contribute a pull request.

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley May 8, 2015

Member

@markharwood would you be able to look at #9103?

Member

clintongormley commented May 8, 2015

@markharwood would you be able to look at #9103?

@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil May 11, 2015

#9103 is dependant on LUCENE-329, hopefully the patch will be accepted soon and from what i can tell, it would solve our problem. Do you think it would be possible to keep ftl around until LUCENE-329 is closed and elasticsearch has upgraded ?

jeantil commented May 11, 2015

#9103 is dependant on LUCENE-329, hopefully the patch will be accepted soon and from what i can tell, it would solve our problem. Do you think it would be possible to keep ftl around until LUCENE-329 is closed and elasticsearch has upgraded ?

@jpountz jpountz referenced this pull request May 12, 2015

Closed

MLT depricated #11115

@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil May 26, 2015

@markharwood @clintongormley Looks like LUCENE-329 has been merged into Lucene 5.x which should improve things.

In the mean time we have been trying to replace FLT with a multi-match fuzzy query (ignoring relevance until the Lucene move and #9103 are available in a release).

We query on 6 fields in a index with ~4000 documents, with queries ranging from 2 to 5 terms
On the same machine on the same index, we see a massive performance drop.

Given a query of 3 tokens:"audi a 3"

  • With FLT the results are consistently delivered in less than 150 ms
  • With multi-match fuzzy results are consistently delivered in over 10 seconds !! (including when repeating that same query over and over)

If we move to a slightly bigger query with 4 terms of avout 6 letters each, the time scale moves to minutes instead of seconds for the multi-match while the FLT remains in the milliseconds.

best_fields and most_fields both suffer of the performance issue. cross_fields and phrase bring the performances back to about what FLT was (we yet have to study what the relevance implications are as this is much harder to analyse)

Is there a guide or something in the documentation (or planned to be written) on how to properly configure multi-match fuzzy to get results comparable to FLT ? is it even possible ?

jeantil commented May 26, 2015

@markharwood @clintongormley Looks like LUCENE-329 has been merged into Lucene 5.x which should improve things.

In the mean time we have been trying to replace FLT with a multi-match fuzzy query (ignoring relevance until the Lucene move and #9103 are available in a release).

We query on 6 fields in a index with ~4000 documents, with queries ranging from 2 to 5 terms
On the same machine on the same index, we see a massive performance drop.

Given a query of 3 tokens:"audi a 3"

  • With FLT the results are consistently delivered in less than 150 ms
  • With multi-match fuzzy results are consistently delivered in over 10 seconds !! (including when repeating that same query over and over)

If we move to a slightly bigger query with 4 terms of avout 6 letters each, the time scale moves to minutes instead of seconds for the multi-match while the FLT remains in the milliseconds.

best_fields and most_fields both suffer of the performance issue. cross_fields and phrase bring the performances back to about what FLT was (we yet have to study what the relevance implications are as this is much harder to analyse)

Is there a guide or something in the documentation (or planned to be written) on how to properly configure multi-match fuzzy to get results comparable to FLT ? is it even possible ?

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley May 26, 2015

Member

@jeantil

cross_fields and phrase bring the performances back to about what FLT was (we yet have to study what the relevance implications are as this is much harder to analyse)

These types don't use fuzziness, which explains the difference :)

Given a query of 3 tokens:"audi a 3"

  • With FLT the results are consistently delivered in less than 150 ms
  • With multi-match fuzzy results are consistently delivered in over 10 seconds !! (including when repeating that same query over and over)

That's crazy - I'm really surprised you're seeing performance like this. FLT uses the old-style fuzzy queries (see annotation "L" and before here http://people.apache.org/~mikemccand/lucenebench/Fuzzy2.html) while multi-match uses the new FST based fuzzy queries (see http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html).

Two questions:

  • are you using "fuzziness": "AUTO" (which takes word length into account)
  • FSTs use memory so do you have enough heap space available?
Member

clintongormley commented May 26, 2015

@jeantil

cross_fields and phrase bring the performances back to about what FLT was (we yet have to study what the relevance implications are as this is much harder to analyse)

These types don't use fuzziness, which explains the difference :)

Given a query of 3 tokens:"audi a 3"

  • With FLT the results are consistently delivered in less than 150 ms
  • With multi-match fuzzy results are consistently delivered in over 10 seconds !! (including when repeating that same query over and over)

That's crazy - I'm really surprised you're seeing performance like this. FLT uses the old-style fuzzy queries (see annotation "L" and before here http://people.apache.org/~mikemccand/lucenebench/Fuzzy2.html) while multi-match uses the new FST based fuzzy queries (see http://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html).

Two questions:

  • are you using "fuzziness": "AUTO" (which takes word length into account)
  • FSTs use memory so do you have enough heap space available?
@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley May 26, 2015

Member

perhaps @markharwood can shed some light on this?

Member

clintongormley commented May 26, 2015

perhaps @markharwood can shed some light on this?

@markharwood

This comment has been minimized.

Show comment
Hide comment
@markharwood

markharwood May 26, 2015

Contributor

The number of terms you search on (regardless of how they are scored/derived) has a direct impact on performance as they each require a disk seek to get at the postings list. It may be that FLT vs other forms of Fuzzy are just generating different volumes of search terms in these cases?
We'd need example queries and possibly data to dig deeper.

Contributor

markharwood commented May 26, 2015

The number of terms you search on (regardless of how they are scored/derived) has a direct impact on performance as they each require a disk seek to get at the postings list. It may be that FLT vs other forms of Fuzzy are just generating different volumes of search terms in these cases?
We'd need example queries and possibly data to dig deeper.

@markharwood

This comment has been minimized.

Show comment
Hide comment
@markharwood

markharwood May 26, 2015

Contributor

I tried various multi_match and fuzzy settings on wikipedia index with fields containing >10m unique values and couldn't get anything approaching 10 second response times, let alone minutes. This was on SSD.

The question about heap space seems like a good next step.

Contributor

markharwood commented May 26, 2015

I tried various multi_match and fuzzy settings on wikipedia index with fields containing >10m unique values and couldn't get anything approaching 10 second response times, let alone minutes. This was on SSD.

The question about heap space seems like a good next step.

@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil May 26, 2015

@clintongormley yes we use "fuzziness": "AUTO" in both cases. Both tests are executed with the same max heap (1GB as it is) I can increase the max heap but ulitmately wouldn't that mean that on the same hardware our scalability will be severely reduced ?

@markharwood
On a index with 3561 structured documents representing consumer cars hosted on a SSD laptop in an ES instance with Xmx1g.
I won't be able to be much more precise in a public discussion as I am bound by non disclosure agreements, but we can arrange for private conversation if needed.

The query for Audi A3 Sportback Quattro using FLT, "took": 121 and looks like

GET vehicle/cartype/_search
{
  "size" : 100,
  "query" : {
    "bool" : {
      "should" : [ {
        "function_score" : {
          "query" : {
            "flt" : {
              "fields" : [ "field1_search", "field1_keywords", "field2_search", "field2_keywords", "field3_search", "field3_keywords" ],
              "like_text" : "Audi A3 Sportback Quattro",
              "fuzziness" : "AUTO"
            }
          },
          "functions" : [ {
            "field_value_factor" : {
              "field" : "preferred"
            }
          } ]
        }
      }, {
        "function_score" : {
          "query" : {
            "multi_match" : {
              "query" : "Audi A3 Sportback Quattro",
              "fields" : [ "field1_search", "field1_keywords", "field2_search", "field2_keywords", "field3_search", "field3_keywords" ],
              "boost" : 3.0
            }
          },
          "functions" : [ {
            "field_value_factor" : {
              "field" : "preferred"
            }
          } ]
        }
      } ]
    }
  },
  "highlight" : {
    "pre_tags" : [ "{" ],
    "post_tags" : [ "}" ],
    "require_field_match" : false,
    "fields" : {
      "field1_search" : { },
      "field1_keywords" : { },
      "field2_search" : { },
      "field2_keywords" : { },
      "field3_search" : { },
      "field3_keywords" : { }
    }
  }
}

The same query using multi-match fuzzy, "took": 94032 and looks like

GET vehicle/cartype/_search
{
  "size" : 100,
  "query" : {
    "bool" : {
      "should" : [ {       
            "multi_match" : {
              "fields" : [ "field1_search", "field1_keywords", "field2_search", "field2_keywords", "field3_search", "field3_keywords" ],
              "query" : "Audi A3 Sportback Quattro",
              "fuzziness" : "AUTO"
            }
      }, {
            "multi_match" : {
              "query" : "Audi A3 Sportback Quattro",
              "fields" : [ "field1_search", "field1_keywords", "field2_search", "field2_keywords", "field3_search", "field3_keywords" ],
              "boost" : 3.0
            }
      } ]
    }
  },
  "highlight" : {
    "pre_tags" : [ "{" ],
    "post_tags" : [ "}" ],
    "require_field_match" : false,
    "fields" : {
      "field1_search" : { },
      "field1_keywords" : { },
      "field2_search" : { },
      "field2_keywords" : { },
      "field3_search" : { },
      "field3_keywords" : { }
    }
  }
}

jeantil commented May 26, 2015

@clintongormley yes we use "fuzziness": "AUTO" in both cases. Both tests are executed with the same max heap (1GB as it is) I can increase the max heap but ulitmately wouldn't that mean that on the same hardware our scalability will be severely reduced ?

@markharwood
On a index with 3561 structured documents representing consumer cars hosted on a SSD laptop in an ES instance with Xmx1g.
I won't be able to be much more precise in a public discussion as I am bound by non disclosure agreements, but we can arrange for private conversation if needed.

The query for Audi A3 Sportback Quattro using FLT, "took": 121 and looks like

GET vehicle/cartype/_search
{
  "size" : 100,
  "query" : {
    "bool" : {
      "should" : [ {
        "function_score" : {
          "query" : {
            "flt" : {
              "fields" : [ "field1_search", "field1_keywords", "field2_search", "field2_keywords", "field3_search", "field3_keywords" ],
              "like_text" : "Audi A3 Sportback Quattro",
              "fuzziness" : "AUTO"
            }
          },
          "functions" : [ {
            "field_value_factor" : {
              "field" : "preferred"
            }
          } ]
        }
      }, {
        "function_score" : {
          "query" : {
            "multi_match" : {
              "query" : "Audi A3 Sportback Quattro",
              "fields" : [ "field1_search", "field1_keywords", "field2_search", "field2_keywords", "field3_search", "field3_keywords" ],
              "boost" : 3.0
            }
          },
          "functions" : [ {
            "field_value_factor" : {
              "field" : "preferred"
            }
          } ]
        }
      } ]
    }
  },
  "highlight" : {
    "pre_tags" : [ "{" ],
    "post_tags" : [ "}" ],
    "require_field_match" : false,
    "fields" : {
      "field1_search" : { },
      "field1_keywords" : { },
      "field2_search" : { },
      "field2_keywords" : { },
      "field3_search" : { },
      "field3_keywords" : { }
    }
  }
}

The same query using multi-match fuzzy, "took": 94032 and looks like

GET vehicle/cartype/_search
{
  "size" : 100,
  "query" : {
    "bool" : {
      "should" : [ {       
            "multi_match" : {
              "fields" : [ "field1_search", "field1_keywords", "field2_search", "field2_keywords", "field3_search", "field3_keywords" ],
              "query" : "Audi A3 Sportback Quattro",
              "fuzziness" : "AUTO"
            }
      }, {
            "multi_match" : {
              "query" : "Audi A3 Sportback Quattro",
              "fields" : [ "field1_search", "field1_keywords", "field2_search", "field2_keywords", "field3_search", "field3_keywords" ],
              "boost" : 3.0
            }
      } ]
    }
  },
  "highlight" : {
    "pre_tags" : [ "{" ],
    "post_tags" : [ "}" ],
    "require_field_match" : false,
    "fields" : {
      "field1_search" : { },
      "field1_keywords" : { },
      "field2_search" : { },
      "field2_keywords" : { },
      "field3_search" : { },
      "field3_keywords" : { }
    }
  }
}
@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil May 26, 2015

@markharwood Boosting the heap to Xmx4g

 -Xms512m -Xmx4g -Djava.awt.headless=true -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8 -Xss200000 -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/local/Cellar/elasticsearch/1.5.1 -Des.config=/usr/local/Cellar/elasticsearch/1.5.1/config/elasticsearch.yml

On the same query, repeated a few times just to be sure, I get "took": 70884. It is a bit better but nowhere near what we used to get with FLT ...

Forcing the instance to run that same query 5-6 times in // makes ES consume 90% of my quadcore CPU where FLT never rose above backgound CPU usage.

jeantil commented May 26, 2015

@markharwood Boosting the heap to Xmx4g

 -Xms512m -Xmx4g -Djava.awt.headless=true -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8 -Xss200000 -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/local/Cellar/elasticsearch/1.5.1 -Des.config=/usr/local/Cellar/elasticsearch/1.5.1/config/elasticsearch.yml

On the same query, repeated a few times just to be sure, I get "took": 70884. It is a bit better but nowhere near what we used to get with FLT ...

Forcing the instance to run that same query 5-6 times in // makes ES consume 90% of my quadcore CPU where FLT never rose above backgound CPU usage.

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley May 26, 2015

Member

@jeantil are there any messages about slow GC in your logs? Or anything else that is interesting? how much free space do you have on your heap after the fuzzy queries? And could you run GET /_nodes/hot_threads while running the queries?

Member

clintongormley commented May 26, 2015

@jeantil are there any messages about slow GC in your logs? Or anything else that is interesting? how much free space do you have on your heap after the fuzzy queries? And could you run GET /_nodes/hot_threads while running the queries?

@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil May 26, 2015

@clintongormley

I don't get anything in the logs (homebrew's default configuration seems to log at INFO) I restarted my elastic search at 10:28 (GMT) to apply the 4GB Xmx

[2015-05-26 10:28:04,530][INFO ][http                     ] [Helio] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.4.202:9200]}
[2015-05-26 10:28:04,604][INFO ][node                     ] [Helio] started
[2015-05-26 10:28:30,216][INFO ][gateway                  ] [Helio] recovered [79] indices into cluster_state
[2015-05-26 10:28:30,318][INFO ][cluster.metadata         ] [Helio] [.marvel-2015.05.26] creating index, cause [auto(bulk api)], templates [marvel], shards [1]/[1], mappings [_default_, shard_event, index_event, index_stats, node_event, routing_event, cluster_event, cluster_state, cluster_stats, node_stats, indices_stats]
[2015-05-26 10:29:13,719][ERROR][marvel.agent.exporter    ] [Helio] error sending data to [http://[0:0:0:0:0:0:0:0]:9200/.marvel-2015.05.26/_bulk]: SocketTimeoutException[Read timed out]
[2015-05-26 10:29:35,360][DEBUG][action.bulk              ] [Helio] observer timed out. notifying listener. timeout setting [1m], time since start [1m]
[2015-05-26 10:30:14,312][ERROR][marvel.agent.exporter    ] [Helio] error sending data to [http://[0:0:0:0:0:0:0:0]:9200/.marvel-2015.05.26/_bulk]: SocketTimeoutException[Read timed out]
[2015-05-26 10:30:14,710][DEBUG][action.bulk              ] [Helio] observer timed out. notifying listener. timeout setting [1m], time since start [1m]
[2015-05-26 10:30:25,778][INFO ][cluster.metadata         ] [Helio] [.marvel-2015.05.26] update_mapping [index_stats] (dynamic)
[2015-05-26 10:30:26,135][INFO ][cluster.metadata         ] [Helio] [.marvel-2015.05.26] update_mapping [indices_stats] (dynamic)
[2015-05-26 10:30:28,859][INFO ][cluster.metadata         ] [Helio] [.marvel-2015.05.26] update_mapping [cluster_stats] (dynamic)

Here is the GET /_nodes/hot_threads while running one multimatch fuzzy querry

::: [Halloween Jack][DVCm9wTyQWO0m-zus6thUQ][byjean][inet[192.168.4.214/192.168.4.214:9300]]
   Hot threads at 2015-05-26T11:25:10.499Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

   113.4% (567.1ms out of 500ms) cpu usage by thread 'elasticsearch[Halloween Jack][search][T#3]'
     2/10 snapshots sharing following 38 elements
       org.apache.lucene.util.InPlaceMergeSorter.mergeSort(InPlaceMergeSorter.java:40)
       org.apache.lucene.util.InPlaceMergeSorter.mergeSort(InPlaceMergeSorter.java:40)
       org.apache.lucene.util.InPlaceMergeSorter.mergeSort(InPlaceMergeSorter.java:40)
       org.apache.lucene.util.InPlaceMergeSorter.sort(InPlaceMergeSorter.java:32)
       org.apache.lucene.util.automaton.Automaton$Builder.finish(Automaton.java:734)
       org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:311)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:197)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.TopTermsRewrite.rewrite(TopTermsRewrite.java:67)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:222)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.elasticsearch.common.lucene.search.XFilteredQuery.rewrite(XFilteredQuery.java:97)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     3/10 snapshots sharing following 19 elements
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:222)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.elasticsearch.common.lucene.search.XFilteredQuery.rewrite(XFilteredQuery.java:97)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 34 elements
       org.apache.lucene.util.automaton.Automaton.addTransition(Automaton.java:152)
       org.apache.lucene.util.automaton.Automaton.addTransition(Automaton.java:135)
       org.apache.lucene.util.automaton.LevenshteinAutomata.toAutomaton(LevenshteinAutomata.java:199)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:174)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 39 elements
       org.apache.lucene.util.InPlaceMergeSorter.mergeSort(InPlaceMergeSorter.java:41)
       org.apache.lucene.util.InPlaceMergeSorter.mergeSort(InPlaceMergeSorter.java:40)
       org.apache.lucene.util.InPlaceMergeSorter.mergeSort(InPlaceMergeSorter.java:41)
       org.apache.lucene.util.InPlaceMergeSorter.sort(InPlaceMergeSorter.java:32)
       org.apache.lucene.util.automaton.Automaton$Builder.finish(Automaton.java:734)
       org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:311)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:197)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     unique snapshot
       org.apache.lucene.util.automaton.Automaton.addTransition(Automaton.java:152)
       org.apache.lucene.util.automaton.Automaton$Builder.finish(Automaton.java:737)
       org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:311)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:197)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)

Here is the GET /_nodes/hot_threads while running 6 multimatch fuzzy querry

::: [Halloween Jack][DVCm9wTyQWO0m-zus6thUQ][byjean][inet[192.168.4.214/192.168.4.214:9300]]
   Hot threads at 2015-05-26T11:29:24.436Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

   113.3% (566.5ms out of 500ms) cpu usage by thread 'elasticsearch[Halloween Jack][search][T#15]'
     2/10 snapshots sharing following 36 elements
       org.apache.lucene.util.InPlaceMergeSorter.sort(InPlaceMergeSorter.java:32)
       org.apache.lucene.util.automaton.Automaton$Builder.finish(Automaton.java:734)
       org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:311)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:197)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     4/10 snapshots sharing following 30 elements
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 37 elements
       java.util.HashMap.put(HashMap.java:611)
       java.util.HashSet.add(HashSet.java:219)
       org.apache.lucene.util.automaton.Automaton.getStartPoints(Automaton.java:575)
       org.apache.lucene.util.automaton.RunAutomaton.<init>(RunAutomaton.java:140)
       org.apache.lucene.util.automaton.ByteRunAutomaton.<init>(ByteRunAutomaton.java:32)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:203)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.TopTermsRewrite.rewrite(TopTermsRewrite.java:67)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:222)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.elasticsearch.common.lucene.search.XFilteredQuery.rewrite(XFilteredQuery.java:97)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 35 elements
       org.apache.lucene.util.InPlaceMergeSorter.sort(InPlaceMergeSorter.java:32)
       org.apache.lucene.util.automaton.Automaton$Builder.finish(Automaton.java:734)
       org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:311)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:197)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.TopTermsRewrite.rewrite(TopTermsRewrite.java:67)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:222)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.elasticsearch.common.lucene.search.XFilteredQuery.rewrite(XFilteredQuery.java:97)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)

   112.8% (563.8ms out of 500ms) cpu usage by thread 'elasticsearch[Halloween Jack][search][T#11]'
     4/10 snapshots sharing following 34 elements
       org.apache.lucene.util.automaton.ByteRunAutomaton.<init>(ByteRunAutomaton.java:32)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:203)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 35 elements
       org.apache.lucene.util.InPlaceMergeSorter.sort(InPlaceMergeSorter.java:32)
       org.apache.lucene.util.automaton.Automaton$Builder.finish(Automaton.java:734)
       org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:311)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:197)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.TopTermsRewrite.rewrite(TopTermsRewrite.java:67)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:222)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.elasticsearch.common.lucene.search.XFilteredQuery.rewrite(XFilteredQuery.java:97)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     4/10 snapshots sharing following 14 elements
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)

   112.7% (563.4ms out of 500ms) cpu usage by thread 'elasticsearch[Halloween Jack][search][T#6]'
     2/10 snapshots sharing following 32 elements
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 33 elements
       org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:311)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:197)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.TopTermsRewrite.rewrite(TopTermsRewrite.java:67)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:222)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.elasticsearch.common.lucene.search.XFilteredQuery.rewrite(XFilteredQuery.java:97)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 34 elements
       org.apache.lucene.util.automaton.Automaton.addTransition(Automaton.java:152)
       org.apache.lucene.util.automaton.Automaton.addTransition(Automaton.java:135)
       org.apache.lucene.util.automaton.LevenshteinAutomata.toAutomaton(LevenshteinAutomata.java:199)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:174)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 30 elements
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:174)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.TopTermsRewrite.rewrite(TopTermsRewrite.java:67)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:222)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.elasticsearch.common.lucene.search.XFilteredQuery.rewrite(XFilteredQuery.java:97)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 14 elements
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)

and here is a jmap -heap of the VM while running the aforementionned 6 queries

jmap -heap 1298
Attaching to process ID 1298, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.40-b25

using thread-local object allocation.
Garbage-First (G1) GC with 8 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 4294967296 (4096.0MB)
   NewSize                  = 1363144 (1.2999954223632812MB)
   MaxNewSize               = 2576351232 (2457.0MB)
   OldSize                  = 5452592 (5.1999969482421875MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 1048576 (1.0MB)

Heap Usage:
G1 Heap:
   regions  = 4096
   capacity = 4294967296 (4096.0MB)
   used     = 1371856408 (1308.3042221069336MB)
   free     = 2923110888 (2787.6957778930664MB)
   31.94102104753256% used
G1 Young Generation:
Eden Space:
   regions  = 560
   capacity = 2275409920 (2170.0MB)
   used     = 587202560 (560.0MB)
   free     = 1688207360 (1610.0MB)
   25.806451612903224% used
Survivor Space:
   regions  = 5
   capacity = 5242880 (5.0MB)
   used     = 5242880 (5.0MB)
   free     = 0 (0.0MB)
   100.0% used
G1 Old Generation:
   regions  = 745
   capacity = 1340080128 (1278.0MB)
   used     = 779410968 (743.3042221069336MB)
   free     = 560669160 (534.6957778930664MB)
   58.16151972667712% used

16084 interned Strings occupying 2325440 bytes.

jeantil commented May 26, 2015

@clintongormley

I don't get anything in the logs (homebrew's default configuration seems to log at INFO) I restarted my elastic search at 10:28 (GMT) to apply the 4GB Xmx

[2015-05-26 10:28:04,530][INFO ][http                     ] [Helio] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.4.202:9200]}
[2015-05-26 10:28:04,604][INFO ][node                     ] [Helio] started
[2015-05-26 10:28:30,216][INFO ][gateway                  ] [Helio] recovered [79] indices into cluster_state
[2015-05-26 10:28:30,318][INFO ][cluster.metadata         ] [Helio] [.marvel-2015.05.26] creating index, cause [auto(bulk api)], templates [marvel], shards [1]/[1], mappings [_default_, shard_event, index_event, index_stats, node_event, routing_event, cluster_event, cluster_state, cluster_stats, node_stats, indices_stats]
[2015-05-26 10:29:13,719][ERROR][marvel.agent.exporter    ] [Helio] error sending data to [http://[0:0:0:0:0:0:0:0]:9200/.marvel-2015.05.26/_bulk]: SocketTimeoutException[Read timed out]
[2015-05-26 10:29:35,360][DEBUG][action.bulk              ] [Helio] observer timed out. notifying listener. timeout setting [1m], time since start [1m]
[2015-05-26 10:30:14,312][ERROR][marvel.agent.exporter    ] [Helio] error sending data to [http://[0:0:0:0:0:0:0:0]:9200/.marvel-2015.05.26/_bulk]: SocketTimeoutException[Read timed out]
[2015-05-26 10:30:14,710][DEBUG][action.bulk              ] [Helio] observer timed out. notifying listener. timeout setting [1m], time since start [1m]
[2015-05-26 10:30:25,778][INFO ][cluster.metadata         ] [Helio] [.marvel-2015.05.26] update_mapping [index_stats] (dynamic)
[2015-05-26 10:30:26,135][INFO ][cluster.metadata         ] [Helio] [.marvel-2015.05.26] update_mapping [indices_stats] (dynamic)
[2015-05-26 10:30:28,859][INFO ][cluster.metadata         ] [Helio] [.marvel-2015.05.26] update_mapping [cluster_stats] (dynamic)

Here is the GET /_nodes/hot_threads while running one multimatch fuzzy querry

::: [Halloween Jack][DVCm9wTyQWO0m-zus6thUQ][byjean][inet[192.168.4.214/192.168.4.214:9300]]
   Hot threads at 2015-05-26T11:25:10.499Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

   113.4% (567.1ms out of 500ms) cpu usage by thread 'elasticsearch[Halloween Jack][search][T#3]'
     2/10 snapshots sharing following 38 elements
       org.apache.lucene.util.InPlaceMergeSorter.mergeSort(InPlaceMergeSorter.java:40)
       org.apache.lucene.util.InPlaceMergeSorter.mergeSort(InPlaceMergeSorter.java:40)
       org.apache.lucene.util.InPlaceMergeSorter.mergeSort(InPlaceMergeSorter.java:40)
       org.apache.lucene.util.InPlaceMergeSorter.sort(InPlaceMergeSorter.java:32)
       org.apache.lucene.util.automaton.Automaton$Builder.finish(Automaton.java:734)
       org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:311)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:197)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.TopTermsRewrite.rewrite(TopTermsRewrite.java:67)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:222)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.elasticsearch.common.lucene.search.XFilteredQuery.rewrite(XFilteredQuery.java:97)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     3/10 snapshots sharing following 19 elements
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:222)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.elasticsearch.common.lucene.search.XFilteredQuery.rewrite(XFilteredQuery.java:97)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 34 elements
       org.apache.lucene.util.automaton.Automaton.addTransition(Automaton.java:152)
       org.apache.lucene.util.automaton.Automaton.addTransition(Automaton.java:135)
       org.apache.lucene.util.automaton.LevenshteinAutomata.toAutomaton(LevenshteinAutomata.java:199)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:174)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 39 elements
       org.apache.lucene.util.InPlaceMergeSorter.mergeSort(InPlaceMergeSorter.java:41)
       org.apache.lucene.util.InPlaceMergeSorter.mergeSort(InPlaceMergeSorter.java:40)
       org.apache.lucene.util.InPlaceMergeSorter.mergeSort(InPlaceMergeSorter.java:41)
       org.apache.lucene.util.InPlaceMergeSorter.sort(InPlaceMergeSorter.java:32)
       org.apache.lucene.util.automaton.Automaton$Builder.finish(Automaton.java:734)
       org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:311)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:197)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     unique snapshot
       org.apache.lucene.util.automaton.Automaton.addTransition(Automaton.java:152)
       org.apache.lucene.util.automaton.Automaton$Builder.finish(Automaton.java:737)
       org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:311)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:197)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)

Here is the GET /_nodes/hot_threads while running 6 multimatch fuzzy querry

::: [Halloween Jack][DVCm9wTyQWO0m-zus6thUQ][byjean][inet[192.168.4.214/192.168.4.214:9300]]
   Hot threads at 2015-05-26T11:29:24.436Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

   113.3% (566.5ms out of 500ms) cpu usage by thread 'elasticsearch[Halloween Jack][search][T#15]'
     2/10 snapshots sharing following 36 elements
       org.apache.lucene.util.InPlaceMergeSorter.sort(InPlaceMergeSorter.java:32)
       org.apache.lucene.util.automaton.Automaton$Builder.finish(Automaton.java:734)
       org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:311)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:197)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     4/10 snapshots sharing following 30 elements
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 37 elements
       java.util.HashMap.put(HashMap.java:611)
       java.util.HashSet.add(HashSet.java:219)
       org.apache.lucene.util.automaton.Automaton.getStartPoints(Automaton.java:575)
       org.apache.lucene.util.automaton.RunAutomaton.<init>(RunAutomaton.java:140)
       org.apache.lucene.util.automaton.ByteRunAutomaton.<init>(ByteRunAutomaton.java:32)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:203)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.TopTermsRewrite.rewrite(TopTermsRewrite.java:67)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:222)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.elasticsearch.common.lucene.search.XFilteredQuery.rewrite(XFilteredQuery.java:97)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 35 elements
       org.apache.lucene.util.InPlaceMergeSorter.sort(InPlaceMergeSorter.java:32)
       org.apache.lucene.util.automaton.Automaton$Builder.finish(Automaton.java:734)
       org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:311)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:197)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.TopTermsRewrite.rewrite(TopTermsRewrite.java:67)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:222)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.elasticsearch.common.lucene.search.XFilteredQuery.rewrite(XFilteredQuery.java:97)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)

   112.8% (563.8ms out of 500ms) cpu usage by thread 'elasticsearch[Halloween Jack][search][T#11]'
     4/10 snapshots sharing following 34 elements
       org.apache.lucene.util.automaton.ByteRunAutomaton.<init>(ByteRunAutomaton.java:32)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:203)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 35 elements
       org.apache.lucene.util.InPlaceMergeSorter.sort(InPlaceMergeSorter.java:32)
       org.apache.lucene.util.automaton.Automaton$Builder.finish(Automaton.java:734)
       org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:311)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:197)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.TopTermsRewrite.rewrite(TopTermsRewrite.java:67)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:222)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.elasticsearch.common.lucene.search.XFilteredQuery.rewrite(XFilteredQuery.java:97)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     4/10 snapshots sharing following 14 elements
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)

   112.7% (563.4ms out of 500ms) cpu usage by thread 'elasticsearch[Halloween Jack][search][T#6]'
     2/10 snapshots sharing following 32 elements
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 33 elements
       org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:311)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:197)
       org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:104)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:176)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.TopTermsRewrite.rewrite(TopTermsRewrite.java:67)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:222)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.elasticsearch.common.lucene.search.XFilteredQuery.rewrite(XFilteredQuery.java:97)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 34 elements
       org.apache.lucene.util.automaton.Automaton.addTransition(Automaton.java:152)
       org.apache.lucene.util.automaton.Automaton.addTransition(Automaton.java:135)
       org.apache.lucene.util.automaton.LevenshteinAutomata.toAutomaton(LevenshteinAutomata.java:199)
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:174)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:105)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:155)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:99)
       org.elasticsearch.search.highlight.CustomQueryScorer$CustomWeightedSpanTermExtractor.extractUnknownQuery(CustomQueryScorer.java:89)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:224)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 30 elements
       org.apache.lucene.search.FuzzyTermsEnum.initAutomata(FuzzyTermsEnum.java:174)
       org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(FuzzyTermsEnum.java:152)
       org.apache.lucene.search.FuzzyTermsEnum.maxEditDistanceChanged(FuzzyTermsEnum.java:211)
       org.apache.lucene.search.FuzzyTermsEnum.bottomChanged(FuzzyTermsEnum.java:205)
       org.apache.lucene.search.FuzzyTermsEnum.<init>(FuzzyTermsEnum.java:143)
       org.apache.lucene.search.FuzzyQuery.getTermsEnum(FuzzyQuery.java:155)
       org.apache.lucene.search.MultiTermQuery$RewriteMethod.getTermsEnum(MultiTermQuery.java:76)
       org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:64)
       org.apache.lucene.search.TopTermsRewrite.rewrite(TopTermsRewrite.java:67)
       org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.apache.lucene.search.DisjunctionMaxQuery.rewrite(DisjunctionMaxQuery.java:222)
       org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:554)
       org.elasticsearch.common.lucene.search.XFilteredQuery.rewrite(XFilteredQuery.java:97)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:217)
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)
     2/10 snapshots sharing following 14 elements
       org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:474)
       org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:217)
       org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:186)
       org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:197)
       org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118)
       org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:192)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:386)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
       org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
       org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       java.lang.Thread.run(Thread.java:745)

and here is a jmap -heap of the VM while running the aforementionned 6 queries

jmap -heap 1298
Attaching to process ID 1298, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.40-b25

using thread-local object allocation.
Garbage-First (G1) GC with 8 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 4294967296 (4096.0MB)
   NewSize                  = 1363144 (1.2999954223632812MB)
   MaxNewSize               = 2576351232 (2457.0MB)
   OldSize                  = 5452592 (5.1999969482421875MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 1048576 (1.0MB)

Heap Usage:
G1 Heap:
   regions  = 4096
   capacity = 4294967296 (4096.0MB)
   used     = 1371856408 (1308.3042221069336MB)
   free     = 2923110888 (2787.6957778930664MB)
   31.94102104753256% used
G1 Young Generation:
Eden Space:
   regions  = 560
   capacity = 2275409920 (2170.0MB)
   used     = 587202560 (560.0MB)
   free     = 1688207360 (1610.0MB)
   25.806451612903224% used
Survivor Space:
   regions  = 5
   capacity = 5242880 (5.0MB)
   used     = 5242880 (5.0MB)
   free     = 0 (0.0MB)
   100.0% used
G1 Old Generation:
   regions  = 745
   capacity = 1340080128 (1278.0MB)
   used     = 779410968 (743.3042221069336MB)
   free     = 560669160 (534.6957778930664MB)
   58.16151972667712% used

16084 interned Strings occupying 2325440 bytes.
@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley May 26, 2015

Member

Hmm could you compare the results without G1GC, and (separately) without highlighting?

Member

clintongormley commented May 26, 2015

Hmm could you compare the results without G1GC, and (separately) without highlighting?

@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil May 26, 2015

Reverting G1GC doesn't improve things (meaning the time scale is still the minute, the exact timings )
here is the corresponding jmap when running 6 queries (warmed up by running the query twice before)

jmap -heap 2689
Attaching to process ID 2689, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.40-b25

using parallel threads in the new generation.
using thread-local object allocation.
Concurrent Mark-Sweep GC

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 4294967296 (4096.0MB)
   NewSize                  = 178913280 (170.625MB)
   MaxNewSize               = 697892864 (665.5625MB)
   OldSize                  = 357957632 (341.375MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
New Generation (Eden + 1 Survivor Space):
   capacity = 161021952 (153.5625MB)
   used     = 141274096 (134.72947692871094MB)
   free     = 19747856 (18.833023071289062MB)
   87.73592311189968% used
Eden Space:
   capacity = 143130624 (136.5MB)
   used     = 140167176 (133.67383575439453MB)
   free     = 2963448 (2.8261642456054688MB)
   97.92955000321943% used
From Space:
   capacity = 17891328 (17.0625MB)
   used     = 1106920 (1.0556411743164062MB)
   free     = 16784408 (16.006858825683594MB)
   6.186907981341575% used
To Space:
   capacity = 17891328 (17.0625MB)
   used     = 0 (0.0MB)
   free     = 17891328 (17.0625MB)
   0.0% used
concurrent mark-sweep generation:
   capacity = 1103278080 (1052.16796875MB)
   used     = 4068346332679738616 (3.8798774077222236E12MB)
   free     = 13712308637745 MB
   3.6875076251671185E11% used

15542 interned Strings occupying 2273448 bytes.

jeantil commented May 26, 2015

Reverting G1GC doesn't improve things (meaning the time scale is still the minute, the exact timings )
here is the corresponding jmap when running 6 queries (warmed up by running the query twice before)

jmap -heap 2689
Attaching to process ID 2689, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.40-b25

using parallel threads in the new generation.
using thread-local object allocation.
Concurrent Mark-Sweep GC

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 4294967296 (4096.0MB)
   NewSize                  = 178913280 (170.625MB)
   MaxNewSize               = 697892864 (665.5625MB)
   OldSize                  = 357957632 (341.375MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
New Generation (Eden + 1 Survivor Space):
   capacity = 161021952 (153.5625MB)
   used     = 141274096 (134.72947692871094MB)
   free     = 19747856 (18.833023071289062MB)
   87.73592311189968% used
Eden Space:
   capacity = 143130624 (136.5MB)
   used     = 140167176 (133.67383575439453MB)
   free     = 2963448 (2.8261642456054688MB)
   97.92955000321943% used
From Space:
   capacity = 17891328 (17.0625MB)
   used     = 1106920 (1.0556411743164062MB)
   free     = 16784408 (16.006858825683594MB)
   6.186907981341575% used
To Space:
   capacity = 17891328 (17.0625MB)
   used     = 0 (0.0MB)
   free     = 17891328 (17.0625MB)
   0.0% used
concurrent mark-sweep generation:
   capacity = 1103278080 (1052.16796875MB)
   used     = 4068346332679738616 (3.8798774077222236E12MB)
   free     = 13712308637745 MB
   3.6875076251671185E11% used

15542 interned Strings occupying 2273448 bytes.
@markharwood

This comment has been minimized.

Show comment
Hide comment
@markharwood

markharwood May 26, 2015

Contributor

On a index with 3561 structured documents

Given the tiny number of documents and the time taken to query I wonder if you have some documents with massive numbers of unique terms? Try the cardinality aggregation on the various fields using a match_all query to get some sense of number of unique terms in your index

Contributor

markharwood commented May 26, 2015

On a index with 3561 structured documents

Given the tiny number of documents and the time taken to query I wonder if you have some documents with massive numbers of unique terms? Try the cardinality aggregation on the various fields using a match_all query to get some sense of number of unique terms in your index

@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil May 26, 2015

@clintongormley good call : the highlighting seems to be the culprit. removing the highlighting query time drops back to the 100 milliseconds scale. This is a big problem for us though as we use highlighting extensively to further process the results.

@markharwood
cardinalities :
field1_search : 1631
field1_keywords : 3401
field2_search : 950
field2_keywords : 983
field3_search : 93
field3_keywords : 88

jeantil commented May 26, 2015

@clintongormley good call : the highlighting seems to be the culprit. removing the highlighting query time drops back to the 100 milliseconds scale. This is a big problem for us though as we use highlighting extensively to further process the results.

@markharwood
cardinalities :
field1_search : 1631
field1_keywords : 3401
field2_search : 950
field2_keywords : 983
field3_search : 93
field3_keywords : 88

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley May 26, 2015

Member

@jeantil good to know. which highlighter are you using?

Member

clintongormley commented May 26, 2015

@jeantil good to know. which highlighter are you using?

@markharwood

This comment has been minimized.

Show comment
Hide comment
@markharwood

markharwood May 26, 2015

Contributor

Stack trace says PlainHighlighter

Contributor

markharwood commented May 26, 2015

Stack trace says PlainHighlighter

@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil May 26, 2015

I was searching and it seems we haven't configured anything specific pertaining to the highlighter. I guess plain is the default one.

jeantil commented May 26, 2015

I was searching and it seems we haven't configured anything specific pertaining to the highlighter. I guess plain is the default one.

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley May 26, 2015

Member

I wonder if it is worth trying the postings highlighter or the FVH? Just to see if it makes a difference?

Member

clintongormley commented May 26, 2015

I wonder if it is worth trying the postings highlighter or the FVH? Just to see if it makes a difference?

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley May 26, 2015

Member

@jeantil i think the other big difference is that FLT limits the number of fuzzy terms quite drastically. Try using max_expansions with the multi-match query and see if that helps with the highlighting too.

In the current implementation, it'd just be the first max_expansions terms that get added, but in ES 2, it'll be the top-scoring max_expansions terms (if i read the patch correctly)

Member

clintongormley commented May 26, 2015

@jeantil i think the other big difference is that FLT limits the number of fuzzy terms quite drastically. Try using max_expansions with the multi-match query and see if that helps with the highlighting too.

In the current implementation, it'd just be the first max_expansions terms that get added, but in ES 2, it'll be the top-scoring max_expansions terms (if i read the patch correctly)

@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil May 26, 2015

what would be the value of max_expansions equivalent to FLT ? (just as a starting point for our investigation)

(i am reindexing with index_options: offsets to see if the postings higlighter behaves differently)

jeantil commented May 26, 2015

what would be the value of max_expansions equivalent to FLT ? (just as a starting point for our investigation)

(i am reindexing with index_options: offsets to see if the postings higlighter behaves differently)

@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil May 26, 2015

both postings and fvh hightlighting are in the 200ms range (about twice as no highlighting).

jeantil commented May 26, 2015

both postings and fvh hightlighting are in the 200ms range (about twice as no highlighting).

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley
Member

clintongormley commented May 26, 2015

@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil May 26, 2015

max_expansions seems to have little or no impact on query time. Using plain higlighter with "max_expansions":1 yields 80s for the query we have been discussing all along. Using postings with "max_expansion": 100 remains within the 200ms range.

The plain higlighter seems to have a problem with multimatch fuzzy queries ...

jeantil commented May 26, 2015

max_expansions seems to have little or no impact on query time. Using plain higlighter with "max_expansions":1 yields 80s for the query we have been discussing all along. Using postings with "max_expansion": 100 remains within the 200ms range.

The plain higlighter seems to have a problem with multimatch fuzzy queries ...

@markharwood

This comment has been minimized.

Show comment
Hide comment
@markharwood

markharwood Jun 1, 2015

Contributor

Having looked at the queries/index this bad performance is the result of several linear costs (num docs being highlighted x num fields being highlighted x num terms).
This is 100 docs x 8 fields x ? terms. The hotspots are creating automatons:

  org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(int, BytesRef) 

..and this is done for each of the terms with fuzzy values. Ultimately the Highlighter wants to get a list of terms to highlight and it looks like there may be some inefficiencies as the same user query is "re-written" 800 times (100 docs x 8 fields) in order to get this list of terms for each field that needs highlighting in each doc.
For now, anything you can do to reduce the number of docs or fields being highlighted will have a direct and linear impact on performance.

Contributor

markharwood commented Jun 1, 2015

Having looked at the queries/index this bad performance is the result of several linear costs (num docs being highlighted x num fields being highlighted x num terms).
This is 100 docs x 8 fields x ? terms. The hotspots are creating automatons:

  org.apache.lucene.search.FuzzyTermsEnum.getAutomatonEnum(int, BytesRef) 

..and this is done for each of the terms with fuzzy values. Ultimately the Highlighter wants to get a list of terms to highlight and it looks like there may be some inefficiencies as the same user query is "re-written" 800 times (100 docs x 8 fields) in order to get this list of terms for each field that needs highlighting in each doc.
For now, anything you can do to reduce the number of docs or fields being highlighted will have a direct and linear impact on performance.

@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil Jun 1, 2015

@markharwood, thanks for the analysis. Do you have any idea

  • why we don't have the problem when using FLT + plain highlighter ?
  • why we don't have the problem when using postings and fvh highlighters ?

For now we are still using FLT since the lucene patch and corresponding elasticsearch modifications to make multimatch fuzzy behave better are planned for 2.0.0 (#9103) but we have moved our index to use fvh in anticipation of moving to multimatch fuzzy

thanks

jeantil commented Jun 1, 2015

@markharwood, thanks for the analysis. Do you have any idea

  • why we don't have the problem when using FLT + plain highlighter ?
  • why we don't have the problem when using postings and fvh highlighters ?

For now we are still using FLT since the lucene patch and corresponding elasticsearch modifications to make multimatch fuzzy behave better are planned for 2.0.0 (#9103) but we have moved our index to use fvh in anticipation of moving to multimatch fuzzy

thanks

@markharwood

This comment has been minimized.

Show comment
Hide comment
@markharwood

markharwood Jun 1, 2015

Contributor
  1. FLT query instances look to cache any rewritten expression of itself so will only do max of 1 rewrite() as opposed to 800.
  2. Postings/FVH are different highlighter implementations so get their list of query terms o be highlighted in a different manner.

I'm looking into why we need 800 rewrites not just 1 for the non FLT queries.

Contributor

markharwood commented Jun 1, 2015

  1. FLT query instances look to cache any rewritten expression of itself so will only do max of 1 rewrite() as opposed to 800.
  2. Postings/FVH are different highlighter implementations so get their list of query terms o be highlighted in a different manner.

I'm looking into why we need 800 rewrites not just 1 for the non FLT queries.

@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil Jun 1, 2015

That's very clear, thanks

jeantil commented Jun 1, 2015

That's very clear, thanks

@clintongormley clintongormley changed the title from Queries: Remove fuzzy-like-this support. to Remove `fuzzy_like_this` query Jun 6, 2015

im-denisenko added a commit to im-denisenko/Elastica that referenced this pull request Sep 17, 2015

im-denisenko added a commit to im-denisenko/Elastica that referenced this pull request Sep 17, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment