find-symbol leads to OutOfMemoryError

[mbuczko]

Expected behavior

find-symbol should return list of symbol occurences

Actual behavior

In some cases find-symbol ends up enormous memory consumption resulting in OutOfMemoryError. Partial results are returned.

This issue may be a duplicate of #245

Following is the stack trace:

#error {
 :cause Java heap space
 :via
 [{:type java.lang.OutOfMemoryError
   :message Java heap space
   :at [java.util.Arrays copyOf Arrays.java 3746]}]
 :trace
 [[java.util.Arrays copyOf Arrays.java 3746]
  [java.lang.AbstractStringBuilder ensureCapacityInternal AbstractStringBuilder.java 227]
  [java.lang.AbstractStringBuilder append AbstractStringBuilder.java 593]
  [java.lang.StringBuffer append StringBuffer.java 306]
  [java.io.StringWriter write StringWriter.java 106]
  [clojure.core$fn__7280 invokeStatic core_print.clj 124]
  [clojure.core$fn__7280 invoke core_print.clj 123]
  [clojure.lang.MultiFn invoke MultiFn.java 234]
  [clojure.core$pr_on invokeStatic core.clj 3674]
  [clojure.core$pr_on invoke core.clj 3668]
  [clojure.core$print_prefix_map$fn__7332 invoke core_print.clj 233]
  [clojure.core$print_sequential invokeStatic core_print.clj 66]
  [clojure.core$print_prefix_map invokeStatic core_print.clj 229]
  [clojure.core$print_map invokeStatic core_print.clj 238]
  [clojure.core$fn__7359 invokeStatic core_print.clj 266]
  [clojure.core$fn__7359 invoke core_print.clj 263]
  [clojure.lang.MultiFn invoke MultiFn.java 234]
  [clojure.core$pr_on invokeStatic core.clj 3674]
  [clojure.core$pr_on invoke core.clj 3668]
  [clojure.core$print_prefix_map$fn__7332 invoke core_print.clj 233]
  [clojure.core$print_sequential invokeStatic core_print.clj 66]
  [clojure.core$print_prefix_map invokeStatic core_print.clj 229]
  [clojure.core$print_map invokeStatic core_print.clj 238]
  [clojure.core$fn__7359 invokeStatic core_print.clj 266]
  [clojure.core$fn__7359 invoke core_print.clj 263]
  [clojure.lang.MultiFn invoke MultiFn.java 234]
  [clojure.core$pr_on invokeStatic core.clj 3674]
  [clojure.core$pr_on invoke core.clj 3668]
  [clojure.core$print_prefix_map$fn__7332 invoke core_print.clj 233]
  [clojure.core$print_sequential invokeStatic core_print.clj 66]
  [clojure.core$print_prefix_map invokeStatic core_print.clj 229]
  [clojure.core$print_map invokeStatic core_print.clj 238]]}

This is memory usage right before calling a find-symbol:

and here is how memory consumption looks like during- and after find-symbol:

[mbuczko]

After a couple of days with profiler I finally found a culprit:

refactor-nrepl/src/refactor_nrepl/find/find_symbol.clj

Line 27 in 75f5255

(str/join "/" (remove nil? [full-class (:field node)]))))

It looks quite innocent, but the problem is what full-class may really be. It's usually a string (or something easily casted to string) although in certain cases it might be also a huge map (I guest AST node) whichstr/join is not able easily serialize to string. Looks like failing "stringification" leads to large memory leaks here.

I prepared already a fix already which eliminates a leak and keeps memory consuption sane:

[benedekfazekas]

this is an amazing analysis. thanks!