Leiningen chokes with Vaadin 14 dependencies with OutOfMemoryError exception

[sanel]

Initial debugging steps
Before creating a report, especially around exceptions being thrown when running Leiningen, please check if the error still occurs after:

• Updating to using the latest released version of Leiningen (lein upgrade).
• Moving your ~/.lein/profiles.clj (if present) out of the way. This contains third-party dependencies and plugins that can cause problems inside Leiningen.
• Updating any old versions of plugins in your project.clj, especially if the problem is with a plugin not working. Old versions of plugins like nREPL and CIDER (as well as others) can cause problems with newer versions of Leiningen.
• (If you are using Java 9 or newer), updating your dependencies to their most recent versions. Recent JDK's have introduced changes which can break some Clojure libraries.

Describe the bug
When I try to pull Vaadin 14 libraries, by using [com.vaadin/vaadin "14.6.8"] dependency, lein will choke and quit with OutOfMemoryError error.

This happens with Vaadin 14. Pulling dependencies for previous versions (Vaadin 6,7,8) works without problems.

To Reproduce

1. Clone demo repository: https://github.com/sanel/vaadin14-lein-problem, it has README with details.
2. Run lein deps in it. It will throw OutOfMemoryError exception, at some point.
3. Repeat that with maven or tools.deps. It works correctly.

Actual behavior
Leiningen will start downloading dependencies and, at some point, stuck with CPU to 100%. After some time it will throw this:

Execution error (OutOfMemoryError) at java.lang.reflect.Method/copy
(Method.java:153). GC overhead limit exceeded

Expected behavior
Leiningen should pull all dependencies without problems.

Link to sample project
https://github.com/sanel/vaadin14-lein-problem

Logs
https://github.com/sanel/vaadin14-lein-problem/blob/master/error-trace.edn

Environment

• Leiningen Version: 2.9.6
• Leiningen installation method: manual
• JDK Version: OpenJDK 1.8.0_292 and 15.0.2
• OS: Slackware 14.2

[glts]

Somehow this reminds me of #1914 ...

[sanel]

Thanks @glts! Setting :pedantic? false, "fixed" it. I've looked quickly at leiningen.core.pedantic/all-paths and added some pprints. Here is the number of paths (just calling (count paths) for every new iteration):

 1
 4
 10
 158
 2514
 46348

Eventually it gets so large, that OutOfMemoryError is thrown.

Down below is excerpt of (pprint paths), and as you can see, it will iterate different versions of vaadin-lumo-styles with the same parents. I'm not very much familiar how this should work, and maybe this will be very dumb question, but shouldn't it store only latest versions and drop nodes with the older versions?

{:node
  #object[org.eclipse.aether.graph.DefaultDependencyNode 0x446b64b3 "org.webjars.bowergithub.vaadin:vaadin-lumo-styles:jar:1.3.2 (compile)"],
  :parents
  [#object[org.eclipse.aether.graph.DefaultDependencyNode 0x14fc9bd "null"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x5e5ddfbc "com.vaadin:vaadin:jar:14.6.8 (compile)"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x5bda157e "com.vaadin:vaadin-core:jar:14.6.8 (compile)"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x527937d0 "com.vaadin:vaadin-avatar-flow:jar:14.6.8 (compile)"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x59d6642a "org.webjars.bowergithub.vaadin:vaadin-list-box:jar:1.3.0 (compile)"]]}
 {:node
  #object[org.eclipse.aether.graph.DefaultDependencyNode 0x35ac9ebd "org.webjars.bowergithub.vaadin:vaadin-lumo-styles:jar:1.3.3 (compile)"],
  :parents
  [#object[org.eclipse.aether.graph.DefaultDependencyNode 0x14fc9bd "null"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x5e5ddfbc "com.vaadin:vaadin:jar:14.6.8 (compile)"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x5bda157e "com.vaadin:vaadin-core:jar:14.6.8 (compile)"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x527937d0 "com.vaadin:vaadin-avatar-flow:jar:14.6.8 (compile)"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x59d6642a "org.webjars.bowergithub.vaadin:vaadin-list-box:jar:1.3.0 (compile)"]]}
 {:node
  #object[org.eclipse.aether.graph.DefaultDependencyNode 0x56c0a61e "org.webjars.bowergithub.vaadin:vaadin-lumo-styles:jar:1.4.0 (compile)"],
  :parents
  [#object[org.eclipse.aether.graph.DefaultDependencyNode 0x14fc9bd "null"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x5e5ddfbc "com.vaadin:vaadin:jar:14.6.8 (compile)"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x5bda157e "com.vaadin:vaadin-core:jar:14.6.8 (compile)"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x527937d0 "com.vaadin:vaadin-avatar-flow:jar:14.6.8 (compile)"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x59d6642a "org.webjars.bowergithub.vaadin:vaadin-list-box:jar:1.3.0 (compile)"]]}
 {:node
  #object[org.eclipse.aether.graph.DefaultDependencyNode 0x421ead7e "org.webjars.bowergithub.vaadin:vaadin-lumo-styles:jar:1.4.1 (compile)"],
  :parents
  [#object[org.eclipse.aether.graph.DefaultDependencyNode 0x14fc9bd "null"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x5e5ddfbc "com.vaadin:vaadin:jar:14.6.8 (compile)"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x5bda157e "com.vaadin:vaadin-core:jar:14.6.8 (compile)"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x527937d0 "com.vaadin:vaadin-avatar-flow:jar:14.6.8 (compile)"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x59d6642a "org.webjars.bowergithub.vaadin:vaadin-list-box:jar:1.3.0 (compile)"]]}
 {:node
  #object[org.eclipse.aether.graph.DefaultDependencyNode 0x5dcf0772 "org.webjars.bowergithub.vaadin:vaadin-lumo-styles:jar:1.4.2 (compile)"],
  :parents
  [#object[org.eclipse.aether.graph.DefaultDependencyNode 0x14fc9bd "null"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x5e5ddfbc "com.vaadin:vaadin:jar:14.6.8 (compile)"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x5bda157e "com.vaadin:vaadin-core:jar:14.6.8 (compile)"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x527937d0 "com.vaadin:vaadin-avatar-flow:jar:14.6.8 (compile)"]
   #object[org.eclipse.aether.graph.DefaultDependencyNode 0x59d6642a "org.webjars.bowergithub.vaadin:vaadin-list-box:jar:1.3.0 (compile)"]]}

[technomancy]

I tried trimming this down but I couldn't find a way to do it safely. In the mean time I've got it emitting a warning when it detects a pathological case so people can find a workaround more easily.

[hewrin]

Not sure if this is the best place to ask this, but what does "pathological" mean in this context?

[sanel]

Cool! May I suggest the following:

(defn- max-path-count-detected? [paths nmax]
  (loop [i 0, it paths]
    (if (>= i nmax)
      true
      (when it
        (recur (inc i) (next it))))))
...

(let [potential-paths (all-paths node)]
  (when (max-path-count-detected? potential-paths max-path-count)
    (warn "Pathological dependency tree detected; consider setting ':pedantic? false'."))

Raw iterating is much faster here and no need to create another sequence just for the sake of counting it. Also, not once I've seen "max" number was skipped due unforeseen reasons. Warning is updated to point to actual configuration option.

Not sure if this is the best place to ask this, but what does "pathological" mean in this context?

@technomancy is more knowledgeable here, I'm curious as well.

[technomancy]

Not sure if this is the best place to ask this, but what does "pathological" mean in this context?

"Pathological" comes from a root word pertaining to disease, but in this case it means likely to have harmful symptoms. The idea is that if the dependency checking is likely to hang or run out of memory, the user can at least understand why and find a workaround immediately rather than having to search to find out what's really going on.

Also, not once I've seen "max" number was skipped due unforeseen reasons.

I don't understand what this means; can you elaborate? We are not creating another sequence; we are simply realizing the first N elements of an existing sequence.

[sanel]

I don't understand what this means; can you elaborate? We are not creating another sequence; we are simply realizing the first N elements of an existing sequence.

My bad; was looking thorough the code after a long day :)

"Pathological" comes from a root word pertaining to disease.... The idea is that if the dependency checking is likely to hang or run out of memory,

I'm more curious to know why this happens in leiningen, but not in maven or tools.deps

[technomancy]

I'm more curious to know why this happens in leiningen, but not in maven or tools.deps

As far as I can tell, (I haven't looked into it in a long time) Maven isn't super concerned with ensuring that you get repeatable builds, since they introduced support for version ranges which you can pull in from dependencies without being aware of it. Leiningen is basically trying to undo the damage to repeatable builds that Maven caused by introducing version ranges; in order to do this it has to walk the whole tree of possible dependency resolutions.

[torkus]

FYI, it looks like CircleCI have upgraded their version of Leiningen being installed and I'm seeing this issue now after encountering something similar here: #2769

I'm using the Docker image: circleci/clojure:openjdk-11

My CircleCI config: https://github.com/ogri-la/strongbox/blob/53ec750451958f596e874d3269cda0463fa91968/.circleci/config.yml

And the change that allowed it to finally go green (thanks @sanel): ogri-la/strongbox@cae9cfe

[technomancy]

The recent changes to pedantic checks seem to fix this. There are still a ton of warnings, but it doesn't exhaust memory and completes relatively quickly.