Improve json generation performance (tweaked #2035) #2105
Conversation
| depToTransitiveDeps(elem) | ||
| else { | ||
| val children0 = children(elem) | ||
| val deps = children0.map(reconciledVersionStr(_)).toSet ++ children0.iterator.flatMap(flattenDeps(_)) |
There was a problem hiding this comment.
In my original PR I thought about using .flatMap on children0 but decided against it because it seems to require O(n^2) memory on dense graphs where n is the size of children0 and the whole point was to minimize memory consumption.
My logic is that in a fully connected directed acyclic graph .flatMap will generate a sequence of size (n-2) + (n-3) + ... + 1 + 0 = (n-2)(n-1)/2 ~ n^2, and only after generating that sequence will it collapse while being converted to Set.
There was a problem hiding this comment.
Yes, but we flatMap on an iterator here, so no large collection actually gets built, I believe. flattenDeps gets called on the first element of children0, the corresponding set is added to the one being built for deps and discarded straightaway, then the next element of children0 is processed, etc.
alexarchambault
force-pushed
the
improve-json-generation-performance
branch
from
0c82ab5
to
21bad4f
Compare
Includes #2035