Slower performance and higher cpu from Mongoose 4 to 5 with large schemas

[mikedavies-dev]

Do you want to request a feature or report a bug:

Bug

What is the current behavior?

When dealing with larger schemas and loading a large number of records Mongoose 5.x seems to be much slower than 4.x.

On my development machine, loading 10k documents using v4 takes approx 4.7 seconds and v5 takes approx 18 seconds to to load the same documents using the same script.

I've created a test project to see the issue including scripts to test under v4 and v5:
https://github.com/ant-fx/mongoose-performance-v4-v5

What is the expected behavior?
Version 5.x should be at least as fast as v4.x

What are the versions of Node.js, Mongoose and MongoDB you are using? Note that "latest" is not a version.

Node: v14.5.0
Mongoose v4: 4.13.21
Mongoose v5: 5.12.10

[vkarpov15]

We found a way to shave 10-20% off of this script by being smarter about how we look up map paths: we had a loop that was O(n^2) in the number of paths in the schema when looking up map paths that we improved by only looking at actual map paths. Won't help if you have a schema with 100 paths that are all maps, but in this case it helps a decent amount.

We'll need to do a little bit of work to make our mapPaths improvement more durable by recalculating map paths on schema compilation rather than relying on clone() to get it right. We'll also search for more places where we can improve performance on this particular example.

[vkarpov15]

We tracked down the primary culprit for the extra overhead: #7104. $getAllSubdocs() is fairly slow, and queries call $session(null) if you don't specify a session, so most of the overhead comes from Mongoose repeatedly calling $getAllSubdocs() to set a null session. We patched that up, but we'll do some work to speed up $getAllSubdocs() since that function is also called in save().

[vkarpov15]

184f10f should be enough to fix this modulo test failures.