[Feature Request] Optional Caching of Retrieved Data

[tylerdotrar]

To the best of my understanding, upon execution rusthound_ce stores objects and queries in the host's memory rather than any other caching methods.

In most cases this is fine, but unfortunately in very large environments it is very possible for the rusthound_ce executable to utilize excessive amounts of RAM (e.g., > 32GB). Upon resource exhaustion and without proper caching, hours of querying and processing is can be lost -- resulting in an "Error: EndOfStream" message.

Similar to AD-miner, a caching option would be nice. I understand this is more of an outlier / niche issue, but a simple toggle parameter, an option to cache every X objects (e.g. --cache 100000), an option caching if the tool fails mid-LDAP object retrieval, or really anything could be a nice quality of life improvment.

[g0h4n]

Indeed actually, this feature is one of the most essential in RustHound CE and requires thorough examination on my part. All help is welcome to develop this feature 💯

[z-jxy]

A good place to start on this would be json::parser::parse_result_type.

Instead of storing all the results, they should be written to disk.

Then, in json::checker::check_all_result read the data from disk as necessary. Ideally, this function would be refactored to operate on the data in steps so that it can release memory that's no longer needed. Otherwise, we still have the same issue of too many objects being in memory at once.

Getting this right will be a bit tricky as some steps depend on data from multiple sources, which is then used in further steps that could further modify the data.

I think this is also a good opportunity to identify other areas where performance/memory usage could be optimized.

[z-jxy]

After looking into this more, most of the memory usage comes from the ldap results.

Saving them to disk as they're processed and later reading them using a BufReader results in much lower peak memory usage (seeing a 70% decrease in my implementation).

The only other option I can think of would require a complete rewrite where multiple ldap queries are made to map the results in stages, and I'm not sure that this would make a big difference compared to the caching approach.

[g0h4n]

Thanks @z-jxy for your awesome work!
Version (2.4.0) now includes an optional feature to store LDAP results on disk instead of in memory, reducing memory usage by more than 70%. 🔥

More information from the original PR