Excessive Memory Usage by LookupCoordinatorManager in master nodes Due to keeping a copy of each Lookups per process

[issam-messai]

Excessive Memory Usage by LookupCoordinatorManager in master nodes Due to keeping a copy of each Lookups per process

Affected Version

The Druid version where the problem was encountered is 24.0.0

Description

I encountered a severe memory usage issue in master nodes when using lookups. The LookupCoordinatorManager keeps a copy of each static lookup in the member knownOldState for each running process on query and data servers (historical, middleManager, peon, broker, router) in the cluster. As a result, the memory consumption scales linearly with the number of processes, causing excessive heap usage on master nodes and ExitOnOutOfMemoryError crashes.

Observed Behavior

• I added lookups with a total size of 100MB.
• My cluster has at least running 14 processes on data and query servers without counting peon processes.
• The lookup memory consumption on the master node reached more than 3.6GB (100MB x 2 x 14), leading to OOM java exception
• Heap dump analysis using Eclipse Memory Analyzer (MAT) showed that org.apache.druid.server.lookup.cache.LookupCoordinatorManager is consuming 90% of the heap. (attaches some screenshots of the heap dump analysis)

Expected Behavior

• LookupCoordinatorManager should not duplicate lookups unnecessarily for each process.
• Lookups should have a shared or optimized memory footprint across processes.
• The memory overhead for lookups should remain proportional to their actual size, not process count.

[gianm]

Are these map type lookups? For those lookups, the definition contains the data. I wonder if what's going on here is that we are storing a copy of the definition per server, which would add up quickly for a larger map lookup.

If that's what's happening, it would be better to deduplicate those in the LookupCoordinatorManager, or perhaps we don't even need to store the current lookup definitions at all. I think the logic would possibly work if we only stored the version of the current lookup definitions on each node.

Also, if that's what's happening, you could work around it by using a uri-based lookup instead of map. Then the definitions will be small enough that this won't be a problem.

[issam-messai]

@gianm Thanks for your response

Yes lookups are of type map.

I totally agree, it would be much better if de-duplicate or storing the only the version on master nodes, is this possible on a newer versions ?

At this point i will test uri-based lookup for my use case.

[gianm]

Even the latest version does not automatically deduplicate lookup definitions in this way. For larger lookups I would definitely suggest using uri lookups. The map ones were intended for use with small lookups, like up to a few MB. It would be a nice enhancement to dedupe the definitions to support somewhat larger maps, so we could treat this issue as a feature request.

In the meantime, if you have any issues with uri lookups please feel free to raise a new issue or contact us on Slack. thank you!