What are the recommended values for directoryCacheMaxEntries and commandCacheMaxEntries?

[DarkMatterV]

Currently, I set the value of directoryCacheMaxEntries, commandCacheMaxEntries, digestToActionCacheMaxEntries, recentServedExecutionsCacheMaxEntries to 6553600. However, MAT shows that directoryCache and commandCache occupy a large amount of memory. I think it's because the size is set too large. Do you have any suggestions?

[werkt]

I'm not sure if we've exposed enough of the runtime metrics for those caches to make decisions from any externally observable measurements.
Pathologically, you want that cache to be large enough to keep the hottest values in without thrashing, and small enough where you only seek to populate when you will get a decent reuse benefit - i.e. it should follow the trends of your builds to incorporate or discard novel or old metadata sets, respectively.
FWIW there is not a lot of additional memory use on the server anyway, so if you're not using it for this, what else do you need it for?

[DarkMatterV]

Currently, I set -Xmx256g and I've noticed that sometimes jvm_memory_bytes_used / jvm_memory_bytes_max goes up to 100% for a short period of time, and I'm not sure what's causing it yet.

Then, I use jmap -dump:live,format=b,file=heap.hprof {pid} dump heap file and MAT analyse it, and MAT shows that ShardInstance、directoryCache and commandCache occupy a large amount of memory. So I'm guessing it's because I set the size too high. Do you think I should reduce the size?

[DarkMatterV]

Or do you have any suggestions for locating OOM-related problems?

[werkt]

I say, yes, reduce the size (of the caches). I bet that a good portion of them decay quickly (but again, I would need to have/add stats to say that for sure), and that a useful comparison will be a sentinel build of some sort with full execution, on a fresh server (twice, to see the impact of population), against one that has a reduced size. If your time is unaffected (and the builds are representative), then you're likely unaffected by thrash.

Dealing with OOMs is a matter of exactly what you're doing. jmap, object counts over time, perturbation of likely causes, and measuring the effects and timing of GC runs.