Skip to content

Conversation

@devmadhuu
Copy link
Contributor

@devmadhuu devmadhuu commented Oct 31, 2025

What changes were proposed in this pull request?

Based on attached alloc_profile.html, this PR change is to optimize NSSummary Memory Allocations in Recon Task Processing.
The NSSummary task was experiencing performance bottlenecks when processing OM DB events due to:

  • Excessive object allocation and garbage collection pressure
  • Redundant database reads for the same parent/directory entries
  • Inefficient collection handling (creating new objects on every access)

Performance Impact

Expected Improvements:

  1. 60-75% reduction in allocations during event processing
  2. Significant DB read reduction - cache hit rate ~90%+ for hot parents
  3. Lower GC pressure - fewer short-lived objects
  4. Faster processing - in-place mutations vs object creation

Trade-offs:

  1. Memory usage - 32K LRU cache (but small compared to benefits)
  2. Code complexity - more helper methods
  3. API changes - getChildDir() never returns null (breaking change)

Why This Matters

For large-scale Ozone deployments:

  • High event volume - thousands of file operations per second
  • Deep directory trees - same parent accessed repeatedly
  • GC sensitivity - reducing allocations critical for latency
  • Resource efficiency - less CPU/memory for background tasks

alloc_profile.html

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11688

How was this patch tested?

Tested with existing junit and integration tests for functionality and plan is to do performance and scalability test in large cluster.

@github-actions
Copy link

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

@github-actions github-actions bot added the stale label Nov 25, 2025
@adoroszlai
Copy link
Contributor

This issue is fixed as part of HDDS-13552 (source).

@adoroszlai adoroszlai closed this Dec 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants