HDDS-9803. Limit total number of buckets to avoid OutOfMemoryError in OM

[sumitagrawl]

What changes were proposed in this pull request?

limit the number of buckets with default to 100000, to avoid OOM when huge number of buckets are created. This is as buckets are full cached in OM.

Its observed that 1 bucket Info consume approx 1KB memory, so 100,000 buckets will take up 100MB approx, so 100,000 is chosen as default and handle most of the usecase.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9803

How was this patch tested?

• Unit test is added to check bucket limit

[nandakumar131]

@sumitagrawl can you check if the test failures are related?

[xichen01]

This is a direct and efficient solution.
But wouldn't it be better if we limited the number of buckets in memory instead of just limiting the number of buckets that can be created, since it's possible that more than just buckets have this problem, and volume is FULL_CACHE as well.

[sumitagrawl]

This is a direct and efficient solution. But wouldn't it be better if we limited the number of buckets in memory instead of just limiting the number of buckets that can be created, since it's possible that more than just buckets have this problem, and volume is FULL_CACHE as well.

Thanks @xichen01 for review, bucket is full cache table, and for performance, it was cached.

• most of read/write operation involves bucket access / update. making it partial cache may impact performance and have impact to most of code where assumption is done for bucket to be full cache.

So IMO, its not viable to make bucket as partial cache.

[adoroszlai]

Thanks @sumitagrawl for working on this.

Shouldn't we also limit volumes at the same time? Creating too many volumes can also result in OOME.

[adoroszlai]

Nit: ozone.om.max.buckets.

[adoroszlai]

I'd prefer adding CacheType type() in TableCache, and using cache.type() == FULL_CACHE instead of instanceof check.

[adoroszlai]

Nit: I think "Cannot create more than " + maxBucket + " buckets" would be a better message.

[sumitagrawl]

Shouldn't we also limit volumes at the same time? Creating too many volumes can also result in OOME.

We already have max number of volume per user restriction, "ozone.om.user.max.volume" with default value 1024. May be this is enough.

[adoroszlai]

Thanks @sumitagrawl for updating the patch.

We already have max number of volume per user restriction, "ozone.om.user.max.volume" with default value 1024. May be this is enough.

Yes, I agree, that's probably fine.

[kerneltime]

@sumitagrawl can you please resolve the conflicting files?

[sumitagrawl]

@sumitagrawl can you please resolve the conflicting files?

done

[sadanand48]

Thanks for the patch @sumitagrawl , LGTM

[sadanand48]

Thanks @sumitagrawl for the change , @xichen01, @adoroszlai for the reviews.