[ADH-8124] Add S3 Gateway ozone objects cache

[tigrulya-exe]

Added caching OM client for S3 gateway component. Currently, it caches metadata for S3 volume, buckets, keys (both regular and extended versions) in local per-user Caffeine caches. Cache entries are evicted on corresponding write operations (delete, rename) or by TTL.

Related options:

Key	Description	Default Value
ozone.s3g.object.cache.enabled	Enable or disable the S3Gateway object cache.	false
ozone.s3g.object.cache.metrics.enabled	Enable or disable metrics for the S3Gateway object cache.	true
ozone.s3g.object.cache.volume.size	Maximum number of entries in the S3Gateway volume cache.	10
ozone.s3g.object.cache.volume.ttl.ms	TTL in milliseconds for entries in the S3Gateway volume cache.	3000
ozone.s3g.object.cache.bucket.size	Maximum number of entries in the S3Gateway bucket cache.	100
ozone.s3g.object.cache.bucket.ttl.ms	TTL in milliseconds for entries in the S3Gateway bucket cache.	3000
ozone.s3g.object.cache.key.size	Maximum number of entries in the S3Gateway key cache.	1000
ozone.s3g.object.cache.key.ttl.ms	TTL in milliseconds for entries in the S3Gateway key cache.	3000

Also a Hadoop Metrics2 source is added to collect and publish the following statistics for all S3 object caches (s3VolumeCache, s3BucketCache, s3HeadKeyCache, s3KeyInfoCache):

Name	Description
hitCount	Cache hit count
hitRate	Cache hit rate
missCount	Cache miss count
loadSuccessCount	Successful cache loads
loadFailureCount	Failed cache loads
loadFailureRate	Failed cache loads rate
evictionCount	Cache eviction count
averageLoadPenalty	Average load penalty (ns)

Example of metrics if Prometheus MetricsSink is used:

# TYPE s3_object_cache_metrics_miss_count gauge
s3_object_cache_metrics_miss_count{cachename="s3BucketCache",hostname="0dfbab4b336f"} 27
s3_object_cache_metrics_miss_count{cachename="s3HeadKeyCache",hostname="0dfbab4b336f"} 10
s3_object_cache_metrics_miss_count{cachename="s3KeyInfoCache",hostname="0dfbab4b336f"} 5
s3_object_cache_metrics_miss_count{cachename="s3VolumeCache",hostname="0dfbab4b336f"} 30

[iamlapa]

Also, BucketEndpoint::handleGetRequest(S3RequestContext context, String bucketName)

ozoneKeyIterator = bucket.listKeys(prefix, prevKey, shallow);

This path still ignores the S3 max-keys value when choosing how much to fetch from OM; the iterator falls back to ozone.client.list.cache, which is 1000 by default, so max-keys=1 can still over-fetch badly. We should add an S3-specific list path that passes a backend fetch size like min(maxKeys + 1, configuredCap).

[tigrulya-exe]

Also, BucketEndpoint::handleGetRequest(S3RequestContext context, String bucketName)

ozoneKeyIterator = bucket.listKeys(prefix, prevKey, shallow);

This path still ignores the S3 max-keys value when choosing how much to fetch from OM; the iterator falls back to ozone.client.list.cache, which is 1000 by default, so max-keys=1 can still over-fetch badly. We should add an S3-specific list path that passes a backend fetch size like min(maxKeys + 1, configuredCap).

Yeah, it was already discussed with Alexander and we decided to skip this requirement for now and check if further work is needed in a separate task. Unfortunately, we can't simply pass the value of max-keys down to OM due to some internal logic, see HDDS-12882