perf: cache WithTags handlers in tallyMetricsHandler to reduce allocations (replaced with https://github.com/temporalio/temporal/pull/9620 )

[mykaul]

Summary

Cache WithTags() child handlers in tallyMetricsHandler via sync.Map to eliminate repeated tagsToMap(), scope.Tagged(), and handler struct allocations on the hot path. On cache hit: zero allocations.

Design

• childCache (sync.Map): maps normalized tag cache keys to *tallyMetricsHandler children. Uses LoadOrStore for safe concurrent access.
• normalizeTagsForCaching: normalizes excluded tags before cache key computation so high-cardinality excluded values (e.g. activityType with thousands of distinct activity names) collapse to a single cache entry, preventing unbounded cache growth. Zero-alloc fast path when no tags need substitution.
• tagsCacheKey: builds compact \x00-separated string keys from tag slices, with a single-tag fast path that avoids byte buffer allocation.
• Empty WithTags() calls short-circuit and return self.

Allocation Reduction (pprof alloc_space, 5min ScyllaDB workload)

Metric	Before	After	Reduction
WithTags cumulative	1,930 MB	316 MB	-83.6%
Total server allocs	18,030 MB	16,481 MB	-8.6%

Benchmark (omes throughput_stress, mc150, 5 min)

Host networking, i7-1270P 4 cores/component, inter-run data resets:

Database	Baseline	After	Change
Cassandra	280	294	+5.0%
ScyllaDB	290	296	+2.1%

Note: Throughput variance at mc150 is ~5-10%. The allocation reduction is confirmed by pprof but throughput gains are within noise at this concurrency level.

Testing

• Cache hit returns same pointer (require.Same)
• Different tags return different handlers
• Cached handler records metrics correctly
• Multi-tag caching
• Tag order sensitivity (documented, not normalized)
• Child caches are independent per handler level
• Excluded tags share a single child handler (prevents unbounded growth)
• Concurrent access (32 goroutines × 100 iterations, -race)
• tagsCacheKey boundary ambiguity prevention
• normalizeTagsForCaching unit tests (zero-alloc fast path, excluded, allowed, mixed, empty)
• All existing tests continue to pass

[mykaul]

#9620 is a more complete PR.