feat(logs): add pattern clustering, tag management, and JSON processor

[DDuongNguyen]

What does this PR do?

Adds the pattern intelligence layer — clustering, tag management, and JSON processing:

• Clustering (pkg/logs/patterns/clustering/): ClusterManager groups logs by structural similarity using tokenized signatures. Includes Pattern abstraction, pattern-level eviction (using the eviction framework from PR 3), and a merging/ sub-package for cluster consolidation
• Tag management (pkg/logs/patterns/tags/): TagManager tracks per-tag pattern state so patterns can be scoped to log sources. Includes tag-level eviction to bound memory per source
• JSON processor (pkg/logs/patterns/processor/): Extracts and processes JSON-structured logs before tokenization, handling nested fields and array normalization

This is PR 4/6 in a stack. Depends on PR 3 (#49217) for the eviction framework. The gRPC sender (PR 5) consumes cluster/tag state for encoding decisions.

Motivation

This is the core intelligence of the stateful encoding feature: deciding which logs share patterns, tracking those patterns per source/tag, and evicting stale patterns to bound memory. These modules sit between raw tokenization (PR 2) and the network sender (PR 5).

Describe how you validated your changes

• Unit tests for cluster formation, pattern matching, and merging
• Tag manager tests covering lifecycle, eviction, and multi-tag scenarios
• Pattern eviction manager tests verifying bounded growth
• JSON processor tests for nested extraction and edge cases

How to Review this PR

1. clustering/cluster_manager.go — the main entry point for pattern grouping
2. clustering/pattern.go + pattern_eviction.go — how patterns are tracked and evicted
3. tags/tag_manager.go — per-source pattern scoping
4. processor/json.go — JSON log preprocessing
5. merging/merging.go — cluster consolidation logic

Additional Notes

~6,000 lines including tests. The clustering and tag modules both implement the Evictable interface from PR 3, demonstrating the framework's reuse.

[DDuongNguyen]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• utf8 #49898 : 3 dependent PRs (#49961 , #50036 , #50047 )
• fix(logs): staging fidelity fixes — tabs, newlines, Unicode positions, UTF-8 safety #49810
• better json #49672
• local proto plus servic #49628
• feat(logs): periodically remove unused entry from dict #49494
• feat(logs): max template size limit + json_as_raw raw bypass #49552
• fix(logs): send delete on tag eviction #49493
• fix(logs): rebuild libpatterns.so targeting glibc 2.30 (was 2.34) #49446
• feat(logs): send JSON via dict-encoded sets of keys #49404
• feat(logs): add dual-send shadow gRPC pipeline #49327
• feat(logs): wire stateful encoding into log agent pipeline #49220
• feat(logs): add pattern extraction to gRPC stateful sender #49219
• feat(logs): add pattern clustering, tag management, and JSON processor #49218 👈 (View in Graphite)
• feat(logs): add generic eviction framework for pattern management #49217
• feat(logs): add Rust FFI tokenizer bridge with FlatBuffers #49216
• feat(logs): add stateful encoding module plumbing and token types #49215
• [Logs Stateful Encoding] transport layer initial implementation #48879
• logs-stateful-main

This stack of pull requests is managed by Graphite. Learn more about stacking.