Support vertical sharding of more complex label_replace function

[yeya24]

Is your proposal related to a problem?

In #5889, I added vertical sharding support for label_replace function. The idea is that if the label we are going to change/join is not included in the by labels, then we can safely shard the query.

However, there is another edge case we miss. Consider the following query, we have nested aggregations with different grouping labels. The outer aggregation labels are namespace, container_name, pod_name while the inner aggregation labels namespace, container, pod. The query is not shardable with the current analyzer because of labels mismatch.

However, with the label replace function, container and pod label will be mapped to container_name and pod_name in the response so this query should be shardable.

avg by (namespace, container_name, pod_name) (
  label_replace (
    label_replace (
      avg by (namespace, container, pod) (
        kube_container_some_metrics
      ),
      "container_name",
      "$1",
      "container",
      "(.+)"
    ),
    "pod_name",
    "$1",
    "pod",
    "(.+)"
  )
)

Describe the solution you'd like

Not sure if there is a good way to do it. I am thinking about translating the replaced label to original label name and see if there is any matches.

[yeya24]

Ah actually with the current implementation, the dynamic label container_name and pod_name will be excluded from the sharding labels and the query can be vertically sharded by namespace.

This is fine, but I think we can do better to shard by all 3 labels.