Adjust FP metrics endpoint

[mjudeikis]

Summary

Adjust how we serve /metrics in the front-proxy

What Type of PR Is This?

/kind cleanup

Related Issue(s)

Fixes #

Release Notes

NONE

[mjudeikis]

/retest

[mjudeikis]

/retest

[SimonTheLeg]

hmm been brooding over this for a little while now. I am not the biggest fan of routing all of these requests to the rootshard. Especially if we think about a distributed system where these requests could have a bigger network penalty.

Even though it is not perfect, I think I would prefer allowing for a local group passthrough, which includes the fixed builtin group and maybe allows for passing in additional groups via flags during startup. Yes that would make it a little inconsistent across, but since metrics scraping is more of a one time setup, I think it is ok and better than to have the long running disadvantage of increasing load on root shard.

[mjudeikis]

hmm been brooding over this for a little while now. I am not the biggest fan of routing all of these requests to the rootshard. Especially if we think about a distributed system where these requests could have a bigger network penalty.

Even though it is not perfect, I think I would prefer allowing for a local group passthrough, which includes the fixed builtin group and maybe allows for passing in additional groups via flags during startup. Yes that would make it a little inconsistent across, but since metrics scraping is more of a one time setup, I think it is ok and better than to have the long running disadvantage of increasing load on root shard.

Thanks for chewing on this one :). I get your worry about sending everything to root shard, especially the network cost when system is distributed.

But honestly, I don't see other way here. The metrics RBAC lives on root shard, so we must route there for delegated authorization. And this is not metrics passthrough — is delegated authorization, so this RBAC must live somewhere anyway, even if later we wire in the groups. We already do exactly same thing for shards, is the same machinery (every other shard than non root will route to root shard). So either we refactor everything, or we stick with what we have — and I don't want to make front-proxy the odd duck.

Also, about the load — I would not worry too much. The delegated authorizer uses cache (default 5 min), so not every request hits the delegate. This should keep us safe from hammering root shard. And worst case, only /metrics endpoint is not working, nothing more.

The local group passthrough is nice idea, but for me the inconsistency it brings is not worth it here. If cache does not hold in practice, then sure, we can look again. Small iterrative changes, not skewed system engineering for problems we dont know we gonna have yet.

TL:DR: Its ok,. let it break (if it gonna break...) and fix later :D

[xrstf]

/approve
/lgtm

[kcp-ci-bot]

LGTM label has been added.

Details

Git tree hash: 5a88dc25b6fa0f698306bdcc6b7c874968db78c3

[kcp-ci-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: xrstf

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [xrstf]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mjudeikis]

/retest

[mjudeikis]

/retest