thanos katacoda course: Error fetching stores/flags list in Thanos UI

[juzhao]

https://www.katacoda.com/thanos/courses/thanos/7-multi-tenancy
step 2
Verification part

Let's check if our read isolation works:

    Query for Fruit Team through Caddy 39091 port
    Query for Veggie Team through Caddy 39092 port

click any link, will see errors
Warning: Error fetching stores list: Unexpected response status when fetching stores:
Warning: Error fetching flags list: Unexpected response status when fetching flags:

see the picture from: https://pasteboard.co/K52vBct.png

Browser:
Chrome Version 86.0.4240.111 (Official Build) (64-bit)
FireFox 60.0 (64-bit)

[yeya24]

I can reproduce this. Help wanted!

[bill3tt]

@yeya24 happy to take this on and investigate a fix 👍

[bill3tt]

The problem is that prom-label-proxy currently only proxies a subset of the Querier endpoints. This is by design as this component is used to enforce multi-tenancy so will not leak global information to a single tenant.

The /stores endpoint can be made multi-tenant (as we could filter the returned stores by tenant), but other endpoints e.g. v1/status/flags are inherently 'global'.

IMO solution here is to update the tutorial to replace the links to the Querier UI that do not work with curl commands.

[yeya24]

@ianbillett Thanks for the investigation. I saw that prom-label-proxy has a flag to pass through some APIs using https://github.com/prometheus-community/prom-label-proxy/blob/38b1382d7368e994e39c0cb853f3f3c9300f5045/main.go#L48 flag.

Is it possible to use this?

[bill3tt]

Aha - good spot! I would expect that it is indeed possible given that flag. Let me experiment with that now...

[bill3tt]

TL;DR - it is possible to fix with the new flags in prom-label-proxy but it is not the nicest user experience.

[yeya24]

@ianbillett Got it. Thanks for checking. If there is no better way, then I think it is good to use the curl command to check the response.

[bill3tt]

@yeya24 reverted to curls and CI is now green - I think we're ready to merge 💪

(you can see it in action here)