Loki returns 404 when no filters are specified

[v1ctorrhs]

Describe the bug
After upgrading loki from 2.3.0 to 2.5.0 several things happened:

• Label values autocomplete doesn't work
• Running a query with no filter returns 404

To Reproduce
Steps to reproduce the behavior:

1. Started Loki (SHA or version) 2.5.0
2. Started Promtail (SHA or version) to tail '...' - I don't think promtail is involved here at all
3. Query: {} term

Expected behavior

• I expect when I type out the label, a dropdown to appear with all possible values and to narrow down as i'm typing it out
• I expect to be able to run {pod="loki-us-east-1-0"} without having to add the |~ "" as this confuses our devs that are end users of the grafana

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm, loki chart, v2.11.1

Screenshots, Promtail config, or terminal output

• No autocomplete for the pod name

• No filter returns 404

• Adding |~ "" returns results

[ayankovhs]

To add to the above, we are using Loki in monolith mode, with the following config:

extraArgs:
  server.path-prefix: "/loki"

Looking at the requests that Grafana makes, there were some interesting trends. For example:

• https://<grafana_url>/api/datasources/proxy/11/loki/api/v1/label/<label_name>/values?start=1656056305758000000&end=1656059905758000000 returns 404, while https://<grafana_url>/api/datasources/proxy/11/loki/api/v1/series?match%5B%5D=%7B<label_name>!%3D%22%22%7D&start=1656056314077000000&end=1656059914077000000 returns just fine.
• For the same Grafana query, there are two query_range requests. One for the metric visualization, one for the logs. The former https://<grafana_url>/api/datasources/proxy/11/loki/api/v1/query_range?direction=BACKWARD&limit=812&query=sum%20by%20(level)%20(count_over_time(%7B<label_name>!%3D%22%22%7D%5B1m%5D))&start=1656057445701000000&end=1656061045701000000&step=60 runs fine, the latter https://<grafana_url>/api/datasources/proxy/11/loki/api/v1/query_range?direction=BACKWARD&limit=1000&query=%7B<label_name>!%3D%22%22%7D&start=1656057445701000000&end=1656061045701000000&step=5 404s.
• https://<loki_url>/loki/loki/api/v1/labels returns fine while https://<loki_url>/loki/loki/api/v1/label/<label_name>/values 404s.

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[v1ctorrhs]

@azuwis Hi, do you have any thoughts on this? Thank you

[jeschkies]

Interesting. I've just ran it on Grafana cloud and {mylabel="foobar"} worked. I wonder whether this has been fixed with 2.6. Could you try a newer version.

[ayankovhs]

Thanks for having a look! I just upgraded to Helm chart v2.14.1, which uses Loki v2.6.1 and see the same issue. Curiously, the last point from my previous comment no longer holds - https://<loki_url>/loki/loki/api/v1/labels also returns 404 now.

[ayankovhs]

Probably related to #4756, which I was able to reproduce locally from latest master and setting server.http_path_prefix to a non-empty string.

I'm happy to work on a solution but I got a bit lost in the codebase while trying to follow the server handlers. Would appreciate any pointers 🙏