Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP path-prefix option results to 404 on 2.4.1 #4756

Closed
vooon opened this issue Nov 15, 2021 · 20 comments · Fixed by #8406
Closed

HTTP path-prefix option results to 404 on 2.4.1 #4756

vooon opened this issue Nov 15, 2021 · 20 comments · Fixed by #8406
Labels
component/querier type/bug Somehing is not working as expected

Comments

@vooon
Copy link

vooon commented Nov 15, 2021

Describe the bug

After update to 2.4.1 i had query errors. During config debugging i tried #4699 (comment) which worked fine until i set server.http_path_prefix="/loki" to match my config.
E.g. logcli --addr http://localhost:3101/loki query {job=~".+"} responds with 404.

To Reproduce
Steps to reproduce the behavior:

  1. Started Loki (2.4.1)
  2. Started promtail
  3. Query: {job=~".+"} term (note: any query, just i used that one)

Expected behavior
Expected to get some lines.

Environment:

  • Infrastructure: bare-metal
  • Deployment tool: manual run of -target=all

Screenshots, Promtail config, or terminal output

$ logcli --addr 'http://192.168.50.10:3100/loki' query '{job=~".+"}'
2021-11-15 15:55:57.263288 I | proto: duplicate proto type registered: purgeplan.DeletePlan
2021-11-15 15:55:57.263374 I | proto: duplicate proto type registered: purgeplan.ChunksGroup
2021-11-15 15:55:57.263382 I | proto: duplicate proto type registered: purgeplan.ChunkDetails
2021-11-15 15:55:57.263388 I | proto: duplicate proto type registered: purgeplan.Interval
2021-11-15 15:55:57.280009 I | proto: duplicate proto type registered: grpc.PutChunksRequest
2021-11-15 15:55:57.280041 I | proto: duplicate proto type registered: grpc.GetChunksRequest
2021-11-15 15:55:57.280052 I | proto: duplicate proto type registered: grpc.GetChunksResponse
2021-11-15 15:55:57.280061 I | proto: duplicate proto type registered: grpc.Chunk
2021-11-15 15:55:57.280078 I | proto: duplicate proto type registered: grpc.ChunkID
2021-11-15 15:55:57.280087 I | proto: duplicate proto type registered: grpc.DeleteTableRequest
2021-11-15 15:55:57.280096 I | proto: duplicate proto type registered: grpc.DescribeTableRequest
2021-11-15 15:55:57.280104 I | proto: duplicate proto type registered: grpc.WriteBatch
2021-11-15 15:55:57.280112 I | proto: duplicate proto type registered: grpc.WriteIndexRequest
2021-11-15 15:55:57.280120 I | proto: duplicate proto type registered: grpc.DeleteIndexRequest
2021-11-15 15:55:57.280128 I | proto: duplicate proto type registered: grpc.QueryIndexResponse
2021-11-15 15:55:57.280136 I | proto: duplicate proto type registered: grpc.Row
2021-11-15 15:55:57.280144 I | proto: duplicate proto type registered: grpc.IndexEntry
2021-11-15 15:55:57.280152 I | proto: duplicate proto type registered: grpc.QueryIndexRequest
2021-11-15 15:55:57.280160 I | proto: duplicate proto type registered: grpc.UpdateTableRequest
2021-11-15 15:55:57.280168 I | proto: duplicate proto type registered: grpc.DescribeTableResponse
2021-11-15 15:55:57.280176 I | proto: duplicate proto type registered: grpc.CreateTableRequest
2021-11-15 15:55:57.280184 I | proto: duplicate proto type registered: grpc.TableDesc
2021-11-15 15:55:57.280192 I | proto: duplicate proto type registered: grpc.TableDesc.TagsEntry
2021-11-15 15:55:57.280200 I | proto: duplicate proto type registered: grpc.ListTablesResponse
2021-11-15 15:55:57.280209 I | proto: duplicate proto type registered: grpc.Labels
2021-11-15 15:55:57.280374 I | proto: duplicate proto type registered: storage.Entry
2021-11-15 15:55:57.280390 I | proto: duplicate proto type registered: storage.ReadBatch
http://192.168.50.10:3100/loki/loki/api/v1/query_range?direction=BACKWARD&end=1636980957281844223&limit=30&query=%7Bjob%3D~%22.%2B%22%7D&start=1636977357281844223
Error response from server: 404 page not found
 (<nil>) attempts remaining: 0
Query failed: Run out of attempts while querying the server
logcli, version 2.4.1 (branch: HEAD, revision: f61a4d261)
  build user:       root@9cbe2f86250f
  build date:       2021-11-08T13:05:10Z
  go version:       go1.16.2
  platform:         linux/amd64
@vooon
Copy link
Author

vooon commented Nov 15, 2021

In addition i can't bind grpc on private network anymore, because frontend tries to connect from 127.0.0.1. Log have errors about eth0 and en0 interfaces, even though i set:

common:
  path_prefix: "/var/lib/loki"
  replication_factor: 1  # default is 3
  ring: &common_ring
    instance_addr: "192.168.50.10"
    instance_interface_names: ["vlan2000"]
    ...

@azuregos
Copy link

I concur that issue exists with Loki 2.4.1 after http_path_prefix is being set.

  1. Create Loki configuration file loki.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  http_path_prefix: /loki

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

ingester:
  wal:
    enabled: true

schema_config:
  configs:
    - from: 2021-01-01
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
  1. Create Docker compose file docker-compose.yml
version: "3.9"
services:

  loki:
    image: grafana/loki:2.4.1
    container_name: loki
    command: -config.file=/etc/loki/config.yaml
    volumes:
      - ./loki.yaml:/etc/loki/config.yaml:ro
    ports:
      - 3100:3100
    networks:
      - loki

  grafana:
    image: grafana/grafana:8.2.5
    container_name: grafana
    ports:
      - 3000:3000
    networks:
      - loki

networks:
  loki:
  1. Run Docker compose with docker compose up -d command
  2. Open Grafana UI in a browser (http://localhost:3000/login) and log is as admin/admin
  3. Add Loki datasource using url http://loki:3100/loki
  4. Open Grafana's Explore panel and attempt to query Loki using query {Job="Foo"}
  5. Note the 404 error reported by Grafana

If http_path_prefix is not being specified in loki.yaml file the same setup will work flawlessly.

@azuregos
Copy link

Loki 2.4.0: Issue also present
Loki 2.3.0: Issue NOT present

@stale
Copy link

stale bot commented Jan 9, 2022

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

  • Mark issues as revivable if we think it's a valid issue but isn't something we are likely
    to prioritize in the future (the issue will still remain closed).
  • Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

@stale stale bot added the stale A stale issue or PR that will automatically be closed. label Jan 9, 2022
@azuregos
Copy link

This issue is still very relevant, a serious regression and a blocker for us.
Without URL path prefix we can't access Loki instance behind a reverse proxy. Not everyone can forward ports so having a control over what URL your APIs have is a must.

@stale stale bot removed the stale A stale issue or PR that will automatically be closed. label Jan 12, 2022
@grzegorznowak
Copy link

This thread saved my a lot of grief, thanks guys.
The only thing to add here is I couldn't still make it to connect grafana -> loki based on the suggestions above. Removal of http_path_prefix didn't fix the 502 and curl -G -s http://10.0.1.190:3100/loki/api/v1/labels was still hanging.

Our local setup binds both loki and grafana to virtual mesh VPN interface and it does work OK under 2.3.0, but no go on 2.4.* so far. Unless we bind everything to public interface of course, then it runs tip-top.

@vooon
Copy link
Author

vooon commented Jan 21, 2022

@grzegorznowak you need to set instance_addr and instance_interface_names and some other parameters, but i'm actually stuck on trouble with some of services not obeying them. So i gave up and set grpc port to listen on '[::]' and just added iptables rule...
And yes, for 2.3.0 it's been enough to set those two variables in ring...

@stale
Copy link

stale bot commented Mar 2, 2022

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

  • Mark issues as revivable if we think it's a valid issue but isn't something we are likely
    to prioritize in the future (the issue will still remain closed).
  • Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

@stale stale bot added the stale A stale issue or PR that will automatically be closed. label Mar 2, 2022
@azuregos
Copy link

azuregos commented Mar 3, 2022

The problem is still very relevant and very critical. NOGO on using Loki 2.4+ until its fixed for us.

@stale stale bot removed the stale A stale issue or PR that will automatically be closed. label Mar 3, 2022
@azuregos
Copy link

In case you were wondering Loki 2.5.0 still has this issue. 😞

@s-frei
Copy link

s-frei commented Apr 26, 2022

Stumbled upon the same issue upgrading from 2.2.1 to 2.5.0. As already mentioned it is not really nice to have it behind a reverse proxy like this... Since I'm not into Go I cannot provide a fix for this, unfortunately.

@MaximeTaqt
Copy link

Same issue too.
I downgraded to 2.3.0

@gjed
Copy link

gjed commented Jun 16, 2022

On a related note: grafana 9.0.0 actively strip the path prefix from loki url (see #50939)

@v1ctorrhs
Copy link

+1 here, we are also facing the same issue, loki 2.6.1

@Q-Hilbtec
Copy link

I had updated to 2.6.1 to get deletion support, but rolled back to 2.3.0 after hitting #6231 which appears directly related to this. Unfortunately had to delete my v12 index as well at the same time.

@debuglevel
Copy link

Just tested Loki on my machine and wondered why it just did not work when I wanted to use it in production. I guess I have to use the over a year old version 2.3.0
o_O

@ryoshindo
Copy link
Contributor

I have confirmed that this bug has not been fixed in v2.7.2. I would love to have this bug fixed as it is a very serious problem for my use of loki.
Thanks

@chaudum chaudum added type/bug Somehing is not working as expected and removed kind/bug labels Jun 14, 2023
MasslessParticle added a commit that referenced this issue Jun 23, 2023
**What this PR does / why we need it**:

Hi, loki developers.

I would like to use the option `http_path_prefix` as described in the
official documentation.
This setting is valid for endpoints defined at
https://github.com/grafana/loki/blob/4d55a24ed3cbed399c189dafb8c4140cbffce21d/pkg/loki/loki.go#L477-L494
but not for the endpoints defined at
https://github.com/grafana/loki/blob/4d55a24ed3cbed399c189dafb8c4140cbffce21d/pkg/loki/modules.go#L366-L392
.

The current latest version with `http_path_prefix` will return the
expected response for the former endpoint, but will return a 404 not
found error when accessing the latter endpoint.
For example, if you define `http_path_prefix = /xxx`, requests for
`/xxx/ready` will work as expected, but requests for
`/xxx/loki/api/v1/query` will result in 404 errors.

I've created a pull request for this issue because I've been
experiencing some inconvenience when using loki. Please check it out.

**Which issue(s) this PR fixes**:
Fixes #4756

**Special notes for your reviewer**:

**Checklist**
- [x] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [x] Tests updated

---------

Co-authored-by: Michel Hollands <42814411+MichelHollands@users.noreply.github.com>
Co-authored-by: Travis Patterson <travis.patterson@grafana.com>
@tiran133
Copy link

Still Happening in v2.9.1 Getting 404 in Grafana.

#8406 (comment)

@fsiler
Copy link

fsiler commented Sep 21, 2023

I have this same issue- manually bisected as described here.

@Pexers
Copy link

Pexers commented Jan 24, 2024

+1
Issue persists in grafana/loki:2.9.3. Like others said, when the prefix path is not specified, everything works just fine.. 😞

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/querier type/bug Somehing is not working as expected
Projects
None yet
Development

Successfully merging a pull request may close this issue.