3.6: Restart loop on AWS S3 bucket (not a valid uri)

[Christoph-AK]

Since the update to 3.6 yesterday our loki doesn't run, saying the s3 bucket URI is not configured correctly. This hasn't been a problem before and has been running fine for months before. We didn't see a related change in the changelog.

loki  | 2025-11-18T10:19:29.460554713Z level=error ts=2025-11-18T10:19:29.459469384Z caller=log.go:223 msg="error running loki" err="init compactor: failed to init delete store: operation error S3: DeleteObject, resolve auth scheme: resolve endpoint: endpoint rule error, Custom endpoint `s3.eu-central-1.amazonaws.com` was not a valid URI\nerror initialising module: compactor\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:549\nmain.main\n\t/src/loki/cmd/loki/main.go:130\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:283\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"
loki  | 2025-11-18T10:20:29.881756367Z init compactor: failed to init delete store: operation error S3: DeleteObject, resolve auth scheme: resolve endpoint: endpoint rule error, Custom endpoint `s3.eu-central-1.amazonaws.com` was not a valid URI
loki  | 2025-11-18T10:20:29.881786236Z error initialising module: compactor
loki  | 2025-11-18T10:20:29.881790177Z github.com/grafana/dskit/modules.(*Manager).initModule
loki  | 2025-11-18T10:20:29.881793218Z  /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138
loki  | 2025-11-18T10:20:29.881796605Z github.com/grafana/dskit/modules.(*Manager).InitModuleServices
loki  | 2025-11-18T10:20:29.881799554Z  /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
loki  | 2025-11-18T10:20:29.881802488Z github.com/grafana/loki/v3/pkg/loki.(*Loki).Run
loki  | 2025-11-18T10:20:29.881805364Z  /src/loki/pkg/loki/loki.go:549
loki  | 2025-11-18T10:20:29.881808369Z main.main
loki  | 2025-11-18T10:20:29.881811010Z  /src/loki/cmd/loki/main.go:130
loki  | 2025-11-18T10:20:29.881813824Z runtime.main
loki  | 2025-11-18T10:20:29.881816426Z  /usr/local/go/src/runtime/proc.go:283
loki  | 2025-11-18T10:20:29.881819116Z runtime.goexit
loki  | 2025-11-18T10:20:29.881821714Z  /usr/local/go/src/runtime/asm_amd64.s:1700

Loki Config:

auth_enabled: false

server:
  http_listen_port: 4100
  http_server_read_timeout: 3m
  http_server_write_timeout: 3m
  grpc_server_max_recv_msg_size: 10485760
  grpc_server_max_send_msg_size: 10485760

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 10s
  chunk_idle_period: 5m
  wal:
    dir: "/loki/wal"
    flush_on_shutdown: true
    replay_memory_ceiling: "512MB"

schema_config:
  configs:
    - store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: aws_index_
        period: 24h

    - from: "2024-04-10"
      store: boltdb-shipper
      object_store: aws
      schema: v13
      index:
        prefix: aws_index_
        period: 24h


compactor:
  working_directory: /loki/boltdb-shipper-compactor
  # shared_store: aws
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  delete_request_store: aws

limits_config:
  # enforce_metric_name: false
  split_queries_by_interval: 24h
  retention_period: 1y
  allow_structured_metadata: false

storage_config:
  aws:
    bucketnames: ak-loki-log
    region: eu-central-1
    access_key_id: <id>
    secret_access_key: <key>
    # sse_encryption: true
    endpoint: s3.eu-central-1.amazonaws.com

  boltdb_shipper:
    active_index_directory: /loki/index
    # shared_store: s3
    cache_location: /loki/boltdb-cache

frontend:
  compress_responses: true
  max_outstanding_per_tenant: 4096

querier:
  max_concurrent: 1024

[Christoph-AK]

changed config to more recent system, still in restart loop. Rolled back to 3.5.8

  aws:
    bucketnames: ak-loki-log
    region: eu-central-1
    s3forcepathstyle: false 
    access_key_id: <id>
    secret_access_key: <key>

[chewrocca]

I had this same issue. Adding https:// to the endpoint seemed to help.

[sfudeus]

I can confirm this against a minio instance in a local K8s cluster. I'm too using the expanded config form and had to add the schema to the endpoint. The documentation at https://grafana.com/docs/loki/latest/configure/examples/configuration-examples/#10-expanded-s3-snippetyaml would state otherwise.

[deltamualpha]

I'm also running into what I think are related issues with the older url style of configuration. Repeated messages about failed to flush chunks: store put chunk: operation error S3: PutObject, failed to resolve service endpoint, endpoint rule error, A region must be set when sending requests to S3.

The key I had was this, which had been working:

aws:
  s3: s3://<key>:<secret>@us-west-2/<bucketname>

[deltamualpha]

Ah, I've found the source of my issues:

e35887b#diff-5d7ab8e45794d457a055e9cebd6ca11077479b6951ce373cd18a7f24bbb557beR76

#19205 changed how Loki consumes the url, going from the general ConfigFromURL method to CredentialsFromURL, pulling only the credentials out of the url config, and assuming that everyone just set all of the other keys on the option. Presumably this is also why the endpoint is suddenly expecting https:// -- a change in what the new SDK expects that was not documented for loki operators.

@sophiewaldman @paul1r I think there are some configuration regressions that either need to be documented or fixed from the AWS SDK v2 upgrade.

[chaudum]

Thanks for reporting this issue. The problem is indeed caused by the upgrade of the AWS SDK. I am working on a fix for both the invalid endpoint and the s3 URL issue.