Loki uses HTTP for S3 instead of HTTPS

[Unactived]

Describe the bug
When using the s3 storage backend with an s3-compatible API, Loki (allegedly) attempts to connect to the specified endpoint with an http scheme rather than https, even though s3_storage_config.insecure is false.

This is an issue for me as the object store I have to use only accepts the https scheme and not http (unrelated to the port).

To Reproduce
Using the official Loki Docker image version 2.6.1, and pushing some logs to it.

Relevant config:

schema_config:
  configs:
    - from: 2022-01-01

      store: boltdb-shipper
      object_store: s3

      schema: v11
      index:
        prefix: loki_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: s3

  aws:
    s3: s3://access_key:secret_key@endpoint:port/bucket
    s3forcepathstyle: true

    # HTTPS
    insecure: false
    sse_encryption: false

Expected behavior
Loki runs and is able to sync index and chunks on the provided object store through https.

Actual behaviour
Loki tries to use http. Which in my situation correctly results in empty replies.

Environment:

• Infrastructure:
• Loki image : Official Docker in an Ubuntu virtual machine
• Object store : Swift (OpenStack) ; (I need to use i through S3, and anyhow this seems like an issue that could affect other users than me who only have S3)
• Deployment tool: docker compose

Screenshots, Promtail config, or terminal output

level=info ts=2022-09-29T11:31:16.471455302Z caller=loki.go:374 msg="Loki started"
level=info ts=2022-09-29T11:31:19.413882007Z caller=scheduler.go:682 msg="this scheduler is in the ReplicationSet, will now accept requests."
level=info ts=2022-09-29T11:31:19.414200346Z caller=worker.go:209 msg="adding connection" addr=172.18.0.2:9095
level=info ts=2022-09-29T11:31:21.472486286Z caller=compactor.go:386 msg="this instance has been chosen to run the compactor, starting compactor"
level=info ts=2022-09-29T11:31:21.472818993Z caller=compactor.go:413 msg="waiting 10m0s for ring to stay stable and previous compactions to finish before starting compactor"
level=info ts=2022-09-29T11:31:26.415269849Z caller=frontend_scheduler_worker.go:101 msg="adding connection to scheduler" addr=172.18.0.2:9095
level=info ts=2022-09-29T11:32:15.310369764Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2022-09-29T11:32:15.314246588Z caller=table_manager.go:167 msg="handing over indexes to shipper"
level=info ts=2022-09-29T11:33:04.406700343Z caller=table_manager.go:180 msg="downloading all files for table loki_19264"
level=error ts=2022-09-29T11:33:04.468122657Z caller=index_set.go:265 table-name=loki_19264 user-id=fake msg="sync failed, retrying it" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2Ffake%2F\": EOF"
level=error ts=2022-09-29T11:33:04.477475008Z caller=index_set.go:265 table-name=loki_19264 user-id=fake msg="sync failed, retrying it" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2Ffake%2F\": EOF"
level=error ts=2022-09-29T11:33:04.477573053Z caller=index_set.go:104 table-name=loki_19264 user-id=fake msg="failed to initialize table loki_19264, cleaning it up" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2Ffake%2F\": EOF"
level=error ts=2022-09-29T11:33:04.477611957Z caller=table.go:294 table-name=loki_19264 msg="failed to init user index set fake" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2Ffake%2F\": EOF"
level=error ts=2022-09-29T11:33:04.477647243Z caller=table.go:312 table-name=loki_19264 org_id=fake msg="index set fake has some problem, cleaning it up" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2Ffake%2F\": EOF"
ts=2022-09-29T11:33:04.477742042Z caller=spanlogger.go:80 table-name=loki_19264 user-id=fake org_id=fake level=info msg="downloaded index set at query time" duration=70.030233ms
level=error ts=2022-09-29T11:33:04.488331735Z caller=index_set.go:265 table-name=loki_19264 msg="sync failed, retrying it" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2F\": EOF"
level=error ts=2022-09-29T11:33:04.498580977Z caller=index_set.go:265 table-name=loki_19264 msg="sync failed, retrying it" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2F\": EOF"
level=error ts=2022-09-29T11:33:04.498710982Z caller=index_set.go:104 table-name=loki_19264 msg="failed to initialize table loki_19264, cleaning it up" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2F\": EOF"
level=error ts=2022-09-29T11:33:04.498764433Z caller=table.go:294 table-name=loki_19264 msg="failed to init user index set " err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2F\": EOF"
level=error ts=2022-09-29T11:33:04.498799709Z caller=table.go:312 table-name=loki_19264 org_id=fake msg="index set  has some problem, cleaning it up" err="RequestError: send request failed\ncaused by: Get \"http://endpoint:port/bucket?delimiter=%2F&list-type=2&prefix=index%2Floki_19264%2F\": EOF"
ts=2022-09-29T11:33:04.498889649Z caller=spanlogger.go:80 table-name=loki_19264 org_id=fake level=info msg="downloaded index set at query time" duration=21.080771ms
ts=2022-09-29T11:33:04.499242534Z caller=spanlogger.go:80 user=fake method=query.Label level=info org_id=fake latency=fast query_type=labels length=1h0m0s duration=94.005268ms status=200 label= throughput=0B total_bytes=0B total_entries=2
level=info ts=2022-09-29T11:33:04.501500869Z caller=metrics.go:170 component=frontend org_id=fake latency=fast query_type=labels length=1h0m0s duration=97.358476ms status=200 label= throughput=0B total_bytes=0B total_entries=2

[Unactived]

It seems as though adding (along the s3 field, and still having insecure: false) endpoint: https://endpoint:port fixes this.

These errors disappear, and status codes go from 500 to 200 in /metrics (at loki_s3_request_duration_seconds_bucket).

However after running for several minutes and accumulating ~ 10k log lines (according to loki_distributor_lines_received_total in /metrics), the bucket is still perfectly empty.

When does Loki start using a provided object store in a default configuration? I'd like to ensure it indeed works.

[Unactived]

Eventually started to fill :D

I'm leaving this opened and still reiterating the very much not fixed issue, that seems like a bug to fix or quirk to document:

Using s3: s3://access_key:secret_key@endpoint:port/bucket and insecure: false, Loki connects with http instead of https.

Additionally specifying endpoint: https://endpoint:port fixes it for some reason. (Got the idea from people in unrelated issues doing that as well).

[chaudum]

Hi @Unactived Thanks for reporting this issue. This is definitely something we should improve documentation on, so I add the docs label.

[GrafanaWriter]

@JStickler - can you please research and assess with @chaudum

[HDegroote]

Posting to inform I had the same issue, and that the advice of specifying insecure: false and endpoint: https://endpoint:port worked. As mentioned, making this explicit in the docs would be helpful. Perhaps as an additional example?

In my case, I was using ceph-based storage (contabo), and getting 403-unauthorized errors:

msg="failed to flush" err="failed to flush chunks: store put chunk: Unauthorized: Unauthorized\n\tstatus code: 401"

[AlyHKafoury]

@GrafanaWriter can I work on this issue ?

[JStickler]

@AlyHKafoury absolutely! It's assigned to me for visibility, but anyone can work on it.

[AlyHKafoury]

I traced this from https://github.com/grafana/loki/blob/main/pkg/storage/chunk/client/aws/s3_storage_client.go
to https://github.com/grafana/loki/blob/main/vendor/github.com/aws/aws-sdk-go/aws/config.go

it seems loki delivers the value correctly it might be a problem with aws sdk version ?. How is the vendor packages managed ?