Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error: "at least one bucket name must be specified" #9824

Open
mac133k opened this issue Jun 29, 2023 · 3 comments
Open

Error: "at least one bucket name must be specified" #9824

mac133k opened this issue Jun 29, 2023 · 3 comments

Comments

@mac133k
Copy link

mac133k commented Jun 29, 2023 •
edited

Describe the bug
After adding named_store_configs block to the storage config Loki failed to start with the following error message:

level=error ts=2023-06-28T13:51:12.95861224Z caller=log.go:171 msg="error running loki" err="at least one bucket name must be specified
github.com/grafana/loki/pkg/storage/chunk/client/aws.buckets
	/drone/src/pkg/storage/chunk/client/aws/s3_storage_client.go:332
github.com/grafana/loki/pkg/storage/chunk/client/aws.NewS3ObjectClient
	/drone/src/pkg/storage/chunk/client/aws/s3_storage_client.go:152
github.com/grafana/loki/pkg/storage.NewObjectClient
	/drone/src/pkg/storage/factory.go:483
github.com/grafana/loki/pkg/storage.NewIndexClient
	/drone/src/pkg/storage/factory.go:294
github.com/grafana/loki/pkg/storage.(*store).storeForPeriod
	/drone/src/pkg/storage/store.go:268
github.com/grafana/loki/pkg/storage.(*store).init
	/drone/src/pkg/storage/store.go:164
github.com/grafana/loki/pkg/storage.NewStore
	/drone/src/pkg/storage/store.go:147
github.com/grafana/loki/pkg/loki.(*Loki).initStore
	/drone/src/pkg/loki/modules.go:655
github.com/grafana/dskit/modules.(*Manager).initModule
	/drone/src/vendor/github.com/grafana/dskit/modules/modules.go:120
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
	/drone/src/vendor/github.com/grafana/dskit/modules/modules.go:92
github.com/grafana/loki/pkg/loki.(*Loki).Run
	/drone/src/pkg/loki/loki.go:457
main.main
	/drone/src/cmd/loki/main.go:110
runtime.main
	/usr/local/go/src/runtime/proc.go:250
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1598
error creating index client
github.com/grafana/loki/pkg/storage.(*store).storeForPeriod
	/drone/src/pkg/storage/store.go:270
github.com/grafana/loki/pkg/storage.(*store).init
	/drone/src/pkg/storage/store.go:164
github.com/grafana/loki/pkg/storage.NewStore
	/drone/src/pkg/storage/store.go:147
github.com/grafana/loki/pkg/loki.(*Loki).initStore
	/drone/src/pkg/loki/modules.go:655
github.com/grafana/dskit/modules.(*Manager).initModule
	/drone/src/vendor/github.com/grafana/dskit/modules/modules.go:120
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
	/drone/src/vendor/github.com/grafana/dskit/modules/modules.go:92
github.com/grafana/loki/pkg/loki.(*Loki).Run
	/drone/src/pkg/loki/loki.go:457
main.main
	/drone/src/cmd/loki/main.go:110
runtime.main
	/usr/local/go/src/runtime/proc.go:250
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1598
error initialising module: store
github.com/grafana/dskit/modules.(*Manager).initModule
	/drone/src/vendor/github.com/grafana/dskit/modules/modules.go:122
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
	/drone/src/vendor/github.com/grafana/dskit/modules/modules.go:92
github.com/grafana/loki/pkg/loki.(*Loki).Run
	/drone/src/pkg/loki/loki.go:457
main.main
	/drone/src/cmd/loki/main.go:110
runtime.main
	/usr/local/go/src/runtime/proc.go:250
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1598"

This error was gone and Loki was able to start after shared_store: s3 was commented out from BoltDB-Shipper and TSDB-Shipper config blocks.

To Reproduce
Steps to reproduce the behavior:

  1. Add named_store block to the config.
  2. Restart Loki to see it failing.
  3. Comment out the shared_store line from storage congs.
  4. Restart Loki without a problem

Expected behavior
I am not sure why commenting out shared_store: s3 helped, but from Loki code it seems like the bucketnames: line was ignored for some reason after introducing named stores. I am also not sure whether removing shared store line from TSDB and BoltDB-Shipper configs has any side effects, but the cluster seems to be working as usual for 2 days since the config change.

Environment:

  • Infrastructure: Microservice mode running Linux VMs with Ceph as S3 storage.

Configuration
Storage config before the change:

common:
  storage: 
    s3:
      access_key_id: kkv5JZnR88LwPdRKNlNt80JmDpakDNHNKDaBSvz7
      bucketnames: loki-logs
      endpoint: ceph.devmain.net:9000
      http_config:
        idle_conn_timeout: 10m
      s3forcepathstyle: true
      secret_access_key: qK3JTwuh5Upy3AXQXc2v4yRVnOEnIid01gZfrum8
storage_config:
  boltdb_shipper:
    active_index_directory: /data1/loki/boltdb
    cache_location: /data1/loki/boltdb-cache
    cache_ttl: 24h
    index_gateway_client:
      grpc_client_config:
        tls_ca_path: /etc/loki/ca.crt
        tls_cert_path: /etc/loki/client.crt
        tls_enabled: true
        tls_key_path: /etc/loki/client.key
      log_gateway_requests: true
      server_address: igw1-loki.devdomain.net:9095
    query_ready_num_days: 1
    shared_store: s3
  tsdb_shipper:
    active_index_directory: /data1/loki/tsdb-index
    cache_location: /data1/loki/tsdb-cache
    cache_ttl: 24h
    index_gateway_client:
      grpc_client_config:
        tls_ca_path: /etc/loki/ca.crt
        tls_cert_path: /etc/loki/client.crt
        tls_enabled: true
        tls_key_path: /etc/loki/client.key
      log_gateway_requests: true
      server_address: igw2-loki.devdomain.net:9095
    query_ready_num_days: 1
    shared_store: s3

Storage config after the change:

storage_config:
  boltdb_shipper:
    active_index_directory: /data1/loki/boltdb
    cache_location: /data1/loki/boltdb-cache
    cache_ttl: 24h
    index_gateway_client:
      grpc_client_config:
        tls_ca_path: /etc/loki/ca.crt
        tls_cert_path: /etc/loki/client.crt
        tls_enabled: true
        tls_key_path: /etc/loki/client.key
      log_gateway_requests: true
      server_address: igw3-loki.devdomain.net:9095
    query_ready_num_days: 1
  tsdb_shipper:
    active_index_directory: /data1/loki/tsdb-index
    cache_location: /data1/loki/tsdb-cache
    cache_ttl: 24h
    index_gateway_client:
      grpc_client_config:
        tls_ca_path: /etc/loki/ca.crt
        tls_cert_path: /etc/loki/client.crt
        tls_enabled: true
        tls_key_path: /etc/loki/client.key
      log_gateway_requests: true
      server_address: igw2-loki.devdomain.net:9095
    query_ready_num_days: 1
  named_stores:
    aws:
      cceph-20220101:
        access_key_id: kkv5JZnR88LwPdRKNlNt80JmDpakDNHNKDaBSvz7
        bucketnames: loki-logs-atdev1
        endpoint: ceph.devdomain.net:9000
        http_config:
          idle_conn_timeout: 10m
        s3forcepathstyle: true
        secret_access_key: qK3JTwuh5Upy3AXQXc2v4yRVnOEnIid01gZfrum8
        signature_version: v4
        storage_class: STANDARD
      cceph-20230511:
        access_key_id: kkv5JZnR88LwPdRKNlNt80JmDpakDNHNKDaBSvz7
        bucketnames: loki-logs-atdev1
        endpoint: ceph.devdomain.net:9000
        http_config:
          idle_conn_timeout: 10m
        s3forcepathstyle: true
        secret_access_key: qK3JTwuh5Upy3AXQXc2v4yRVnOEnIid01gZfrum8
        signature_version: v4
        storage_class: STANDARD
      cceph-20230608:
        access_key_id: kkv5JZnR88LwPdRKNlNt80JmDpakDNHNKDaBSvz7
        bucketnames: loki-logs-atdev1
        endpoint: ceph.devdomain.net:9000
        http_config:
          idle_conn_timeout: 10m
        s3forcepathstyle: true
        secret_access_key: qK3JTwuh5Upy3AXQXc2v4yRVnOEnIid01gZfrum8
        signature_version: v4
        storage_class: STANDARD
schema_config:
  configs:
  - chunks:
      prefix: chunk_
    from: '2022-01-01'
    index:
      period: 24h
      prefix: index_
    object_store: cceph-20220101
    schema: v11
    store: boltdb-shipper
  - chunks:
      prefix: chunk_
    from: '2023-05-11'
    index:
      period: 24h
      prefix: index_
    object_store: cceph-20230511
    row_shards: 64
    schema: v12
    store: boltdb-shipper
  - chunks:
      prefix: chunk_
    from: '2023-06-08'
    index:
      period: 24h
      prefix: index_
    object_store: cceph-20230608
    schema: v12
    store: tsdb
@ashwanthgoli
Copy link
Contributor

Was the common storage config removed after named_stores section is added?
If yes, setting shared_store to s3 is forcing the index shipper to init an object client with the config provided under common storage config (which is now empty)

@mac133k
Copy link
Author

mac133k commented Jun 30, 2023

Yes, I removed the common storage config before introducing names stores.

So I guess it means the described behaviour is expected. If that is the case and you do not have any questions for me please close this issue.

@mac133k
Copy link
Author

mac133k commented Jul 3, 2023

I think it would be helpful to include more information in the documentation about the shared_store param and how or when to use it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ashwanthgoli @mac133k