Loki Does not Create Tables on DynamoDB

[fayas-akram]

Loki Does not Create Tables on DynamoDB
I have configured Loki to Store chunks in s3 and indexes in DynamoDB. Each Day Loki looks for a Different table name and which is not created and therefore, it fails to show logs on my dashboard.

my index prefix is loki_index_
Loki looks for different index suffixes each day. Example loki_index_18997, loki_index_18998, loki_index_18999, etc

For the record, I used roles for my instance and it was given full access with loki_index_* resources

My Loki Configuration is as follows

schema_config:
  configs:
    - from: 2020-04-15
      store: aws
      object_store: s3
      schema: v11
      index:
        prefix: loki_index_
        period: 24h

storage_config:
  aws:
    s3: s3://us-east-2/my-bucket
    s3forcepathstyle: true
    dynamodb:
      dynamodb_url: dynamodb://us-east-2

compactor:
  working_directory: /loki/compactor
  shared_store: s3
  compaction_interval: 5m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

**Error **

level=error ts=2022-01-07T04:52:20.345618928Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found"
level=error ts=2022-01-07T04:52:20.34734157Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found\n"
level=error ts=2022-01-07T04:52:20.361839228Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found"
level=error ts=2022-01-07T04:52:20.362431505Z caller=retry.go:73 org_id=fake msg="error processing request" try=1 err="rpc error: code = Code(500) desc = QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found\n"
level=error ts=2022-01-07T04:52:20.374077785Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found"
level=error ts=2022-01-07T04:52:20.374709622Z caller=retry.go:73 org_id=fake msg="error processing request" try=2 err="rpc error: code = Code(500) desc = QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found\n"
level=error ts=2022-01-07T04:52:20.385779626Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found"

[cyriltovena]

You need to activate table_manager, the compactor is not compatible with NoSQL index.

[fayas-akram]

@cyriltovena Still not working and I have removed the compactor part and add table manager with 24 hours retention period and it did not work

Configuration

schema_config:
  configs:
    - from: 2018-04-15
      store: aws
      object_store: s3
      schema: v11
      index:
        prefix: loki_index_
        period: 24h

storage_config:
  aws:
    s3: s3://us-east-1/my-bucket
    s3forcepathstyle: true
    dynamodb:
      dynamodb_url: dynamodb://us-east-1

table_manager:
  retention_deletes_enabled: true
  retention_period: 24h
  index_tables_provisioning:
    provisioned_write_throughput: 10
    provisioned_read_throughput: 10

Same error - Index not found

level=error ts=2022-01-07T09:44:35.614402403Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found"
level=error ts=2022-01-07T09:44:35.616080133Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found\n"
level=error ts=2022-01-07T09:44:35.62789012Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found"
level=error ts=2022-01-07T09:44:35.62842623Z caller=retry.go:73 org_id=fake msg="error processing request" try=1 err="rpc error: code = Code(500) desc = QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found\n"
level=error ts=2022-01-07T09:44:35.639516483Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found"
level=error ts=2022-01-07T09:44:35.640028302Z caller=retry.go:73 org_id=fake msg="error processing request" try=2 err="rpc error: code = Code(500) desc = QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found\n"
level=error ts=2022-01-07T09:44:35.652181158Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found"
level=error ts=2022-01-07T09:44:35.652662619Z caller=retry.go:73 org_id=fake msg="error processing request" try=3 err="rpc error: code = Code(500) desc = QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found\n"
level=error ts=2022-01-07T09:44:35.66374471Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found"
level=error ts=2022-01-07T09:44:35.664210449Z caller=retry.go:73 org_id=fake msg="error processing request" try=4 err="rpc error: code = Code(500) desc = QueryPages error: table=loki_index_18999: ResourceNotFoundException: Requested resource not found\n"

Also, please let me know what is the recommended DynamoDb Read/Write ratio for production and I am planning to store my logs for a year

[cyriltovena]

Do you have a pod that is running as the table-manager ?

[fayas-akram]

@cyriltovena Still not. For Prod, Are you recommend any suggestion
I am currently testing this stuff on Staging environments. Let me know why the table_manger settings fail to create tables on DynamoDB?
Please note that if configure Loki for boltdb for index and it works
Only fails to create DynanmoDB tables
Used AWS IAM Role With FullAccess too

[robcoward]

I have found the same problem running the --target=all single binary deployment (not containerised). No sign of errors being logged, loki doesnt appear to even try to create the tables.

[fayas-akram]

Guys Any Updates on this

[chaudum]

I have found the same problem running the --target=all single binary deployment (not containerised). No sign of errors being logged, loki doesnt appear to even try to create the tables.

The DynamoDB tables are created by the table-manager target, however table-manager is not part of the all target.

When starting the table manager, you should see log lines like these:

level=info ts=2022-01-13T11:01:11.913914962Z caller=table_manager.go:322 msg="synching tables" expected_tables=...
level=info ts=2022-01-13T11:01:11.91492943Z caller=table_manager.go:473 msg="creating table" table=...

[mpekhota]

I have found the same problem running the --target=all single binary deployment (not containerised). No sign of errors being logged, loki doesnt appear to even try to create the tables.

Could you give an example of your config. I use grafana/loki chart to run Loki and it also doesn't create a database, I tried to create a table manually but I don't understand where in the config I need to specify its name.

[andino6869]

"-target=all,table-manager" could be the solution as it did in my case.

(From https://github.com/grafana/loki/blob/main/docs/sources/upgrading/_index.md#the-single-binary-no-longer-runs-a-table-manager)

[mpekhota]

"-target=all,table-manager" could be the solution as it did in my case.

(From https://github.com/grafana/loki/blob/main/docs/sources/upgrading/_index.md#the-single-binary-no-longer-runs-a-table-manager)

Yes, it helped me. I needed to add in default values extraArgs like

extraArgs:
target: all,table-manager

Thanks!

[RonaldoNazo]

If anyone is still struggling with this one , on helm config add this code :

loki:
  extraArgs:
    target: "all,table-manager"