Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ResourceNotFoundException: Requested resource not found when settings AWS as backend #5021

Closed
weastur opened this issue Dec 30, 2021 · 12 comments
Closed

ResourceNotFoundException: Requested resource not found when settings AWS as backend #5021

weastur opened this issue Dec 30, 2021 · 12 comments
Labels
stale A stale issue or PR that will automatically be closed.

Comments

@weastur
Copy link

weastur commented Dec 30, 2021

I'm using 2.4.1 version of Loki/promtail running in simple scalable mode as described here. I'm trying to store chunks in AWS S3, index in AWS DynamoDB, Hash ring in Consul.

My config:

auth_enabled: false

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 8080
  grpc_listen_address: 0.0.0.0
  grpc_listen_port: 9095
  register_instrumentation: true

distributor:
  ring:
    kvstore:
      store: consul
      consul:
        acl_token: -=REDACTED=-

ingester:
  lifecycler:
    ring:
      kvstore:
        store: consul
        consul:
          acl_token: -=REDACTED=-
      replication_factor: 3

schema_config:
  configs:
  - from: 2021-12-27
    store: aws-dynamo
    object_store: aws
    schema: v11
    index:
      prefix: loki_

storage_config:
  aws:
    s3: s3://-=REDACTED=-:-=REDACTED=-@ca-central-1/s3-ca-loki
    dynamodb:
      dynamodb_url: dynamodb://-=REDACTED=-:-=REDACTED=-@ca-central-1

ruler:
  storage:
    type: s3
    s3:
      s3: s3://-=REDACTED=-:-=REDACTED=-@ca-central-1/s3-ca-loki-ruler
  rule_path: /opt/loki/rule

Reader logs:

Dec 30 11:31:27 tor1-infra-loki-reader-1 systemd[1]: Started Loki read service.
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: purgeplan.DeletePlan
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: purgeplan.ChunksGroup
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: purgeplan.ChunkDetails
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: purgeplan.Interval
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.PutChunksRequest
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.GetChunksRequest
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.GetChunksResponse
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.Chunk
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.ChunkID
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.DeleteTableRequest
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.DescribeTableRequest
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.WriteBatch
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.WriteIndexRequest
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.DeleteIndexRequest
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.QueryIndexResponse
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.Row
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.IndexEntry
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.QueryIndexRequest
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.UpdateTableRequest
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.DescribeTableResponse
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.CreateTableRequest
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.TableDesc
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.TableDesc.TagsEntry
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.ListTablesResponse
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: grpc.Labels
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: storage.Entry
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: proto: duplicate proto type registered: storage.ReadBatch
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.467191147Z caller=main.go:94 msg="Starting Loki" version="(version=2.4.1, branch=HEAD, revision=f61a4d261)"
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.467589654Z caller=server.go:260 http=[::]:8080 grpc=[::]:9095 msg="server listening on addresses"
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.467836308Z caller=modules.go:682 msg="Not using boltdb-shipper index, not starting compactor"
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.467852718Z caller=modules.go:432 msg="initializing query frontend tripperware"
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=warn ts=2021-12-30T11:31:27.468663141Z caller=experimental.go:19 msg="experimental feature in use" feature="In-memory (FIFO) cache"
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.469103469Z caller=modules.go:452 msg="initializing query frontend" config="{Handler:{LogQueriesLongerThan:0s MaxBodySize:10485760 QueryStatsEnabled:false} FrontendV1:{MaxOutstandingPerTenant:100 QuerierForgetDelay:0s} FrontendV2:{SchedulerAddress: DNSLookupPeriod:10s WorkerConcurrency:5 GRPCClientConfig:{MaxRecvMsgSize:104857600 MaxSendMsgSize:16777216 GRPCCompression: RateLimit:0 RateLimitBurst:0 BackoffOnRatelimits:false BackoffConfig:{MinBackoff:100ms MaxBackoff:10s MaxRetries:10} TLSEnabled:false TLS:{CertPath: KeyPath: CAPath: ServerName: InsecureSkipVerify:false}} InfNames:[eth0 en0 lo] Addr: Port:0} CompressResponses:false DownstreamURL: TailProxyURL:}"
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.469254251Z caller=modules.go:479 msg="using query frontend" version=v2
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.469665438Z caller=worker_service.go:149 msg="determining if querier is running as standalone target" runningStandalone=false queryFrontendEnabled=false queryScheduleEnabled=false readEnabled=true allEnabled=false
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.469713409Z caller=worker.go:109 msg="Starting querier worker using query-scheduler and scheduler ring for addresses"
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.469846541Z caller=mapper.go:46 msg="cleaning up mapped rules directory" path=/opt/loki/rule
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.471065421Z caller=module_service.go:54 msg="module waiting for initialization" module=store waiting_for=ingester-querier
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.471242674Z caller=module_service.go:54 msg="module waiting for initialization" module=querier waiting_for=ingester-querier
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.471247384Z caller=module_service.go:54 msg="module waiting for initialization" module=ruler waiting_for=ingester-querier
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.471279205Z caller=module_service.go:64 msg=initialising module=server
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.471297265Z caller=module_service.go:64 msg=initialising module=memberlist-kv
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.471337266Z caller=module_service.go:54 msg="module waiting for initialization" module=query-frontend-tripperware waiting_for=server
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.471353536Z caller=module_service.go:54 msg="module waiting for initialization" module=ring waiting_for=memberlist-kv
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.471361466Z caller=module_service.go:54 msg="module waiting for initialization" module=ring waiting_for=server
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.471395797Z caller=module_service.go:54 msg="module waiting for initialization" module=ingester-querier waiting_for=memberlist-kv
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.471410037Z caller=module_service.go:54 msg="module waiting for initialization" module=ingester-querier waiting_for=ring
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.471421327Z caller=module_service.go:54 msg="module waiting for initialization" module=query-scheduler waiting_for=memberlist-kv
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.471428637Z caller=module_service.go:54 msg="module waiting for initialization" module=query-scheduler waiting_for=server
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.471438657Z caller=module_service.go:54 msg="module waiting for initialization" module=query-frontend waiting_for=query-scheduler
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.471500238Z caller=module_service.go:64 msg=initialising module=query-scheduler
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.471659781Z caller=module_service.go:64 msg=initialising module=query-frontend-tripperware
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.471826594Z caller=module_service.go:64 msg=initialising module=ring
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.476826356Z caller=module_service.go:54 msg="module waiting for initialization" module=ingester-querier waiting_for=server
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.476852447Z caller=module_service.go:64 msg=initialising module=ingester-querier
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.476874037Z caller=module_service.go:54 msg="module waiting for initialization" module=ruler waiting_for=memberlist-kv
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.476884707Z caller=module_service.go:54 msg="module waiting for initialization" module=ruler waiting_for=ring
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.476892927Z caller=module_service.go:54 msg="module waiting for initialization" module=ruler waiting_for=server
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.476902267Z caller=module_service.go:54 msg="module waiting for initialization" module=ruler waiting_for=store
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.476895017Z caller=module_service.go:54 msg="module waiting for initialization" module=querier waiting_for=memberlist-kv
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.476904567Z caller=module_service.go:54 msg="module waiting for initialization" module=store waiting_for=memberlist-kv
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.476923908Z caller=module_service.go:54 msg="module waiting for initialization" module=querier waiting_for=query-scheduler
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.476944238Z caller=module_service.go:54 msg="module waiting for initialization" module=store waiting_for=ring
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.476958598Z caller=module_service.go:54 msg="module waiting for initialization" module=store waiting_for=server
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.476969988Z caller=module_service.go:64 msg=initialising module=store
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.477003429Z caller=module_service.go:64 msg=initialising module=ruler
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.47703321Z caller=ruler.go:442 msg="ruler up and running"
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:27.4770461Z caller=ruler.go:480 msg="syncing rules" reason=initial
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.4770443Z caller=basic_lifecycler.go:252 msg="instance not found in the ring" instance=tor1-infra-loki-reader-1 ring=scheduler
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.47706923Z caller=basic_lifecycler_delegates.go:63 msg="not loading tokens from file, tokens file path is empty"
Dec 30 11:31:27 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:27.481408272Z caller=scheduler.go:611 msg="waiting until scheduler is JOINING in the ring"
Dec 30 11:31:29 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:29.446110716Z caller=scheduler.go:615 msg="scheduler is JOINING in the ring"
Dec 30 11:31:29 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:29.457797958Z caller=scheduler.go:625 msg="waiting until scheduler is ACTIVE in the ring"
Dec 30 11:31:29 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:29.626900707Z caller=scheduler.go:629 msg="scheduler is ACTIVE in the ring"
Dec 30 11:31:29 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:29.627016059Z caller=module_service.go:54 msg="module waiting for initialization" module=query-frontend waiting_for=server
Dec 30 11:31:29 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:29.627019639Z caller=module_service.go:54 msg="module waiting for initialization" module=querier waiting_for=ring
Dec 30 11:31:29 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:29.627038329Z caller=module_service.go:54 msg="module waiting for initialization" module=query-frontend waiting_for=memberlist-kv
Dec 30 11:31:29 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:29.627045009Z caller=module_service.go:54 msg="module waiting for initialization" module=querier waiting_for=server
Dec 30 11:31:29 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:29.627049629Z caller=module_service.go:54 msg="module waiting for initialization" module=query-frontend waiting_for=query-frontend-tripperware
Dec 30 11:31:29 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:29.627055739Z caller=module_service.go:54 msg="module waiting for initialization" module=querier waiting_for=store
Dec 30 11:31:29 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:29.627061359Z caller=module_service.go:64 msg=initialising module=query-frontend
Dec 30 11:31:29 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:29.627068479Z caller=module_service.go:64 msg=initialising module=querier
Dec 30 11:31:29 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:29.627239402Z caller=loki.go:318 msg="Loki started"
Dec 30 11:31:30 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:30.414692256Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-2
Dec 30 11:31:30 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:30.414733826Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-2
Dec 30 11:31:30 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:30.414749026Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-2
Dec 30 11:31:30 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:30.414692266Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-2
Dec 30 11:31:30 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:30.414751397Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-2
Dec 30 11:31:30 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:30.414718356Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-2
Dec 30 11:31:30 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:30.414763987Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-2
Dec 30 11:31:30 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:30.414771437Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-2
Dec 30 11:31:30 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:30.414778537Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-2
Dec 30 11:31:30 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:30.414727256Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-2
Dec 30 11:31:32 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:32.627092753Z caller=scheduler.go:663 msg="this scheduler is in the ReplicationSet, will now accept requests."
Dec 30 11:31:32 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:32.627169474Z caller=ring_watcher.go:89 component=querier-scheduler-worker msg="adding connection to address: 169.53.181.213:9095"
Dec 30 11:31:32 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:32.627201195Z caller=worker.go:205 msg="adding connection" addr=169.53.181.213:9095
Dec 30 11:31:32 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:32.62747947Z caller=ring_watcher.go:89 component=querier-scheduler-worker msg="adding connection to address: 169.53.181.221:9095"
Dec 30 11:31:32 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:32.62749402Z caller=worker.go:205 msg="adding connection" addr=169.53.181.221:9095
Dec 30 11:31:32 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:32.62930306Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-1
Dec 30 11:31:32 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:32.629360381Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-1
Dec 30 11:31:32 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:32.629629985Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-1
Dec 30 11:31:32 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:32.629776757Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-1
Dec 30 11:31:32 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:32.629795118Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-1
Dec 30 11:31:32 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:32.629885529Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-1
Dec 30 11:31:32 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:32.62994613Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-1
Dec 30 11:31:32 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:32.630106843Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-1
Dec 30 11:31:32 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:32.630624091Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-1
Dec 30 11:31:32 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:32.630815354Z caller=scheduler.go:438 msg="querier connected" querier=tor1-infra-loki-reader-1
Dec 30 11:31:37 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:37.414595917Z caller=scheduler.go:332 msg="frontend connected" address=169.53.181.221:9095
Dec 30 11:31:37 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:37.414622408Z caller=scheduler.go:332 msg="frontend connected" address=169.53.181.221:9095
Dec 30 11:31:37 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:37.414591067Z caller=scheduler.go:332 msg="frontend connected" address=169.53.181.221:9095
Dec 30 11:31:37 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:37.414623958Z caller=scheduler.go:332 msg="frontend connected" address=169.53.181.221:9095
Dec 30 11:31:37 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:37.414633398Z caller=scheduler.go:332 msg="frontend connected" address=169.53.181.221:9095
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:39.627873958Z caller=ring_watcher.go:89 component=frontend-scheduler-worker msg="adding connection to address: 169.53.181.213:9095"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:39.627956229Z caller=frontend_scheduler_worker.go:100 msg="adding connection to scheduler" addr=169.53.181.213:9095
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:39.628089071Z caller=ring_watcher.go:89 component=frontend-scheduler-worker msg="adding connection to address: 169.53.181.221:9095"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=info ts=2021-12-30T11:31:39.628103762Z caller=frontend_scheduler_worker.go:100 msg="adding connection to scheduler" addr=169.53.181.221:9095
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:39.62917785Z caller=scheduler.go:332 msg="frontend connected" address=169.53.181.213:9095
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:39.62921084Z caller=scheduler.go:332 msg="frontend connected" address=169.53.181.213:9095
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:39.629256341Z caller=scheduler.go:332 msg="frontend connected" address=169.53.181.213:9095
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:39.629283501Z caller=scheduler.go:332 msg="frontend connected" address=169.53.181.213:9095
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:39.629377853Z caller=scheduler.go:332 msg="frontend connected" address=169.53.181.213:9095
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:39.798187506Z caller=grpc_logging.go:41 method=/frontendv2pb.FrontendForQuerier/QueryResult duration=55.301µs msg="gRPC (success)"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=error ts=2021-12-30T11:31:39.79840947Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = QueryPages error: table=loki_: ResourceNotFoundException: Requested resource not found\n"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.800780419Z caller=spanlogger.go:87 org_id=fake method=SeriesStore.LabelNamesForMetricName level=debug metric=logs
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.80084099Z caller=spanlogger.go:87 org_id=fake method=SeriesStore.lookupSeriesByMetricNameMatchers level=debug metricName=logs matchers=0
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.800906061Z caller=spanlogger.go:87 org_id=fake method=Store.lookupIdsByMetricNameMatcher level=debug metricName=logs matcher=nil
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.800939631Z caller=spanlogger.go:87 org_id=fake method=Store.lookupIdsByMetricNameMatcher level=debug matcher=nil queries=16
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.800953532Z caller=spanlogger.go:87 org_id=fake method=Store.lookupIdsByMetricNameMatcher level=debug matcher=nil filteredQueries=16
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.801013133Z caller=spanlogger.go:87 org_id=fake method=cachingIndexClient.cacheFetch level=debug hits=0 misses=16
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=error ts=2021-12-30T11:31:39.855753935Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="QueryPages error: table=loki_: ResourceNotFoundException: Requested resource not found"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:39.856706231Z caller=grpc_logging.go:41 method=/frontendv2pb.FrontendForQuerier/QueryResult duration=54.781µs msg="gRPC (success)"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=error ts=2021-12-30T11:31:39.856761952Z caller=retry.go:73 org_id=fake msg="error processing request" try=1 err="rpc error: code = Code(500) desc = QueryPages error: table=loki_: ResourceNotFoundException: Requested resource not found\n"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.858099154Z caller=spanlogger.go:87 org_id=fake method=SeriesStore.LabelNamesForMetricName level=debug metric=logs
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.858150135Z caller=spanlogger.go:87 org_id=fake method=SeriesStore.lookupSeriesByMetricNameMatchers level=debug metricName=logs matchers=0
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.858218106Z caller=spanlogger.go:87 org_id=fake method=Store.lookupIdsByMetricNameMatcher level=debug metricName=logs matcher=nil
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.858244606Z caller=spanlogger.go:87 org_id=fake method=Store.lookupIdsByMetricNameMatcher level=debug matcher=nil queries=16
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.858256796Z caller=spanlogger.go:87 org_id=fake method=Store.lookupIdsByMetricNameMatcher level=debug matcher=nil filteredQueries=16
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.858336368Z caller=spanlogger.go:87 org_id=fake method=cachingIndexClient.cacheFetch level=debug hits=0 misses=16
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=error ts=2021-12-30T11:31:39.872815536Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="QueryPages error: table=loki_: ResourceNotFoundException: Requested resource not found"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:39.873256744Z caller=grpc_logging.go:41 method=/frontendv2pb.FrontendForQuerier/QueryResult duration=59.721µs msg="gRPC (success)"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=error ts=2021-12-30T11:31:39.873344545Z caller=retry.go:73 org_id=fake msg="error processing request" try=2 err="rpc error: code = Code(500) desc = QueryPages error: table=loki_: ResourceNotFoundException: Requested resource not found\n"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.874408693Z caller=spanlogger.go:87 org_id=fake method=SeriesStore.LabelNamesForMetricName level=debug metric=logs
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.874440693Z caller=spanlogger.go:87 org_id=fake method=SeriesStore.lookupSeriesByMetricNameMatchers level=debug metricName=logs matchers=0
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.874483984Z caller=spanlogger.go:87 org_id=fake method=Store.lookupIdsByMetricNameMatcher level=debug metricName=logs matcher=nil
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.874519455Z caller=spanlogger.go:87 org_id=fake method=Store.lookupIdsByMetricNameMatcher level=debug matcher=nil queries=16
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.874533885Z caller=spanlogger.go:87 org_id=fake method=Store.lookupIdsByMetricNameMatcher level=debug matcher=nil filteredQueries=16
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.874584376Z caller=spanlogger.go:87 org_id=fake method=cachingIndexClient.cacheFetch level=debug hits=0 misses=16
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=error ts=2021-12-30T11:31:39.88940077Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="QueryPages error: table=loki_: ResourceNotFoundException: Requested resource not found"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:39.890508048Z caller=grpc_logging.go:41 method=/frontendv2pb.FrontendForQuerier/QueryResult duration=49.301µs msg="gRPC (success)"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=error ts=2021-12-30T11:31:39.89061037Z caller=retry.go:73 org_id=fake msg="error processing request" try=3 err="rpc error: code = Code(500) desc = QueryPages error: table=loki_: ResourceNotFoundException: Requested resource not found\n"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.892533851Z caller=spanlogger.go:87 org_id=fake method=SeriesStore.LabelNamesForMetricName level=debug metric=logs
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.892564492Z caller=spanlogger.go:87 org_id=fake method=SeriesStore.lookupSeriesByMetricNameMatchers level=debug metricName=logs matchers=0
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.892609653Z caller=spanlogger.go:87 org_id=fake method=Store.lookupIdsByMetricNameMatcher level=debug metricName=logs matcher=nil
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.892642273Z caller=spanlogger.go:87 org_id=fake method=Store.lookupIdsByMetricNameMatcher level=debug matcher=nil queries=16
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.892654123Z caller=spanlogger.go:87 org_id=fake method=Store.lookupIdsByMetricNameMatcher level=debug matcher=nil filteredQueries=16
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: ts=2021-12-30T11:31:39.892705794Z caller=spanlogger.go:87 org_id=fake method=cachingIndexClient.cacheFetch level=debug hits=0 misses=16
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=error ts=2021-12-30T11:31:39.90763013Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="QueryPages error: table=loki_: ResourceNotFoundException: Requested resource not found"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:39.907928175Z caller=grpc_logging.go:41 method=/frontendv2pb.FrontendForQuerier/QueryResult duration=40.41µs msg="gRPC (success)"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=error ts=2021-12-30T11:31:39.908030757Z caller=retry.go:73 org_id=fake msg="error processing request" try=4 err="rpc error: code = Code(500) desc = QueryPages error: table=loki_: ResourceNotFoundException: Requested resource not found\n"
Dec 30 11:31:39 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=warn ts=2021-12-30T11:31:39.908158629Z caller=logging.go:72 traceID=67c1d775bf03ea6a orgID=fake msg="GET /loki/api/v1/label?start=1640863299382000000 (500) 184.40158ms Response: \"QueryPages error: table=loki_: ResourceNotFoundException: Requested resource not found\\n\" ws: false; Accept: application/json, text/plain, */*; Accept-Encoding: gzip; Accept-Language: en-GB,en;q=0.9; Cdn-Loop: cloudflare; Cf-Connecting-Ip: 159.69.70.237; Cf-Ipcountry: DE; Cf-Ray: 6c5af16cdf3b8bc9-YYZ; Cf-Visitor: {\"scheme\":\"https\"}; User-Agent: Grafana/8.3.3; X-Forwarded-For: 159.69.70.237, 159.69.70.237, 127.0.0.1, 127.0.0.1; X-Grafana-Nocache: true; X-Grafana-Org-Id: 2; X-Real-Ip: 159.69.70.237; "
Dec 30 11:31:42 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:42.414275318Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=64.962µs msg="gRPC (success)"
Dec 30 11:31:44 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:44.627780941Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=69.991µs msg="gRPC (success)"
Dec 30 11:31:47 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:47.41418322Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=107.382µs msg="gRPC (success)"
Dec 30 11:31:49 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:49.628551528Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=50.11µs msg="gRPC (success)"
Dec 30 11:31:52 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:52.413713685Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=58.771µs msg="gRPC (success)"
Dec 30 11:31:54 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:54.628537887Z caller=grpc_logging.go:41 duration=68.301µs method=/grpc.health.v1.Health/Check msg="gRPC (success)"
Dec 30 11:31:57 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:57.413098815Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=61.531µs msg="gRPC (success)"
Dec 30 11:31:59 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:31:59.628582419Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=88.452µs msg="gRPC (success)"
Dec 30 11:32:02 tor1-infra-loki-reader-1 loki-linux-amd64[1746994]: level=debug ts=2021-12-30T11:32:02.413364688Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=90.862µs msg="gRPC (success)"

Writer logs:

Dec 30 11:29:24 tor1-infra-loki-writer-1 systemd[1]: Started Loki write service.
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: purgeplan.DeletePlan
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: purgeplan.ChunksGroup
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: purgeplan.ChunkDetails
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: purgeplan.Interval
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.PutChunksRequest
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.GetChunksRequest
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.GetChunksResponse
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.Chunk
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.ChunkID
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.DeleteTableRequest
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.DescribeTableRequest
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.WriteBatch
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.WriteIndexRequest
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.DeleteIndexRequest
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.QueryIndexResponse
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.Row
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.IndexEntry
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.QueryIndexRequest
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.UpdateTableRequest
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.DescribeTableResponse
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.CreateTableRequest
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.TableDesc
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.TableDesc.TagsEntry
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.ListTablesResponse
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: grpc.Labels
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: storage.Entry
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: proto: duplicate proto type registered: storage.ReadBatch
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.581405024Z caller=main.go:94 msg="Starting Loki" version="(version=2.4.1, branch=HEAD, revision=f61a4d261)"
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.581775503Z caller=server.go:260 http=[::]:8080 grpc=[::]:9095 msg="server listening on addresses"
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=warn ts=2021-12-30T11:29:24.58207216Z caller=experimental.go:19 msg="experimental feature in use" feature="In-memory (FIFO) cache"
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.584784277Z caller=module_service.go:64 msg=initialising module=memberlist-kv
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.584808708Z caller=module_service.go:64 msg=initialising module=store
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:24.584844019Z caller=module_service.go:54 msg="module waiting for initialization" module=distributor waiting_for=ring
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:24.584827668Z caller=module_service.go:54 msg="module waiting for initialization" module=ring waiting_for=memberlist-kv
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:24.58489086Z caller=module_service.go:54 msg="module waiting for initialization" module=ingester waiting_for=memberlist-kv
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:24.5849074Z caller=module_service.go:54 msg="module waiting for initialization" module=ingester waiting_for=server
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:24.58488789Z caller=module_service.go:54 msg="module waiting for initialization" module=ring waiting_for=server
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.584866989Z caller=module_service.go:64 msg=initialising module=server
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.586683734Z caller=module_service.go:64 msg=initialising module=ring
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:24.587265299Z caller=module_service.go:54 msg="module waiting for initialization" module=ingester waiting_for=store
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.5873424Z caller=module_service.go:64 msg=initialising module=ingester
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.587373931Z caller=ingester.go:353 msg="recovering from checkpoint"
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:24.590328885Z caller=module_service.go:54 msg="module waiting for initialization" module=distributor waiting_for=server
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:24.590380116Z caller=module_service.go:54 msg="module waiting for initialization" module=distributor waiting_for=memberlist-kv
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.590398406Z caller=module_service.go:64 msg=initialising module=distributor
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.590502559Z caller=lifecycler.go:540 msg="not loading tokens from file, tokens file path is empty"
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.592467468Z caller=lifecycler.go:569 msg="instance not found in ring, adding with no tokens" ring=distributor
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:24.596630551Z caller=lifecycler.go:412 msg="JoinAfter expired" ring=distributor
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.596679542Z caller=lifecycler.go:416 msg="auto-joining cluster after timeout" ring=distributor
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.620708738Z caller=ingester.go:369 msg="recovered WAL checkpoint recovery finished" elapsed=33.343937ms errors=false
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.62078292Z caller=ingester.go:375 msg="recovering from WAL"
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.625399694Z caller=ingester.go:391 msg="WAL segment recovery finished" elapsed=38.034683ms errors=false
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.625468415Z caller=ingester.go:339 msg="closing recoverer"
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.625489656Z caller=ingester.go:347 msg="WAL recovery finished" time=38.127825ms
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.625537397Z caller=lifecycler.go:540 msg="not loading tokens from file, tokens file path is empty"
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.625787663Z caller=wal.go:156 msg=started component=wal
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.625864235Z caller=loki.go:318 msg="Loki started"
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.629944246Z caller=lifecycler.go:569 msg="instance not found in ring, adding with no tokens" ring=ingester
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:24.633122545Z caller=lifecycler.go:412 msg="JoinAfter expired" ring=ingester
Dec 30 11:29:24 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=info ts=2021-12-30T11:29:24.633192157Z caller=lifecycler.go:416 msg="auto-joining cluster after timeout" ring=ingester
Dec 30 11:29:38 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:38.071068717Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=60.091µs msg="gRPC (success)"
Dec 30 11:29:38 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:38.789966879Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=57.872µs msg="gRPC (success)"
Dec 30 11:29:53 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:53.074315725Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=73.642µs msg="gRPC (success)"
Dec 30 11:29:53 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:53.788321175Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=61.292µs msg="gRPC (success)"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.627057285Z caller=flush.go:217 msg="flushing stream" userid=fake fp=19b82441ff5a35f8 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.62724894Z caller=flush.go:217 msg="flushing stream" userid=fake fp=de583670c827868e immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.627419724Z caller=flush.go:217 msg="flushing stream" userid=fake fp=0fa6fee3bbc6e1b4 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.627485296Z caller=flush.go:217 msg="flushing stream" userid=fake fp=49b242d327f9eb4c immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.627501626Z caller=flush.go:217 msg="flushing stream" userid=fake fp=ad750f1a779229ba immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.627516726Z caller=flush.go:217 msg="flushing stream" userid=fake fp=4aa893e362e4e3d3 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.627530967Z caller=flush.go:217 msg="flushing stream" userid=fake fp=fe2e86dec2d4e4c5 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.627546097Z caller=flush.go:217 msg="flushing stream" userid=fake fp=0f519842ca444c02 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.627560658Z caller=flush.go:217 msg="flushing stream" userid=fake fp=3f2aa0b97cb108cf immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.627575368Z caller=flush.go:217 msg="flushing stream" userid=fake fp=8e5fa644888a63f1 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.627592228Z caller=flush.go:217 msg="flushing stream" userid=fake fp=e566e6e63aed3c50 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.627606659Z caller=flush.go:217 msg="flushing stream" userid=fake fp=bf13eacd8232e849 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.627620219Z caller=flush.go:217 msg="flushing stream" userid=fake fp=f833b54bd7f90806 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.766443729Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.766515081Z caller=flush.go:217 msg="flushing stream" userid=fake fp=ec45f6b1377c29c8 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.766567532Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.766593933Z caller=flush.go:217 msg="flushing stream" userid=fake fp=c57575fb7a016a20 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.768448769Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.768474809Z caller=flush.go:217 msg="flushing stream" userid=fake fp=173043024fb9e459 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.769182667Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.769217878Z caller=flush.go:217 msg="flushing stream" userid=fake fp=ae3e24e64fdc7bba immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.769913215Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.771294559Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.77132209Z caller=flush.go:217 msg="flushing stream" userid=fake fp=f4a97b4d712b6722 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.77132507Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.771353481Z caller=flush.go:217 msg="flushing stream" userid=fake fp=0c9522055cc4378e immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.771296089Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.771410812Z caller=flush.go:217 msg="flushing stream" userid=fake fp=af1bc42387df77f5 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.772046458Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.772076918Z caller=flush.go:217 msg="flushing stream" userid=fake fp=948ebe4e3389bbf6 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.780099087Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.780184539Z caller=flush.go:217 msg="flushing stream" userid=fake fp=b6f37a889b95df24 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.780285831Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.788048444Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.788222718Z caller=flush.go:217 msg="flushing stream" userid=fake fp=28256186738a62ef immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.794237077Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.806494381Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.814520199Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.814576591Z caller=flush.go:217 msg="flushing stream" userid=fake fp=5944aa6d0e38140a immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.815158755Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.815279098Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.817315829Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.820072307Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.833290105Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.833363187Z caller=flush.go:217 msg="flushing stream" userid=fake fp=472100009cf2b1df immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.834006983Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.834172727Z caller=flush.go:217 msg="flushing stream" userid=fake fp=0f98ec8be845ebf6 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.862941129Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.863063162Z caller=flush.go:217 msg="flushing stream" userid=fake fp=cd01a2781f5107fa immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.874982468Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.875304356Z caller=flush.go:217 msg="flushing stream" userid=fake fp=44818c47b2e88444 immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.875713656Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.877966222Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.907222376Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.907307669Z caller=flush.go:217 msg="flushing stream" userid=fake fp=9a4e037c3b69a04e immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.915601334Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.915648365Z caller=flush.go:217 msg="flushing stream" userid=fake fp=7ab71da3d49bc1aa immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.948555001Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.960479646Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:29:54.960826445Z caller=flush.go:217 msg="flushing stream" userid=fake fp=7a4674416e251d1a immediate=false
Dec 30 11:29:54 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:54.968937926Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:29:55 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=error ts=2021-12-30T11:29:55.005590154Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"
Dec 30 11:30:08 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:30:08.071194929Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=61.422µs msg="gRPC (success)"
Dec 30 11:30:08 tor1-infra-loki-writer-1 loki-linux-amd64[1702383]: level=debug ts=2021-12-30T11:30:08.789659159Z caller=grpc_logging.go:41 method=/grpc.health.v1.Health/Check duration=79.462µs msg="gRPC (success)"
@DylanGuedes
Copy link
Contributor

Hey looking around I found this, not exactly related to Loki. Could you confirm that your creds are all right?

@weastur
Copy link
Author

weastur commented Jan 3, 2022

I have arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess for DynamoDB and for both buckets I have the next one policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObjectTagging",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketTagging",
                "s3:GetObjectRetention",
                "s3:ListBucketVersions",
                "s3:RestoreObject",
                "s3:ListBucket",
                "s3:GetBucketVersioning",
                "s3:PutObject",
                "s3:GetObject",
                "s3:AbortMultipartUpload",
                "s3:GetObjectTagging",
                "s3:PutBucketAcl",
                "s3:PutObjectTagging",
                "s3:DeleteObject",
                "s3:GetBucketLocation",
                "s3:PutObjectAcl",
                "s3:GetObjectVersion",
                "s3:GetBucketOwnershipControls",
                "s3:PutBucketOwnershipControls"
            ],
            "Resource": [
                "arn:aws:s3:::${bucket}/*",
                "arn:aws:s3:::${bucket}"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:GetAccessPoint",
                "s3:ListAllMyBuckets",
                "s3:ListAccessPoints",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        }
    ]
}

where ${bucket} is s3-ca-loki or s3-ca-loki-ruler

@sbkg0002
Copy link

sbkg0002 commented Jan 6, 2022

Same trouble here with loki-stack version 2.5.1. AmazonDynamoDBFullAccess and s3:* permissions. Clean install.

@Probotect0r
Copy link

Probotect0r commented Jan 6, 2022 •
edited

Same issue running the 2.4.1 docker image on an EC2 instance in monolithic mode.

I also tried using BoltDB as the index store, and S3 as the chunk store and that worked perfectly, so there is definitely something going on with the DynamoDB store. The TableManager doesn't seem to be creating any tables in DynamoDB.

@sbkg0002
Copy link

sbkg0002 commented Jan 7, 2022

I reverted to the loki-stack version 2.5.0 and a fresh install worked.

@william00179
Copy link

william00179 commented Jan 12, 2022 •
edited

I can confirm I'm also having this issue with 2.4.1. Gave full permissions on DynamoDB but the table was never created. As above, just got the level=error ts=2021-12-30T11:29:54.794237077Z caller=flush.go:221 org_id=fake msg="failed to flush user" err="ResourceNotFoundException: Requested resource not found"

Also attempted with main at main-fc49a87 and the issue is still occurring.

@ofiry-tm ofiry-tm mentioned this issue Jan 12, 2022
Closed
@fayas-akram
Copy link

I am having the same issue and it's for the local environment ( not containerized env ). There's no clue that Loki DynamoDB Tables.
issue thread here 5070

Please let me know if there's any workaround

@andino6869
Copy link

I use loki 2.4.2 (started with loki 2.4.1. from enterprise-logs some days ago).

Today I detected upgrade notes to loki 2.4.0 which say:

"The single binary no longer runs a table-manager"
"Anyone ... not using boltdb/shipper ... should include '-target=all,table-manager' ..."

Afterwards dynamodb index tables were created. May be some of you have the same case !?

The whole text can be read here: https://github.com/grafana/loki/blob/main/docs/sources/upgrading/_index.md#the-single-binary-no-longer-runs-a-table-manager

@solidnerd
Copy link

Target can be configured via extraArgs for everyone using the helm chart.

# values.yaml
# ....
extraArgs:
  target: all,table-manager

It was posted here: #5021 (comment)

@5sfayas
Copy link

5sfayas commented Jan 19, 2022

I use loki 2.4.2 (started with loki 2.4.1. from enterprise-logs some days ago).

Today I detected upgrade notes to loki 2.4.0 which say:

"The single binary no longer runs a table-manager" "Anyone ... not using boltdb/shipper ... should include '-target=all,table-manager' ..."

Afterwards dynamodb index tables were created. May be some of you have the same case !?

The whole text can be read here: https://github.com/grafana/loki/blob/main/docs/sources/upgrading/_index.md#the-single-binary-no-longer-runs-a-table-manager

I tried this and it works as expected.

@weastur
Copy link
Author

weastur commented Jan 21, 2022

It's worked for me too. I believe it should be clarified in the documentation. Because, as I can see, there is no mention of the table-manager service at all in the architecture doc.

@stale
Copy link

stale bot commented Mar 2, 2022

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

  • Mark issues as revivable if we think it's a valid issue but isn't something we are likely
    to prioritize in the future (the issue will still remain closed).
  • Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

@stale stale bot added the stale A stale issue or PR that will automatically be closed. label Mar 2, 2022
@stale stale bot closed this as completed Apr 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale A stale issue or PR that will automatically be closed.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@solidnerd @sbkg0002 @Probotect0r @DylanGuedes @william00179 @weastur @andino6869 @5sfayas @fayas-akram