Insanely high S3 bucket reads on Tempo

[openPablo]

I'm using loki/tempo/mimir with s3 storage.

Checked my AWS billing, I've got 9 million S3 puts, which is expected, since I'm tracing like 20 services. This costs 40 dollars a month
But then I found 1.3 BILLION s3 read events (600 dollars a month). After some digging I confirmed the read events are originating from the tempo storage s3 bucket.
This has to be a misconfiguration of some sort, because the discrepancy between puts and reads doesn't make sense.

Any ideas?

This is my config file:

server:
  http_listen_port: 3200
  grpc_listen_port: 9095
  grpc_server_max_recv_msg_size: 8388608
  grpc_server_max_send_msg_size: 8388608
  graceful_shutdown_timeout: 10s
  log_level: info

distributor:
  receivers:
    otlp:
      protocols:
        http:
        grpc:

ingester:
  trace_idle_period: 10s
  max_block_bytes: 1_000_000
  max_block_duration: 5m
  lifecycler:
    heartbeat_period: 100ms
    ring:
      replication_factor: 3
      kvstore:
        store: memberlist

metrics_generator:
  registry:
    external_labels:
      source: tempo-metrics
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: ${MIMIR_URL}/api/v1/push
        queue_config:
          max_samples_per_send: 2000
          capacity: 10000
          min_shards: 4
          max_shards: 1500

compactor:
  ring:
    kvstore:
      store: memberlist
  compaction:
    compaction_window: 1h
    max_block_bytes: 100_000_000
    block_retention: 336h
    compacted_block_retention: 1h
    compaction_cycle: 10m


memberlist:
  left_ingesters_timeout: 30m
  pull_push_interval: 15s
  rejoin_interval: 30s
  node_name: tempo
  randomize_node_name: true
  join_members: 
    - dns+${TEMPO_DNS}:7946

querier:
  frontend_worker:
    frontend_address: localhost:9095

storage:
  trace:
    backend: s3
    block:
      bloom_filter_false_positive: .05
    wal:
      path: /tmp/tempo/wal
    s3:
      bucket: ${S3_TEMPO_BUCKET}
      endpoint: s3.dualstack.${REGION}.amazonaws.com
      region: ${REGION}
    pool:
      max_workers: 100
      queue_depth: 10000

overrides:
  metrics_generator_processors: ["span-metrics", "service-graphs"]
  max_search_bytes_per_trace: 50000
  max_bytes_per_tag_values_query: 50000000

[joe-elliott]

The primary drivers of S3 bucket GETs are:

• Querying
• Compaction
• Polling

Querying
For the query path, using caching can reduce GETs quite a bit especially on the trace by id path. We are also looking at adding more caching options to reduce this even more (#2470). This doc is a little dated but covers the most important thing which is configuring caching for bloom filters: https://grafana.com/docs/tempo/latest/operations/caching/. We use memcached and our config looks like:

storage:
    trace:
        cache: memcached
        cache_max_block_age: 504h
        cache_min_compaction_level: 0
        memcached:
            host: memcached
            max_idle_conns: 100
            service: memcached-client
            timeout: 200ms

Another option to reduce querier GETs is to increase the read buffer sizes. Defaults listed:

storage:
  trace:
    search:
      read_buffer_count: 32
      read_buffer_size_bytes: 1048576

Compaction
Unfortunately there doesn't seem to be much we can do about compaction. I just dug into the code and we hardcode the reads at 2MB. If you find this is a major driver of GETs we can expose this as a configuration option. I suppose you could just scale down compactors to compact less.

Polling
This is likely the largest cause of your GETs. Something you can do safely is change blocklist_poll_tenant_index_builders to 1. This is set at 2 for redundancy but it's almost certainly fine to just run it at one.

The polling interval also defaults to 5m. You can likely safely increase this to 15m or more, but it's a bit more tricky b/c it interacts with a few other components. For simplicity I would keep all components polling at the same interval. Then you'll also have to be thoughtful complete_block_timeout and compacted_block_retention when increasing your polling timeout. Details here:

https://grafana.com/docs/tempo/latest/configuration/polling/

Finally, I'd recommend watching the metric tempodb_backend_request_duration_seconds_count sum/rated by container to see how the changes you are making impact GETs.

Let me know how it goes!

[openPablo]

Hey Joe,

Thanks a lot for your in depth reply.
I suspect querying is not the culprit since the amount of S3 bucket reads slowly increases day to day, even during weekends when barely no one checks these dashboards.
It also looks like the costs/S3 reads are increasing exponentially. Which makes me suspect that it's compaction working on the stored data.
I set blocklist_poll_tenant_index_builders to 1, and also changed my s3 bucket lifecycle policy to only hold data for a week instead of a month.

It seems like being able to fine tune compaction more would be useful!

I created a dashboard to watch tempodb_backend_request_duration_seconds_count, I'll keep you updated. Thanks again.

[joe-elliott]

and also changed my s3 bucket lifecycle policy ...

This will work, but will cause conflicts with Tempo. We'd advise setting retention using Tempo here:

compactor:
  compaction:
    block_retention: <duration val>

We actually use both with our lifecycle policy set several days after our Tempo retention.

[openPablo]

Hi @joe-elliott ,

Thanks again for your help!
S3 read costs have lowered a lot, (120 a day to 6 dollar a day). My changes were:
Setting s3 bucket lifecycle to 9 days from 90 days
These tempo config changes:

 storage:
   trace:
     blocklist_poll_tenant_index_builders: 1
     blocklist_poll: 30m

 ingester:
   complete_block_timeout: 30m

 compactor:
   compaction:
     compaction_cycle: 30m
     block_retention: 168h

I'm left wondering if the major drop in costs wasn't simply due to deleting a lot of data, and I still feel like this cost could be further dropped by fine tuning compaction more. I think it costs me more to run the steady compaction cycles than I gain from it.

Full config file, incase somebody finds this from google:

server:
  http_listen_port: 3200
  grpc_listen_port: 9095
  grpc_server_max_recv_msg_size: 8388608
  grpc_server_max_send_msg_size: 8388608
  graceful_shutdown_timeout: 10s
  log_level: info

distributor:
  receivers:
    otlp:
      protocols:
        http:
        grpc:

ingester:
  complete_block_timeout: 30m
  trace_idle_period: 10s
  max_block_bytes: 1_000_000
  max_block_duration: 5m
  lifecycler:
    heartbeat_period: 100ms
    ring:
      replication_factor: 3
      kvstore:
        store: memberlist

metrics_generator:
  registry:
    external_labels:
      source: tempo-metrics
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: ${MIMIR_URL}/api/v1/push
        queue_config:
          max_samples_per_send: 2000
          capacity: 10000
          min_shards: 4
          max_shards: 1500

compactor:
  ring:
    kvstore:
      store: memberlist
  compaction:
    compaction_window: 1h
    max_block_bytes: 100_000_000
    block_retention: 168h
    compacted_block_retention: 1h
    compaction_cycle: 30m


memberlist:
  left_ingesters_timeout: 30m
  pull_push_interval: 15s
  rejoin_interval: 30s
  node_name: tempo
  randomize_node_name: true
  join_members: 
    - dns+${TEMPO_DNS}:7946

querier:
  frontend_worker:
    frontend_address: localhost:9095

storage:
  trace:
    blocklist_poll: 30m
    blocklist_poll_tenant_index_builders: 1
    backend: s3
    block:
      bloom_filter_false_positive: .05
    wal:
      path: /tmp/tempo/wal
    s3:
      bucket: ${S3_TEMPO_BUCKET}
      endpoint: s3.dualstack.${REGION}.amazonaws.com
      region: ${REGION}
    pool:
      max_workers: 100
      queue_depth: 10000

overrides:
  metrics_generator_processors: ["span-metrics", "service-graphs"]
  max_search_bytes_per_trace: 50000
  max_bytes_per_tag_values_query: 50000000

[joe-elliott]

Nice! I will point out that your complete_block_timeout needs to be at least 2x your poll cycle or you may not be able to retrieve some traces.

Other than that, looks good!