How to improve the distributor's performance

[liaol]

I am using Pyroscope with the opentelemetry-ebpf-profiler to profile an entire host.

Here is my load test setup:

• Distributor: 16 cores, 32 GB RAM
• Ingesters: 8cores, 16GB RAM, SSD disk，3 instances
• Method: Uses the /opentelemetry.proto.collector.profiles.v1development.ProfilesService/Export gRPC endpoint
• Workload: Sends 8,000 samples (4 MB after gzip compression) every 5 seconds
• Client Configuration: Write timeout set to 5 seconds

The distributor's performance is suboptimal, handling only 7 requests per second, indicating a bottleneck. The ingesters, however, are performing well.

How can I optimize the distributor's performance to handle higher throughput? Thanks~

Here is my config

storage:
    backend: "s3"

distributor:
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/

ingester:
    lifecycler:
        ring:
            kvstore:
                store: memberlist
                prefix: collectors/

limits:
    max_profile_size_bytes: 33554432
    ingestion_rate_mb: 1024000
    max_global_series_per_tenant: 0
    max_profile_stacktrace_depth: 0
    compactor_blocks_retention_period: 48h

[korniltsev]

Do you have a pprof from the suffering distributor?

[liaol]

Do you have a pprof from the suffering distributor?

@korniltsev

cpu usage: 10 of 16cores cpu profile

memory usage: 18GB rss before GC heap profile

[liaol]

The distributor processes a request with 8,000 samples in 5 seconds, with the following breakdown:

1. Read request & unmarshal: 1 second
2. Convert Otel to Google format: 1 second
3. Push parsed data: 3 seconds
   • Normalize: 1 second
   • Send to ingester: 2 seconds

[liaol]

I modified the setup to send pprof data directly to the /ingester endpoint, bypassing the need for conversion, and it works seamlessly.