[exporter/clickhouse] exporter fails with IO timeout error under load

[MacNale]

Component(s)

exporter/clickhouse

What happened?

Description

Using prometeus-receiver to stress test the clickhouse exporter. Clickhouse exporter fails with IO timeout error.

Steps to Reproduce

1. Use Prometheus receiver or any other receiver (100K metric samples/minute)
2. Configure clickhouse exporter using the config below

Single node clickhouse running on linux with 16 core and 64 GB memory

Expected Result

• Load testing result needs to be published.
• Exporter is able to ingest 100K metric samples per minute

Actual Result

Timeout error

Collector version

0.81.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
CentOS 8

OpenTelemetry Collector configuration

prometheus:
      config:
        scrape_configs:
          - job_name: 'otel-collector-prom-load'
            scrape_interval: 30s
            static_configs:
              - targets: xxxx
processors:
  batch:
    send_batch_size: 20000
    timeout: 30s
    
 exporters:
  clickhouse:
    endpoint: xxxx
    ...
    timeout: 100s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
      
 pipelines:
    metrics:
      receivers: [prometheus, otlp]
      processors: [resource, batch]
      exporters: [clickhouse]

Log output

2023-12-22T03:25:33.335Z        debug   internal/gauge_metrics.go:145   insert gauge metrics fail       
{"kind": "exporter", "data_type": "metrics", "name": "clickhouse", "cost": 99.529687419}
2023-12-22T03:25:33.335Z        error   exporterhelper/queued_retry.go:165      Exporting failed. 
No more retries left. Dropping data.  {"kind": "exporter", "data_type": "metrics", "name": "clickhouse", "error": "max elapsed time expired insert gauge metrics fail:read: read tcp xxxxxxx:9000 i/o timeout", "dropped_items": 200005}
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure
        go.opentelemetry.io/collector/exporter@v0.81.0/exporterhelper/queued_retry.go:165
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
        go.opentelemetry.io/collector/exporter@v0.81.0/exporterhelper/queued_retry.go:407
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
        go.opentelemetry.io/collector/exporter@v0.81.0/exporterhelper/metrics.go:125
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
        go.opentelemetry.io/collector/exporter@v0.81.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
        go.opentelemetry.io/collector/exporter@v0.81.0/exporterhelper/internal/bounded_memory_queue.go:47

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/clickhouse: @hanjm @dmitryax @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Hello @MacNale, it looks like from the performance guide the exporter should be able to handle this load.

@hanjm, @dmitryax, @Frapschen: Is there something simple here that we're missing? Have you been able to take a look at this yet?

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/clickhouse: @hanjm @dmitryax @Frapschen @SpencerTorres

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mshustov]

@SpencerTorres PTAL