Are we doing something wrong or having performance problems with Elasticsearch?

[DarkAvery1]

Question

Hello, Vector.dev community!
We're currently migrating our logging pipeline "from Kafka -> Logstash -> Elasticsearch" to "Kafka -> Vector -> Elasticsearch" and have run into a couple of performance issues. We'd really appreciate any advice or recommendations.
Our Elasticsearch cluster consists of three hot nodes. The previous "Kafka -> Logstash -> Elasticsearch" pipeline handled the indexing workload without any noticeable problems. However, after switching to Vector, we've observed the following:

1. Vector is unable to keep up with Elasticsearch's indexing throughput. At peak load, its internal buffers occasionally fill up, causing backpressure and delays.
2. Horizontal scaling doesn't improve overall throughput. A single Vector instance(16 cpu 24 ram) processes around 2 million events per minute. When we scale to two Vector instances, each processes only about 1 million events per minute, so the total throughput remains the same instead of increasing.
   We understand that the bottleneck may be in Elasticsearch or Kafka rather than Vector itself, but so far we haven't been able to identify the root cause.
   Has anyone experienced similar behavior or can suggest which components or settings we should investigate? Any recommendations for tuning Vector, Kafka, or Elasticsearch for higher indexing throughput would be greatly appreciated.

Vector Config

log_schema:
  timestamp_key: "@timestamp"

sources:
  wazuh_indexing_source:
    type: kafka
    bootstrap_servers: kafka01:9093, kafka02:9093, kafka03:9093, kafka04:9093
    group_id: "indexing_es"
    topics:
      - topic
    auto_offset_reset: 'earliest'
    decoding:
      codec: json
    tls:
      enabled: true
      ca_file: '/etc/vector/cert/ca.pem'
      verify_certificate: true
      verify_hostname: true
      crt_file: '/etc/vector/cert/user-cert.pem'
      key_file: '/etc/vector/cert/priv.pem'
    acknowledgements:
      enabled: true

sinks:
  wazuh_indexing_sink:
    type: elasticsearch
    inputs:
      - wazuh_indexing_source
    endpoints:
      - https://ingest1:9200
      - https://ingest2:9200
      - https://ingest3:9200
    opensearch_service_type: managed
    auth:
      strategy: basic
      user: admin
      password: admin
    tls:
      ca_file: /etc/vector/cert/root-ca.pem
    bulk:
      action: "create"
      index: "{{ .fields.index_prefix }}%Y.%m.%d"
    compression: zstd
    request_retry_partial: true
    batch:
      max_events: 10000
      max_bytes: 200000000
      timeout_secs: 0.1
    request:
      rate_limit_duration_secs: 60
      rate_limit_num: 300
      retry_max_duration_secs: 300
      concurrency: 32
    buffer:
    - type: memory
      max_events: 50000

[dlanov]

Two separate issues are visible directly in your config, and they have different root causes.

Issue 1 — Rate limiter is capping your throughput

Your current settings:

  request:
    rate_limit_duration_secs: 60
    rate_limit_num: 300
    concurrency: 32

This caps Vector at 300 bulk requests per minute, regardless of concurrency. Combined with batch.timeout_secs: 0.1 (100ms flush timeout), batches are flushed every 100ms and will almost never reach max_events: 10000 — so each request carries far fewer than 10k events. With small batches and a 300 req/min ceiling you're leaving most of your concurrency headroom unused.

Fix — remove or raise the rate limit:

  request:
    concurrency: 32           # keep this
    rate_limit_num: 9999999   # effectively unlimited, let ES backpressure control flow
    rate_limit_duration_secs: 1

Also extend the batch timeout so batches actually fill up before flushing:

  batch:
    max_events: 10000
    max_bytes: 10485760       # 10MB is a more reasonable bulk payload than 200MB
    timeout_secs: 5           # give it 5 seconds to accumulate events

The 200MB max_bytes is also unusually large — Elasticsearch's default http.max_content_length is 100MB, so you may already be hitting rejection errors silently. Drop it to 10–20MB.

Issue 2 — Horizontal scaling is capped by Kafka partition count

Two Vector instances each processing 1M events/min instead of 2M combined is the classic Kafka consumer group symptom. Within a single group_id, Kafka assigns partitions 1:1 to consumers — if your topic has N partitions, you get at most N consumers doing useful work. A second consumer beyond that either idles or steals a partition from the first.

Check your partition count:

  kafka-topics.sh --bootstrap-server kafka01:9093 \
    --describe --topic topic

If partitions = 1, both Vector instances are racing for the same partition and Kafka assigns it to one at a time. If partitions = 2, each instance gets one — and you see exactly what you're seeing (same total, just split).

Fix — increase topic partitions to match (or exceed) the number of Vector replicas you want to run:

  kafka-topics.sh --bootstrap-server kafka01:9093 \
    --alter --topic topic --partitions 8

Then scale Vector to 8 replicas and throughput will scale linearly (Elasticsearch permitting).

Summary

Problem 1: Buffer backpressure at peak

• Root cause: rate_limit_num: 300 is too low, capping you at 300 bulk requests/min regardless of concurrency

• Fix: Set rate limit to unlimited and extend batch.timeout_secs to let batches fill up properly

  Problem 2: No throughput gain from a second instance

• Root cause: Kafka topic has too few partitions — consumers in the same group_id are limited to one partition each, so adding a replica just splits existing throughput rather than adding to it

• Fix: Increase topic partitions to match or exceed your desired replica count

  Why Logstash didn't have the partition problem

• Logstash uses internal threading per consumer thread rather than true Kafka consumer group semantics, so partition count mattered less

• Vector is more faithful to the consumer group contract which makes it surface this ceiling immediately

[DarkAvery1]

Sorry for the delayed response. Thanks for getting back to me.