kafka indexing service qps very low

[camelus0211]

our druid cluster consumer kafka data by kafka indexing service, but we find kis qps very low about ~1600/s per task. The consumer lag is very large.

Affected Version

druid 0.13.0

Description

vm config

24cores 96G

middlemanger config

druid.service=druid/middleManager
druid.plaintextPort=8091

# Number of tasks per middleManager
druid.worker.capacity=16

# Task launch parameters
druid.indexer.runner.javaOpts=-server -Xmx3g -XX:MaxDirectMemorySize=5g -Duser.timezone=UTC+08:00 -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
druid.indexer.task.baseTaskDir=var/druid/task
druid.indexer.task.restoreTasksOnRestart=true

# HTTP server threads
druid.server.http.numThreads=32

# Processing threads and buffers on Peons
druid.indexer.fork.property.druid.processing.buffer.sizeBytes=268435456
druid.indexer.fork.property.druid.processing.numThreads=8

# Hadoop indexing
druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp

kafka partition

24

supervisor

{
  "type": "kafka",
  "dataSchema": {
    "dataSource": "stream-report-request",
    "parser": {
      "type": "string",
      "parseSpec": {
        "format": "json",
        "timestampSpec": {
          "column": "processTime",
          "format": "auto"
        },
        "dimensionsSpec": {
          "dimensions": [
            "adGroupId",
            "adPositionId",
            "creativeId",
            "orderId",
            "province",
            "city",
            "dspName",
            "adClass",
            "resourceFreqControlMode",
            "companyId",
            "dspPosId",
            "targetGame",
            "targetScene",
            "targetKeyword",
            "targetRoom",
            "gbCode",
            "filterReason",
            "configId",
            "blockIds",
            "strBlockIds",
            "itemId",
            "pctrBucket"
          ],
          "dimensionExclusions": [],
          "spatialDimensions": []
        }
      }
    },
    "metricsSpec": [{
      "type": "longSum",
      "name": "adRequestCount",
      "fieldName": "adRequestCount"
    },
      {
        "type": "longSum",
        "name": "adRecallCount",
        "fieldName": "adRecallCount"
      },
      {
        "type": "thetaSketch",
        "name": "uid",
        "fieldName": "uid"
      }
    ],
    "granularitySpec": {
      "type": "uniform",
      "segmentGranularity": "ten_minute",
      "queryGranularity": "NONE"
    }
  },
  "tuningConfig": {
    "type": "kafka",
    "maxRowsInMemory": 20000,
    "maxBytesInMemory": -1,
    "intermediatePersistPeriod": "PT10M"
  },
  "ioConfig": {
    "topic": "stream-report-request-topic",
    "consumerProperties": {
      "bootstrap.servers": "",
      "group.id": "consumer_group_druid"
    },
    "useEarliestOffset": true,
    "taskCount": 12,
    "replicas": 1,
    "taskDuration": "PT10M",
    "completionTimeout": "PT10M"
  }
}

[jihoonson]

Hi @camelus0211, maxRowsInMemory looks too low. Try setting it to a higher value if you can. The default value is 1m in the latest version.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.