Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Index Task crashes in middle manager when ingesting JSON keys that contain a comma #15736

Open
winsmith opened this issue Jan 22, 2024 · 0 comments
Open

Index Task crashes in middle manager when ingesting JSON keys that contain a comma #15736

winsmith opened this issue Jan 22, 2024 · 0 comments
Labels
Area - Ingestion Bug

Comments

@winsmith
Copy link

winsmith commented Jan 22, 2024 •
edited

Affected Version

28.0.1

Description

I encountered an error during compaction and ingestion where tasks in middle manager would crash randomly when ingesting a JSON line where the key contains a comma (,). We call these "thorny" data because tasks that touch them crash.

Here's a row that crashes on ingest:

{"receivedAt": "2024-01-20","A,B": "C"}

Here's an ingestion spec that triggers the error:

{
  "type": "index_parallel",
  "spec": {
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "inline",
        "data": "{\"receivedAt\": \"2024-01-20\",\"A,B\": \"C\"}\n"
      },
      "inputFormat": {
        "type": "json"
      }
    },
    "dataSchema": {
      "dataSource": "telemetry-signals-thorny-test",
      "timestampSpec": {
        "column": "receivedAt",
        "format": "iso"
      },
      "dimensionsSpec": {
        "dimensionExclusions": [
          "__time",
          "count",
          "receivedAt"
        ]
      }
    }
}

This results in task status failed:

{
  "id": "index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
  "groupId": "index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
  "type": "index_parallel",
  "createdTime": "2024-01-22T14:37:20.082Z",
  "queueInsertionTime": "1970-01-01T00:00:00.000Z",
  "statusCode": "FAILED",
  "status": "FAILED",
  "runnerStatusCode": "WAITING",
  "duration": 5685,
  "location": {
    "host": "10.244.7.175",
    "port": 8102,
    "tlsPort": -1
  },
  "dataSource": "telemetry-signals-thorny-test",
  "errorMsg": "java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org..."
}

And here's the complete index log in the middle manager:

bash-5.1$ cat index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14\:37\:20.080Z.log 
2024-01-22T14:37:21,125 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 6.2.5.Final
2024-01-22T14:37:21,839 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-kafka-indexing-service], jars: lz4-java-1.8.0.jar, druid-kafka-indexing-service-28.0.1.jar, zstd-jni-1.5.2-3.jar, snappy-java-1.1.10.4.jar, kafka-clients-3.5.1.jar
2024-01-22T14:37:21,843 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-azure-extensions], jars: azure-keyvault-core-1.0.0.jar, azure-storage-8.6.0.jar, druid-azure-extensions-28.0.1.jar
2024-01-22T14:37:21,844 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [postgresql-metadata-storage], jars: postgresql-metadata-storage-28.0.1.jar, checker-qual-3.31.0.jar, postgresql-42.6.0.jar
2024-01-22T14:37:21,846 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-datasketches], jars: druid-datasketches-28.0.1.jar, commons-math3-3.6.1.jar
2024-01-22T14:37:21,848 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-stats], jars: commons-math3-3.6.1.jar, druid-stats-28.0.1.jar
2024-01-22T14:37:22,116 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-kafka-indexing-service], jars: lz4-java-1.8.0.jar, druid-kafka-indexing-service-28.0.1.jar, zstd-jni-1.5.2-3.jar, snappy-java-1.1.10.4.jar, kafka-clients-3.5.1.jar
2024-01-22T14:37:22,119 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-azure-extensions], jars: azure-keyvault-core-1.0.0.jar, azure-storage-8.6.0.jar, druid-azure-extensions-28.0.1.jar
2024-01-22T14:37:22,122 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [postgresql-metadata-storage], jars: postgresql-metadata-storage-28.0.1.jar, checker-qual-3.31.0.jar, postgresql-42.6.0.jar
2024-01-22T14:37:22,123 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-datasketches], jars: druid-datasketches-28.0.1.jar, commons-math3-3.6.1.jar
2024-01-22T14:37:22,132 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-stats], jars: commons-math3-3.6.1.jar, druid-stats-28.0.1.jar
2024-01-22T14:37:23,384 INFO [main] org.apache.druid.server.emitter.EmitterModule - Using emitter [NoopEmitter{}] for metrics and alerts, with dimensions [{version=28.0.1}].
2024-01-22T14:37:23,714 WARN [main] org.apache.druid.segment.indexing.DataSchema - Rollup is enabled for dataSource [telemetry-signals-thorny-test] but no metricsSpec has been provided. Are you sure this is what you want?
2024-01-22T14:37:23,747 INFO [main] org.apache.druid.server.metrics.MetricsModule - Loaded 3 monitors: org.apache.druid.query.ExecutorServiceMonitor, org.apache.druid.curator.DruidConnectionStateListener, org.apache.druid.server.initialization.jetty.JettyServerModule$JettyMonitor
2024-01-22T14:37:23,763 INFO [main] org.apache.druid.cli.CliPeon - Starting up with processors [7], memory [2,147,483,648], maxMemory [3,221,225,472], directMemory [17,179,869,184]. Properties follow.
2024-01-22T14:37:23,779 WARN [main] org.apache.druid.indexing.common.config.TaskConfig - Batch processing mode argument value is null or not valid:[null], defaulting to[CLOSED_SEGMENTS] 
2024-01-22T14:37:23,982 INFO [main] org.apache.druid.query.DruidProcessingConfig - Detected max direct memory size of [17,179,869,184] bytes
2024-01-22T14:37:24,000 INFO [main] org.apache.druid.segment.loading.SegmentLocalCacheManager - Using storage location strategy: [LeastBytesUsedStorageLocationSelectorStrategy]
2024-01-22T14:37:24,050 INFO [main] org.apache.druid.guice.StorageNodeModule - Segment cache not configured on ServerType [indexer-executor]. It will not be assignable for segment placement
2024-01-22T14:37:24,108 INFO [main] org.eclipse.jetty.util.log - Logging initialized @4005ms to org.eclipse.jetty.util.log.Slf4jLog
2024-01-22T14:37:24,121 INFO [main] org.apache.druid.server.initialization.jetty.JettyServerModule - Creating http connector with port [8102]
2024-01-22T14:37:24,250 WARN [main] org.eclipse.jetty.server.handler.gzip.GzipHandler - minGzipSize of 0 is inefficient for short content, break even is size 23
2024-01-22T14:37:24,256 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[0] of size[7,000,000]
2024-01-22T14:37:24,262 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[1] of size[7,000,000]
2024-01-22T14:37:24,266 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[0] of size[7,000,000]
2024-01-22T14:37:24,270 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[1] of size[7,000,000]
2024-01-22T14:37:24,333 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [INIT]
2024-01-22T14:37:24,334 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [NORMAL]
2024-01-22T14:37:24,338 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2024-01-22T14:37:24,344 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.10-a32c7183d42325b03e44a06aade6a0f16955bf13, built on 05/29/2022 16:59 GMT
2024-01-22T14:37:24,344 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:host.name=druid-druid-cluster-middlemanagers-1.druid-druid-cluster-middlemanagers.druid-main.svc.cluster.local
2024-01-22T14:37:24,344 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.version=17.0.9
2024-01-22T14:37:24,344 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Debian
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-17-openjdk-amd64
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/tmp/conf/druid/cluster/_common:/tmp/conf/druid/cluster/data/middleManager:lib/commons-io-2.11.0.jar:lib/jboss-logging-3.4.1.Final.jar:lib/aws-java-sdk-core-1.12.497.jar:lib/derbynet-10.14.2.0.jar:lib/error_prone_annotations-2.20.0.jar:lib/jetty-continuation-9.4.53.v20231009.jar:lib/antlr4-runtime-4.5.3.jar:lib/guice-assistedinject-4.1.0.jar:lib/druid-indexing-hadoop-28.0.1.jar:lib/jakarta.inject-api-1.0.3.jar:lib/jetty-security-9.4.53.v20231009.jar:lib/jna-5.13.0.jar:lib/calcite-core-1.35.0.jar:lib/asm-commons-9.3.jar:lib/commons-compress-1.23.0.jar:lib/jakarta.activation-api-1.2.1.jar:lib/commons-codec-1.13.jar:lib/commons-dbcp2-2.0.1.jar:lib/jna-platform-5.13.0.jar:lib/druid-gcp-common-28.0.1.jar:lib/druid-indexing-service-28.0.1.jar:lib/google-oauth-client-1.34.1.jar:lib/resilience4j-core-1.3.1.jar:lib/config-magic-0.9.jar:lib/maven-repository-metadata-3.1.1.jar:lib/httpcore-4.4.11.jar:lib/proj4j-1.2.2.jar:lib/aws-java-sdk-s3-1.12.497.jar:lib/xz-1.9.jar:lib/httpclient5-5.1.3.jar:lib/jaxb-api-2.3.1.jar:lib/jetty-util-ajax-9.4.53.v20231009.jar:lib/commons-compiler-3.1.9.jar:lib/druid-processing-28.0.1.jar:lib/jmespath-java-1.12.497.jar:lib/curator-framework-5.5.0.jar:lib/rhino-1.7.14.jar:lib/lz4-java-1.8.0.jar:lib/jersey-guice-1.19.4.jar:lib/aopalliance-1.0.jar:lib/google-http-client-jackson2-1.42.3.jar:lib/netty-codec-4.1.100.Final.jar:lib/gson-2.10.jar:lib/web-console-28.0.1.jar:lib/aws-java-sdk-kms-1.12.497.jar:lib/jetty-server-9.4.53.v20231009.jar:lib/commons-pool2-2.2.jar:lib/jackson-annotations-2.12.7.jar:lib/plexus-utils-3.0.24.jar:lib/elasticache-java-cluster-client-1.2.0.jar:lib/avatica-metrics-1.23.0.jar:lib/jetty-http-9.4.53.v20231009.jar:lib/aether-util-0.9.0.M2.jar:lib/airline-io-2.8.4.jar:lib/fastutil-core-8.5.4.jar:lib/maven-settings-3.1.1.jar:lib/disruptor-3.3.6.jar:lib/metrics-core-4.2.19.jar:lib/avatica-server-1.23.0.jar:lib/maven-settings-builder-3.1.1.jar:lib/fastutil-8.5.4.jar:lib/netty-codec-http-4.1.100.Final.jar:lib/jcl-over-slf4j-1.7.36.jar:lib/calcite-linq4j-1.35.0.jar:lib/guava-31.1-jre.jar:lib/jetty-util-9.4.53.v20231009.jar:lib/maven-artifact-3.6.0.jar:lib/grpc-context-1.27.2.jar:lib/sigar-1.6.5.132.jar:lib/aether-connector-okhttp-0.0.9.jar:lib/json-simple-1.1.1.jar:lib/hibernate-validator-6.2.5.Final.jar:lib/jetty-io-9.4.53.v20231009.jar:lib/ion-java-1.0.2.jar:lib/jackson-jq-0.0.10.jar:lib/jersey-core-1.19.4.jar:lib/aether-spi-0.9.0.M2.jar:lib/netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:lib/audience-annotations-0.5.0.jar:lib/httpclient-4.5.13.jar:lib/aggdesigner-algorithm-6.0.jar:lib/javax.activation-api-1.2.0.jar:lib/derbyclient-10.14.2.0.jar:lib/log4j-1.2-api-2.18.0.jar:lib/netty-transport-4.1.100.Final.jar:lib/log4j-api-2.18.0.jar:lib/jetty-client-9.4.53.v20231009.jar:lib/avatica-core-1.23.0.jar:lib/guice-4.1.0.jar:lib/jersey-server-1.19.4.jar:lib/druid-services-28.0.1.jar:lib/aws-java-sdk-ec2-1.12.497.jar:lib/oshi-core-6.4.4.jar:lib/netty-handler-4.1.100.Final.jar:lib/jackson-datatype-guava-2.12.7.jar:lib/joni-2.1.27.jar:lib/jackson-module-jaxb-annotations-2.12.7.jar:lib/javax.servlet-api-3.1.0.jar:lib/opencsv-4.6.jar:lib/derby-10.14.2.0.jar:lib/google-http-client-apache-v2-1.42.3.jar:lib/vavr-match-0.10.2.jar:lib/asm-9.3.jar:lib/zstd-jni-1.5.2-3.jar:lib/asm-tree-9.3.jar:lib/jackson-jaxrs-smile-provider-2.12.7.jar:lib/jackson-jaxrs-base-2.12.7.jar:lib/log4j-core-2.18.0.jar:lib/vavr-0.10.2.jar:lib/ipaddress-5.3.4.jar:lib/aether-connector-file-0.9.0.M2.jar:lib/checker-qual-2.5.7.jar:lib/jetty-servlet-9.4.53.v20231009.jar:lib/jts-io-common-1.19.0.jar:lib/datasketches-java-4.2.0.jar:lib/druid-server-28.0.1.jar:lib/jaxb-runtime-2.3.1.jar:lib/asm-analysis-9.3.jar:lib/jakarta.validation-api-2.0.2.jar:lib/datasketches-memory-2.2.0.jar:lib/joda-time-2.12.5.jar:lib/protobuf-java-3.24.0.jar:lib/accessors-smart-2.4.11.jar:lib/jakarta.xml.bind-api-2.3.2.jar:lib/netty-resolver-dns-4.1.100.Final.jar:lib/jetty-rewrite-9.4.53.v20231009.jar:lib/httpcore5-5.1.3.jar:lib/janino-3.1.9.jar:lib/jackson-databind-2.12.7.jar:lib/google-api-client-2.2.0.jar:lib/curator-client-5.5.0.jar:lib/netty-common-4.1.100.Final.jar:lib/log4j-slf4j-impl-2.18.0.jar:lib/caffeine-2.8.0.jar:lib/stax-ex-1.8.jar:lib/airline-2.8.4.jar:lib/okhttp-1.0.2.jar:lib/jakarta.el-3.0.4.jar:lib/jts-core-1.19.0.jar:lib/compress-lzf-1.0.4.jar:lib/txw2-2.3.1.jar:lib/httpcore5-h2-5.1.3.jar:lib/FastInfoset-1.2.15.jar:lib/curator-recipes-5.5.0.jar:lib/netty-buffer-4.1.100.Final.jar:lib/jackson-dataformat-smile-2.12.7.jar:lib/apiguardian-api-1.1.2.jar:lib/jetty-servlets-9.4.53.v20231009.jar:lib/netty-codec-dns-4.1.100.Final.jar:lib/jackson-datatype-joda-2.12.7.jar:lib/maven-model-3.1.1.jar:lib/javax.activation-1.2.0.jar:lib/google-http-client-gson-1.42.3.jar:lib/slf4j-api-1.7.36.jar:lib/uzaygezen-core-0.2.jar:lib/jdbi-2.63.1.jar:lib/classmate-1.5.1.jar:lib/netty-codec-socks-4.1.100.Final.jar:lib/jetty-proxy-9.4.53.v20231009.jar:lib/async-http-client-2.5.3.jar:lib/aether-api-0.9.0.M2.jar:lib/jackson-jaxrs-json-provider-2.12.7.jar:lib/javax.el-api-3.0.0.jar:lib/tesla-aether-0.0.5.jar:lib/async-http-client-netty-utils-2.5.3.jar:lib/jsr311-api-1.1.1.jar:lib/guice-multibindings-4.1.0.jar:lib/netty-reactive-streams-2.0.0.jar:lib/istack-commons-runtime-3.0.7.jar:lib/zookeeper-3.5.10.jar:lib/netty-3.10.6.Final.jar:lib/json-path-2.3.0.jar:lib/netty-resolver-4.1.100.Final.jar:lib/jcodings-1.0.43.jar:lib/commons-collections4-4.2.jar:lib/jersey-servlet-1.19.4.jar:lib/commons-lang-2.6.jar:lib/RoaringBitmap-0.9.49.jar:lib/guice-servlet-4.1.0.jar:lib/opencensus-api-0.31.1.jar:lib/commons-lang3-3.12.0.jar:lib/commons-math3-3.6.1.jar:lib/druid-sql-28.0.1.jar:lib/shims-0.9.49.jar:lib/resilience4j-bulkhead-1.3.1.jar:lib/druid-aws-common-28.0.1.jar:lib/netty-transport-native-unix-common-4.1.100.Final.jar:lib/aether-impl-0.9.0.M2.jar:lib/curator-x-discovery-5.5.0.jar:lib/cron-scheduler-0.1.jar:lib/jackson-dataformat-cbor-2.12.7.jar:lib/zookeeper-jute-3.5.10.jar:lib/aws-java-sdk-sts-1.12.497.jar:lib/json-smart-2.4.11.jar:lib/maven-aether-provider-3.1.1.jar:lib/jackson-module-guice-2.12.7.jar:lib/jackson-core-2.12.7.jar:lib/commons-logging-1.1.1.jar:lib/netty-handler-proxy-4.1.100.Final.jar:lib/maven-model-builder-3.1.1.jar:lib/reactive-streams-1.0.2.jar:lib/icu4j-73.2.jar:lib/google-http-client-1.42.3.jar:lib/wagon-provider-api-2.4.jar:lib/opencensus-contrib-http-util-0.31.1.jar:lib/jsr305-2.0.1.jar:lib/failureaccess-1.0.1.jar:lib/log4j-jul-2.18.0.jar:lib/commons-collections-3.2.2.jar:lib/commons-beanutils-1.9.4.jar:lib/netty-transport-classes-epoll-4.1.100.Final.jar:lib/commons-text-1.10.0.jar:lib/plexus-interpolation-1.19.jar:lib/fastutil-extra-8.5.4.jar:lib/j2objc-annotations-1.3.jar:lib/javax.inject-1.jar:
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/druid/data
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=5.15.131.1-2.cm2
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=druid
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/opt/druid
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/druid
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=1906MB
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=3072MB
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=2048MB
2024-01-22T14:37:24,349 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=tiny-cluster-zk-0.tiny-cluster-zk,tiny-cluster-zk-1.tiny-cluster-zk,tiny-cluster-zk-2.tiny-cluster-zk sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@731ab49b
2024-01-22T14:37:24,355 INFO [main] org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2024-01-22T14:37:24,359 INFO [main] org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
2024-01-22T14:37:24,367 INFO [main] org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
2024-01-22T14:37:24,374 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Default schema
2024-01-22T14:37:24,380 INFO [main-SendThread(tiny-cluster-zk-2.tiny-cluster-zk:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server tiny-cluster-zk-2.tiny-cluster-zk/10.244.22.125:2181. Will not attempt to authenticate using SASL (unknown error)
2024-01-22T14:37:24,391 INFO [main-SendThread(tiny-cluster-zk-2.tiny-cluster-zk:2181)] org.apache.zookeeper.ClientCnxn - Socket error occurred: tiny-cluster-zk-2.tiny-cluster-zk/10.244.22.125:2181: Connection refused
2024-01-22T14:37:24,415 WARN [main] org.apache.druid.segment.indexing.DataSchema - Rollup is enabled for dataSource [telemetry-signals-thorny-test] but no metricsSpec has been provided. Are you sure this is what you want?
2024-01-22T14:37:24,455 INFO [main] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Running with task: {
  "type" : "index_parallel",
  "id" : "index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
  "groupId" : "index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
  "resource" : {
    "availabilityGroup" : "index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
    "requiredCapacity" : 1
  },
  "spec" : {
    "dataSchema" : {
      "dataSource" : "telemetry-signals-thorny-test",
      "timestampSpec" : {
        "column" : "receivedAt",
        "format" : "iso",
        "missingValue" : null
      },
      "dimensionsSpec" : {
        "dimensions" : [ ],
        "dimensionExclusions" : [ "__time", "count", "receivedAt" ],
        "includeAllDimensions" : false,
        "useSchemaDiscovery" : false
      },
      "metricsSpec" : [ ],
      "granularitySpec" : {
        "type" : "uniform",
        "segmentGranularity" : "DAY",
        "queryGranularity" : {
          "type" : "none"
        },
        "rollup" : true,
        "intervals" : [ ]
      },
      "transformSpec" : {
        "filter" : null,
        "transforms" : [ ]
      }
    },
    "ioConfig" : {
      "type" : "index_parallel",
      "inputSource" : {
        "type" : "inline",
        "data" : "{\"receivedAt\": \"2024-01-20\",\"A,B\": \"C\"}\n"
      },
      "inputFormat" : {
        "type" : "json",
        "keepNullColumns" : false,
        "assumeNewlineDelimited" : false,
        "useJsonNodeReader" : false
      },
      "appendToExisting" : false,
      "dropExisting" : false
    },
    "tuningConfig" : {
      "type" : "index_parallel",
      "maxRowsPerSegment" : null,
      "appendableIndexSpec" : {
        "type" : "onheap",
        "preserveExistingMetrics" : false
      },
      "maxRowsInMemory" : 1000000,
      "maxBytesInMemory" : 0,
      "skipBytesInMemoryOverheadCheck" : false,
      "maxTotalRows" : null,
      "numShards" : null,
      "splitHintSpec" : null,
      "partitionsSpec" : null,
      "indexSpec" : {
        "bitmap" : {
          "type" : "roaring"
        },
        "dimensionCompression" : "lz4",
        "stringDictionaryEncoding" : {
          "type" : "utf8"
        },
        "metricCompression" : "lz4",
        "longEncoding" : "longs"
      },
      "indexSpecForIntermediatePersists" : {
        "bitmap" : {
          "type" : "roaring"
        },
        "dimensionCompression" : "lz4",
        "stringDictionaryEncoding" : {
          "type" : "utf8"
        },
        "metricCompression" : "lz4",
        "longEncoding" : "longs"
      },
      "maxPendingPersists" : 0,
      "forceGuaranteedRollup" : false,
      "reportParseExceptions" : false,
      "pushTimeout" : 0,
      "segmentWriteOutMediumFactory" : null,
      "maxNumConcurrentSubTasks" : 1,
      "maxRetry" : 3,
      "taskStatusCheckPeriodMs" : 1000,
      "chatHandlerTimeout" : "PT10S",
      "chatHandlerNumRetries" : 5,
      "maxNumSegmentsToMerge" : 100,
      "totalNumMergeTasks" : 10,
      "logParseExceptions" : false,
      "maxParseExceptions" : 2147483647,
      "maxSavedParseExceptions" : 0,
      "maxColumnsToMerge" : -1,
      "awaitSegmentAvailabilityTimeoutMillis" : 0,
      "maxAllowedLockCount" : -1,
      "partitionDimensions" : [ ]
    }
  },
  "context" : {
    "forceTimeChunkLock" : true,
    "useLineageBasedSegmentAllocation" : true
  },
  "dataSource" : "telemetry-signals-thorny-test"
}
2024-01-22T14:37:24,457 INFO [main] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Attempting to lock file[/opt/druid/var/druid/task/slot1/index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z/lock].
2024-01-22T14:37:24,462 INFO [main] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Acquired lock file[/opt/druid/var/druid/task/slot1/index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z/lock] in 4ms.
2024-01-22T14:37:24,464 INFO [main] org.apache.druid.indexing.common.task.AbstractBatchIndexTask - Using time chunk lock since forceTimeChunkLock is [true] and mode is [REPLACE_LEGACY].
2024-01-22T14:37:24,467 INFO [main] org.apache.druid.segment.loading.SegmentLocalCacheManager - Using storage location strategy: [LeastBytesUsedStorageLocationSelectorStrategy]
2024-01-22T14:37:24,474 INFO [task-runner-0-priority-0] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Running task: index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z
2024-01-22T14:37:24,476 WARN [task-runner-0-priority-0] org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask - Intervals are missing in granularitySpec while this task is potentially overwriting existing segments. Forced to use timeChunk lock.
2024-01-22T14:37:24,477 WARN [task-runner-0-priority-0] org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask - firehoseFactory[InlineInputSource] is not splittable. Running sequentially.
2024-01-22T14:37:24,477 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [SERVER]
2024-01-22T14:37:24,479 INFO [task-runner-0-priority-0] org.apache.druid.indexing.common.task.AbstractBatchIndexTask - Using time chunk lock since forceTimeChunkLock is [true] and mode is [REPLACE_LEGACY].
2024-01-22T14:37:24,479 INFO [main] org.eclipse.jetty.server.Server - jetty-9.4.53.v20231009; built: 2023-10-09T12:29:09.265Z; git: 27bde00a0b95a1d5bbee0eae7984f891d2d0f8c9; jvm 17.0.9+9-Debian-1deb12u1
2024-01-22T14:37:24,480 WARN [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Chat handler is already registered. Skipping chat handler registration.
2024-01-22T14:37:24,489 INFO [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Determining intervals and shardSpecs
2024-01-22T14:37:24,499 INFO [main-SendThread(tiny-cluster-zk-0.tiny-cluster-zk:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server tiny-cluster-zk-0.tiny-cluster-zk/10.244.2.104:2181. Will not attempt to authenticate using SASL (unknown error)
2024-01-22T14:37:24,501 INFO [main-SendThread(tiny-cluster-zk-0.tiny-cluster-zk:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.244.7.175:43330, server: tiny-cluster-zk-0.tiny-cluster-zk/10.244.2.104:2181
2024-01-22T14:37:24,510 INFO [main-SendThread(tiny-cluster-zk-0.tiny-cluster-zk:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server tiny-cluster-zk-0.tiny-cluster-zk/10.244.2.104:2181, sessionid = 0x101bf1fb80d0233, negotiated timeout = 30000
2024-01-22T14:37:24,517 INFO [main-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2024-01-22T14:37:24,530 INFO [main-EventThread] org.apache.curator.framework.imps.EnsembleTracker - New config event received: {}
2024-01-22T14:37:24,530 INFO [main-EventThread] org.apache.curator.framework.imps.EnsembleTracker - New config event received: {}
2024-01-22T14:37:24,538 WARN [NodeRoleWatcher[COORDINATOR]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher - Ignored event type [CONNECTION_RECONNECTED] for node watcher of role [coordinator].
2024-01-22T14:37:24,538 WARN [NodeRoleWatcher[OVERLORD]] org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher - Ignored event type [CONNECTION_RECONNECTED] for node watcher of role [overlord].
2024-01-22T14:37:24,549 INFO [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Found intervals and shardSpecs in 59ms
2024-01-22T14:37:24,549 INFO [task-runner-0-priority-0] org.apache.druid.indexing.common.task.AbstractBatchIndexTask - Using time chunk lock since forceTimeChunkLock is [true] and mode is [REPLACE_LEGACY].
2024-01-22T14:37:24,562 INFO [NodeRoleWatcher[COORDINATOR]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node [http://10.244.1.147:8088] of role [coordinator] detected.
2024-01-22T14:37:24,562 INFO [NodeRoleWatcher[OVERLORD]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node [http://10.244.1.150:8088] of role [overlord] detected.
2024-01-22T14:37:24,562 INFO [NodeRoleWatcher[COORDINATOR]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node watcher of role [coordinator] is now initialized with 1 nodes.
2024-01-22T14:37:24,562 INFO [NodeRoleWatcher[OVERLORD]] org.apache.druid.discovery.BaseNodeRoleWatcher - Node watcher of role [overlord] is now initialized with 1 nodes.
2024-01-22T14:37:24,696 INFO [main] org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0
2024-01-22T14:37:24,697 INFO [main] org.eclipse.jetty.server.session - No SessionScavenger set, using defaults
2024-01-22T14:37:24,697 INFO [main] org.eclipse.jetty.server.session - node0 Scavenging every 600000ms
2024-01-22T14:37:24,721 WARN [task-runner-0-priority-0] org.apache.druid.segment.indexing.DataSchema - Rollup is enabled for dataSource [telemetry-signals-thorny-test] but no metricsSpec has been provided. Are you sure this is what you want?
2024-01-22T14:37:24,771 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider as a provider class
2024-01-22T14:37:24,771 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider as a provider class
2024-01-22T14:37:24,772 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering org.apache.druid.server.initialization.jetty.CustomExceptionMapper as a provider class
2024-01-22T14:37:24,772 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering org.apache.druid.server.initialization.jetty.ForbiddenExceptionMapper as a provider class
2024-01-22T14:37:24,772 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering org.apache.druid.server.initialization.jetty.BadRequestExceptionMapper as a provider class
2024-01-22T14:37:24,772 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering org.apache.druid.server.initialization.jetty.ServiceUnavailableExceptionMapper as a provider class
2024-01-22T14:37:24,772 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering org.apache.druid.server.StatusResource as a root resource class
2024-01-22T14:37:24,774 INFO [main] com.sun.jersey.server.impl.application.WebApplicationImpl - Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
2024-01-22T14:37:24,799 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver - Pushing [1] segments in background
2024-01-22T14:37:24,802 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver - Pushing segments: [telemetry-signals-thorny-test_2024-01-20T00:00:00.000Z_2024-01-21T00:00:00.000Z_2024-01-22T14:37:24.693Z]
2024-01-22T14:37:24,805 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Persisted rows[1] and (estimated) bytes[154]
2024-01-22T14:37:24,837 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.initialization.jetty.CustomExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,839 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.initialization.jetty.ForbiddenExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,839 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.initialization.jetty.BadRequestExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,840 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.initialization.jetty.ServiceUnavailableExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,840 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,848 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider to GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,925 INFO [[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[telemetry-signals-thorny-test_2024-01-20T00:00:00.000Z_2024-01-21T00:00:00.000Z_2024-01-22T14:37:24.693Z] spill[0] to disk in [120] ms (1 rows).
2024-01-22T14:37:24,927 INFO [[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data with commit metadata [null] for segments: telemetry-signals-thorny-test_2024-01-20T00:00:00.000Z_2024-01-21T00:00:00.000Z_2024-01-22T14:37:24.693Z
2024-01-22T14:37:24,927 INFO [[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Persisted stats: processed rows: [1], persisted rows[1], sinks: [1], total fireHydrants (across sinks): [1], persisted fireHydrants (across sinks): [1]
2024-01-22T14:37:24,928 INFO [[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Preparing to push (stats): processed rows: [1], sinks: [1], fireHydrants (across sinks): [1]
2024-01-22T14:37:24,934 WARN [[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Failed to push merged index for segment[telemetry-signals-thorny-test_2024-01-20T00:00:00.000Z_2024-01-21T00:00:00.000Z_2024-01-22T14:37:24.693Z].
org.apache.druid.java.util.common.ISE: Wrong number of splits[5] in line[A,B,0,187,436]
        at org.apache.druid.java.util.common.io.smoosh.SmooshedFileMapper.load(SmooshedFileMapper.java:89) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.java.util.common.io.smoosh.Smoosh.map(Smoosh.java:32) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.IndexIO$V9IndexLoader.load(IndexIO.java:550) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:205) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:195) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:913) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:786) ~[druid-server-28.0.1.jar:28.0.1]
        at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:250) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:240) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:122) ~[guava-31.1-jre.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-01-22T14:37:24,941 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS.
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.druid.java.util.common.ISE: Wrong number of splits[5] in line[A,B,0,187,436]
        at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:1050) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.common.task.IndexTask.runTask(IndexTask.java:543) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:178) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runSequential(ParallelIndexSupervisorTask.java:1212) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runTask(ParallelIndexSupervisorTask.java:551) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:178) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:478) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:450) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) ~[guava-31.1-jre.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.druid.java.util.common.ISE: Wrong number of splits[5] in line[A,B,0,187,436]
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:567) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91) ~[guava-31.1-jre.jar:?]
        at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAndClear(BatchAppenderatorDriver.java:151) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAllAndClear(BatchAppenderatorDriver.java:134) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.common.task.InputSourceProcessor.process(InputSourceProcessor.java:131) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:940) ~[druid-indexing-service-28.0.1.jar:28.0.1]
        ... 13 more
Caused by: java.lang.RuntimeException: org.apache.druid.java.util.common.ISE: Wrong number of splits[5] in line[A,B,0,187,436]
        at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:998) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:786) ~[druid-server-28.0.1.jar:28.0.1]
        at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:250) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:240) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:122) ~[guava-31.1-jre.jar:?]
        ... 3 more
Caused by: org.apache.druid.java.util.common.ISE: Wrong number of splits[5] in line[A,B,0,187,436]
        at org.apache.druid.java.util.common.io.smoosh.SmooshedFileMapper.load(SmooshedFileMapper.java:89) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.java.util.common.io.smoosh.Smoosh.map(Smoosh.java:32) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.IndexIO$V9IndexLoader.load(IndexIO.java:550) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:205) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:195) ~[druid-processing-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:913) ~[druid-server-28.0.1.jar:28.0.1]
        at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:786) ~[druid-server-28.0.1.jar:28.0.1]
        at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:250) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:240) ~[guava-31.1-jre.jar:?]
        at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:122) ~[guava-31.1-jre.jar:?]
        ... 3 more
2024-01-22T14:37:24,948 WARN [task-runner-0-priority-0] org.apache.druid.segment.realtime.firehose.ServiceAnnouncingChatHandlerProvider - handler[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z] not currently registered, ignoring.
2024-01-22T14:37:24,950 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
  "status" : "FAILED",
  "duration" : 474,
  "errorMsg" : "java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org...",
  "location" : {
    "host" : null,
    "port" : -1,
    "tlsPort" : -1
  }
}
2024-01-22T14:37:25,157 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.http.security.StateResourceFilter to GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,181 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.http.SegmentListerResource to GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,189 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.QueryResource to GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,190 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.segment.realtime.firehose.ChatHandlerResource to GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,193 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.http.security.ConfigResourceFilter to GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,197 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.query.lookup.LookupListeningResource to GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,199 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.query.lookup.LookupIntrospectionResource to GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,200 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.StatusResource to GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:25,221 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@573f2e67{/,null,AVAILABLE}
2024-01-22T14:37:25,232 INFO [main] org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@69fe8c75{HTTP/1.1, (http/1.1)}{0.0.0.0:8102}
2024-01-22T14:37:25,233 INFO [main] org.eclipse.jetty.server.Server - Started @5130ms
2024-01-22T14:37:25,233 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [ANNOUNCEMENTS]
2024-01-22T14:37:25,233 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Successfully started lifecycle [module]
2024-01-22T14:37:25,252 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
2024-01-22T14:37:25,255 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
2024-01-22T14:37:25,261 INFO [main] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@69fe8c75{HTTP/1.1, (http/1.1)}{0.0.0.0:8102}
2024-01-22T14:37:25,261 INFO [main] org.eclipse.jetty.server.session - node0 Stopped scavenging
2024-01-22T14:37:25,262 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@573f2e67{/,null,STOPPED}
2024-01-22T14:37:25,291 INFO [main] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting graceful shutdown of task[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z].
2024-01-22T14:37:25,292 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
2024-01-22T14:37:25,307 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore.
2024-01-22T14:37:25,315 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2024-01-22T14:37:25,422 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x101bf1fb80d0233
2024-01-22T14:37:25,422 INFO [main] org.apache.zookeeper.ZooKeeper - Session: 0x101bf1fb80d0233 closed
2024-01-22T14:37:25,434 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]
Finished peon task
@winsmith winsmith changed the title Task crashes in middle manager when ingesting JSON keys that contain a comma Index Task crashes in middle manager when ingesting JSON keys that contain a comma Jan 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Area - Ingestion Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@winsmith @asdf2014