Kafka indexing service duplicate key exception in druid_pendingSegments table #16587

baaaaast · 2024-06-12T13:31:16Z

Affected Version

29.0.1

Description

KIS configuration :

replica > 1
lateMessageRejectionPeriod of several days

{
    "type": "kafka",
    "dataSchema":{
        "dataSource":"my-datasource",
        "parser":{
            "type":"avro_stream",
            "avroBytesDecoder" : {
                "type" : "schema_registry",
                "url" : "https://schema-registry.url"
            },
            "parseSpec": {
                "format":"timeAndDims",
                "timestampSpec": {
                    "column": "timestamp"
                },
                "dimensionsSpec":{
                    "dimensions":["my-dimension1","my-dimension2",...],
                    "spatialDimensions":[]
                }
            }
        },
        "metricsSpec":[
          {
              "type" : "longSum",
              "name" : "my-name",
              "fieldName": "my-name"
          },...
        ],
        "granularitySpec":{
            "type":"uniform",
            "segmentGranularity":"HOUR",
            "queryGranularity":"HOUR"
        }
    },
    "tuningConfig" : {
        "type" : "kafka",
        "maxRowsInMemory": 150000,
        "maxBytesInMemory" : 0,
        "maxRowsPerSegment": 5000000,
        "workerThreads": 5,
        "intermediatePersistPeriod" : "PT10M",
        "maxPendingPersists" : 0,
        "forceExtendableShardSpecs": true,
        "indexSpec" : {
            "bitmap" : {
                "type" : "roaring"
            },
            "dimensionCompression" : "lz4",
            "metricCompression" : "lz4",
            "longEncoding" : "longs"
        },
        "resetOffsetAutomatically" : false
    },
    "ioConfig": {
        "type" : "kafka",
        "topic": "my-topic",
        "taskCount": 2,
        "replicas": 2,
        "taskDuration": "PT1H",
        "useEarliestOffset": false,
        "lateMessageRejectionPeriod": "P8D",
        "earlyMessageRejectionPeriod": "PT3H",
        "consumerProperties": {
            "bootstrap.servers": "broker-1:9092,broker-2:9092,broker-3:9092"
        }
    }
}

Particularity of the topic :

consumed topic can have several million messages per minute
messages with a timestamp to be dated several days in the past

=> KIS writes many segments in the past every hour.

Problem

After a while (after several intermediate persist periods), the overlord starts generating errors and returns a status 500 to the middlemanager. This causes the indexing task to fail.
I can't reproduce the problem if the kafka topic has a low rate of messages produced.

Overlord error

2024-06-12T08:46:42,877 WARN [qtp1884568750-98] org.apache.druid.indexing.overlord.http.OverlordResource - Failed to perform task action
java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.skife.jdbi.v2.exceptions.CallbackFailedException: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: java.sql.BatchUpdateException: Batch entry 1 INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, "end", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (('my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5'), ('my-datasource'), ('2024-06-12T08:46:42.875Z'), ('2024-06-12T01:00:00.000Z'), ('2024-06-12T02:00:00.000Z'), ('index_kafka_my-datasource_0f3e4661a8cd3ee_0'), ('my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6'), ('A10883453F9AA74CDD231465CFF98E73DE3A2864'), ?) was aborted: ERROR: duplicate key value violates unique constraint "druid_pendingsegments_sequence_name_prev_id_sha1_key"
  Detail: Key (sequence_name_prev_id_sha1)=(A10883453F9AA74CDD231465CFF98E73DE3A2864) already exists.  Call getNextException to see other errors in the batch. [statement:"INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, \"end\", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (:id, :dataSource, :created_date, :start, :end, :sequence_name, :sequence_prev_id, :sequence_name_prev_id_sha1, :payload)", located:"null", rewritten:"null", arguments:{ positional:{}, named:{sequence_prev_id:'my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6',payload:[*, *...],start:'2024-06-12T01:00:00.000Z',end:'2024-06-12T02:00:00.000Z',id:'my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5',created_date:'2024-06-12T08:46:42.875Z',dataSource:'my-datasource',sequence_name:'index_kafka_my-datasource_0f3e4661a8cd3ee_0',sequence_name_prev_id_sha1:'A10883453F9AA74CDD231465CFF98E73DE3A2864'}, finder:[]}]
	at org.apache.druid.indexing.common.actions.LocalTaskActionClient.performAction(LocalTaskActionClient.java:98) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.LocalTaskActionClient.submit(LocalTaskActionClient.java:80) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:627) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:616) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.overlord.http.OverlordResource.asLeaderWith(OverlordResource.java:1108) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.overlord.http.OverlordResource.doAction(OverlordResource.java:613) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at jdk.internal.reflect.GeneratedMethodAccessor246.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[jersey-servlet-1.19.4.jar:1.19.4]
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.4.jar:1.19.4]
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[jersey-servlet-1.19.4.jar:1.19.4]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) ~[guice-servlet-4.1.0.jar:?]
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) ~[guice-servlet-4.1.0.jar:?]
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) ~[guice-servlet-4.1.0.jar:?]
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) ~[guice-servlet-4.1.0.jar:?]
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) ~[guice-servlet-4.1.0.jar:?]
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) ~[guice-servlet-4.1.0.jar:?]
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) ~[guice-servlet-4.1.0.jar:?]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.apache.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:73) ~[druid-server-29.0.1.jar:29.0.1]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:84) ~[druid-server-29.0.1.jar:29.0.1]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.apache.druid.server.initialization.jetty.StandardResponseHeaderFilterHolder$StandardResponseHeaderFilter.doFilter(StandardResponseHeaderFilterHolder.java:164) ~[druid-server-29.0.1.jar:29.0.1]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.apache.druid.server.security.AllowHttpMethodsResourceFilter.doFilter(AllowHttpMethodsResourceFilter.java:78) ~[druid-server-29.0.1.jar:29.0.1]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:74) ~[druid-server-29.0.1.jar:29.0.1]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.apache.druid.server.security.AllowAllAuthenticator$1.doFilter(AllowAllAuthenticator.java:84) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59) ~[druid-server-29.0.1.jar:29.0.1]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77) ~[druid-server-29.0.1.jar:29.0.1]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.util.concurrent.ExecutionException: org.skife.jdbi.v2.exceptions.CallbackFailedException: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: java.sql.BatchUpdateException: Batch entry 1 INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, "end", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (('my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5'), ('my-datasource'), ('2024-06-12T08:46:42.875Z'), ('2024-06-12T01:00:00.000Z'), ('2024-06-12T02:00:00.000Z'), ('index_kafka_my-datasource_0f3e4661a8cd3ee_0'), ('my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6'), ('A10883453F9AA74CDD231465CFF98E73DE3A2864'), ?) was aborted: ERROR: duplicate key value violates unique constraint "druid_pendingsegments_sequence_name_prev_id_sha1_key"
  Detail: Key (sequence_name_prev_id_sha1)=(A10883453F9AA74CDD231465CFF98E73DE3A2864) already exists.  Call getNextException to see other errors in the batch. [statement:"INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, \"end\", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (:id, :dataSource, :created_date, :start, :end, :sequence_name, :sequence_prev_id, :sequence_name_prev_id_sha1, :payload)", located:"null", rewritten:"null", arguments:{ positional:{}, named:{sequence_prev_id:'my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6',payload:[*, *...],start:'2024-06-12T01:00:00.000Z',end:'2024-06-12T02:00:00.000Z',id:'my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5',created_date:'2024-06-12T08:46:42.875Z',dataSource:'my-datasource',sequence_name:'index_kafka_my-datasource_0f3e4661a8cd3ee_0',sequence_name_prev_id_sha1:'A10883453F9AA74CDD231465CFF98E73DE3A2864'}, finder:[]}]
	at java.util.concurrent.CompletableFuture.reportGet(Unknown Source) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(Unknown Source) ~[?:?]
	at org.apache.druid.indexing.common.actions.LocalTaskActionClient.performAction(LocalTaskActionClient.java:90) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	... 85 more
Caused by: org.skife.jdbi.v2.exceptions.CallbackFailedException: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: java.sql.BatchUpdateException: Batch entry 1 INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, "end", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (('my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5'), ('my-datasource'), ('2024-06-12T08:46:42.875Z'), ('2024-06-12T01:00:00.000Z'), ('2024-06-12T02:00:00.000Z'), ('index_kafka_my-datasource_0f3e4661a8cd3ee_0'), ('my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6'), ('A10883453F9AA74CDD231465CFF98E73DE3A2864'), ?) was aborted: ERROR: duplicate key value violates unique constraint "druid_pendingsegments_sequence_name_prev_id_sha1_key"
  Detail: Key (sequence_name_prev_id_sha1)=(A10883453F9AA74CDD231465CFF98E73DE3A2864) already exists.  Call getNextException to see other errors in the batch. [statement:"INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, \"end\", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (:id, :dataSource, :created_date, :start, :end, :sequence_name, :sequence_prev_id, :sequence_name_prev_id_sha1, :payload)", located:"null", rewritten:"null", arguments:{ positional:{}, named:{sequence_prev_id:'my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6',payload:[*, *...],start:'2024-06-12T01:00:00.000Z',end:'2024-06-12T02:00:00.000Z',id:'my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5',created_date:'2024-06-12T08:46:42.875Z',dataSource:'my-datasource',sequence_name:'index_kafka_my-datasource_0f3e4661a8cd3ee_0',sequence_name_prev_id_sha1:'A10883453F9AA74CDD231465CFF98E73DE3A2864'}, finder:[]}]
	at org.skife.jdbi.v2.DBI.withHandle(DBI.java:284) ~[jdbi-2.63.1.jar:2.63.1]
	at org.apache.druid.metadata.SQLMetadataConnector.lambda$retryWithHandle$0(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:161) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.allocatePendingSegments(IndexerSQLMetadataStorageCoordinator.java:645) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegmentIds(TaskLockbox.java:696) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegments(TaskLockbox.java:466) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForInterval(SegmentAllocationQueue.java:491) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForBatch(SegmentAllocationQueue.java:433) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatch(SegmentAllocationQueue.java:351) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatchesDue(SegmentAllocationQueue.java:266) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	... 1 more
Caused by: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: java.sql.BatchUpdateException: Batch entry 1 INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, "end", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (('my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5'), ('my-datasource'), ('2024-06-12T08:46:42.875Z'), ('2024-06-12T01:00:00.000Z'), ('2024-06-12T02:00:00.000Z'), ('index_kafka_my-datasource_0f3e4661a8cd3ee_0'), ('my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6'), ('A10883453F9AA74CDD231465CFF98E73DE3A2864'), ?) was aborted: ERROR: duplicate key value violates unique constraint "druid_pendingsegments_sequence_name_prev_id_sha1_key"
  Detail: Key (sequence_name_prev_id_sha1)=(A10883453F9AA74CDD231465CFF98E73DE3A2864) already exists.  Call getNextException to see other errors in the batch. [statement:"INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, \"end\", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (:id, :dataSource, :created_date, :start, :end, :sequence_name, :sequence_prev_id, :sequence_name_prev_id_sha1, :payload)", located:"null", rewritten:"null", arguments:{ positional:{}, named:{sequence_prev_id:'my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6',payload:[*, *...],start:'2024-06-12T01:00:00.000Z',end:'2024-06-12T02:00:00.000Z',id:'my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5',created_date:'2024-06-12T08:46:42.875Z',dataSource:'my-datasource',sequence_name:'index_kafka_my-datasource_0f3e4661a8cd3ee_0',sequence_name_prev_id_sha1:'A10883453F9AA74CDD231465CFF98E73DE3A2864'}, finder:[]}]
	at org.skife.jdbi.v2.PreparedBatch.execute(PreparedBatch.java:148) ~[jdbi-2.63.1.jar:2.63.1]
	at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.insertPendingSegmentsIntoMetastore(IndexerSQLMetadataStorageCoordinator.java:1394) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.allocatePendingSegments(IndexerSQLMetadataStorageCoordinator.java:976) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.lambda$allocatePendingSegments$9(IndexerSQLMetadataStorageCoordinator.java:646) ~[druid-server-29.0.1.jar:29.0.1]
	at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
	at org.apache.druid.metadata.SQLMetadataConnector.lambda$retryWithHandle$0(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:161) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.allocatePendingSegments(IndexerSQLMetadataStorageCoordinator.java:645) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegmentIds(TaskLockbox.java:696) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegments(TaskLockbox.java:466) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForInterval(SegmentAllocationQueue.java:491) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForBatch(SegmentAllocationQueue.java:433) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatch(SegmentAllocationQueue.java:351) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatchesDue(SegmentAllocationQueue.java:266) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	... 1 more
Caused by: java.sql.BatchUpdateException: Batch entry 1 INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, "end", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (('my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5'), ('my-datasource'), ('2024-06-12T08:46:42.875Z'), ('2024-06-12T01:00:00.000Z'), ('2024-06-12T02:00:00.000Z'), ('index_kafka_my-datasource_0f3e4661a8cd3ee_0'), ('my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6'), ('A10883453F9AA74CDD231465CFF98E73DE3A2864'), ?) was aborted: ERROR: duplicate key value violates unique constraint "druid_pendingsegments_sequence_name_prev_id_sha1_key"
  Detail: Key (sequence_name_prev_id_sha1)=(A10883453F9AA74CDD231465CFF98E73DE3A2864) already exists.  Call getNextException to see other errors in the batch.
	at org.postgresql.jdbc.BatchResultHandler.handleCompletion(BatchResultHandler.java:186) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:590) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:912) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:936) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1733) ~[postgresql-42.7.2.jar:42.7.2]
	at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
	at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
	at org.skife.jdbi.v2.PreparedBatch.execute(PreparedBatch.java:138) ~[jdbi-2.63.1.jar:2.63.1]
	at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.insertPendingSegmentsIntoMetastore(IndexerSQLMetadataStorageCoordinator.java:1394) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.allocatePendingSegments(IndexerSQLMetadataStorageCoordinator.java:976) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.lambda$allocatePendingSegments$9(IndexerSQLMetadataStorageCoordinator.java:646) ~[druid-server-29.0.1.jar:29.0.1]
	at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
	at org.apache.druid.metadata.SQLMetadataConnector.lambda$retryWithHandle$0(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:161) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.allocatePendingSegments(IndexerSQLMetadataStorageCoordinator.java:645) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegmentIds(TaskLockbox.java:696) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegments(TaskLockbox.java:466) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForInterval(SegmentAllocationQueue.java:491) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForBatch(SegmentAllocationQueue.java:433) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatch(SegmentAllocationQueue.java:351) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatchesDue(SegmentAllocationQueue.java:266) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	... 1 more
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "druid_pendingsegments_sequence_name_prev_id_sha1_key"
  Detail: Key (sequence_name_prev_id_sha1)=(A10883453F9AA74CDD231465CFF98E73DE3A2864) already exists.
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2725) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2412) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:579) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:912) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:936) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1733) ~[postgresql-42.7.2.jar:42.7.2]
	at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
	at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
	at org.skife.jdbi.v2.PreparedBatch.execute(PreparedBatch.java:138) ~[jdbi-2.63.1.jar:2.63.1]
	at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.insertPendingSegmentsIntoMetastore(IndexerSQLMetadataStorageCoordinator.java:1394) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.allocatePendingSegments(IndexerSQLMetadataStorageCoordinator.java:976) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.lambda$allocatePendingSegments$9(IndexerSQLMetadataStorageCoordinator.java:646) ~[druid-server-29.0.1.jar:29.0.1]
	at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
	at org.apache.druid.metadata.SQLMetadataConnector.lambda$retryWithHandle$0(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:161) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.allocatePendingSegments(IndexerSQLMetadataStorageCoordinator.java:645) ~[druid-server-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegmentIds(TaskLockbox.java:696) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegments(TaskLockbox.java:466) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForInterval(SegmentAllocationQueue.java:491) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForBatch(SegmentAllocationQueue.java:433) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatch(SegmentAllocationQueue.java:351) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatchesDue(SegmentAllocationQueue.java:266) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	... 1 more

Postgresql error

2024-06-12 08:02:13.731 UTC [424455] druid_u@druid_db LOG:  execute S_7: INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, "end", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
2024-06-12 08:02:13.731 UTC [424455] druid_u@druid_db DETAIL:  parameters: $1 = 'my-datasource_2024-06-12T07:00:00.000Z_2024-06-12T08:00:00.000Z_2024-06-12T07:03:58.908Z_4', $2 = 'my-datasource', $3 = '2024-06-12T08:02:13.731Z', $4 = '2024-06-12T07:00:00.000Z', $5 = '2024-06-12T08:00:00.000Z', $6 = 'index_kafka_my-datasource_658fb00f8020ee2_0', $7 = 'my-datasource_2024-06-09T03:00:00.000Z_2024-06-09T04:00:00.000Z_2024-06-09T03:00:00.440Z_35', $8 = '53051B2F1036FED82C2E66195320F7A1EC8609C1', $9 = '\x***'
2024-06-12 08:02:13.731 UTC [424455] druid_u@druid_db LOG:  execute S_7: INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, "end", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
2024-06-12 08:02:13.731 UTC [424455] druid_u@druid_db DETAIL:  parameters: $1 = 'my-datasource_2024-06-12T07:00:00.000Z_2024-06-12T08:00:00.000Z_2024-06-12T07:03:58.908Z_3', $2 = 'my-datasource', $3 = '2024-06-12T08:02:13.731Z', $4 = '2024-06-12T07:00:00.000Z', $5 = '2024-06-12T08:00:00.000Z', $6 = 'index_kafka_my-datasource_658fb00f8020ee2_0', $7 = '', $8 = '53051B2F1036FED82C2E66195320F7A1EC8609C1', $9 = '\x***'
2024-06-12 08:02:13.731 UTC [424455] druid_u@druid_db ERROR:  duplicate key value violates unique constraint "druid_pendingsegments_sequence_name_prev_id_sha1_key"

=> We can see that the overlord tries to perform 2 consecutive INSERTs with the same sequence_name_prev_id_sha1.

Temporary solution

If I disable the batching segments allocation option by setting the druid.indexer.tasklock.batchSegmentAllocation property to false, the problem disappears.

The text was updated successfully, but these errors were encountered:

zargor · 2024-06-28T06:25:46Z

A couple days ago we upgraded to v30. Started to notice this error too. It turned out there is increased number of failed tasks. Now thinking of rolling back to v28, most because of kafka ingestion.

baaaaast · 2024-06-28T07:17:22Z

A couple days ago we upgraded to v30. Started to notice this error too. It turned out there is increased number of failed tasks. Now thinking of rolling back to v28, most because of kafka ingestion.
Did you try to set the property druid.indexer.tasklock.batchSegmentAllocation=false to the overlord ?

zargor · 2024-06-28T07:36:20Z

ruid.indexer.tasklock.batchSegmentAllocation=false

No we did not yet. Wondering how (much) it can hit the ingestion performance.

baaaaast · 2024-06-28T07:42:34Z

I haven't seen any difference at all: no lag for millions of messages consumed and no extra load on the overlord side.
I just don't have any more failed tasks

kfaraz · 2024-06-28T08:07:56Z

Thanks for reporting this issue, @baaaaast !
I will try to debug this and get back to you.

zargor · 2024-06-28T08:27:42Z

I haven't seen any difference at all: no lag for millions of messages consumed and no extra load on the overlord side. I just don't have any more failed tasks

Thanks, sure we'll try out and get back here. For the context, normally we ingest few millions messages per second.

zargor · 2024-06-28T21:15:11Z

Setting druid.indexer.tasklock.batchSegmentAllocation=false fixed the duplicate key error.
Looking forward to the complete resolution of this github issue 👍

zargor · 2024-09-25T07:50:53Z

@kfaraz is there any news on this matter? Thanks!

AmatyaAvadhanula · 2024-10-06T15:41:04Z

@zargor Here is a potential fix: #17262

Fixes #16587 Streaming ingestion tasks operate by allocating segments before ingesting rows. These allocations happen across replicas which may send different requests but must get the same segment id for a given (datasource, interval, version, sequenceName) across replicas. This patch fixes the bug by ignoring the previousSegmentId when skipLineageCheck is true.

Fixes apache#16587 Streaming ingestion tasks operate by allocating segments before ingesting rows. These allocations happen across replicas which may send different requests but must get the same segment id for a given (datasource, interval, version, sequenceName) across replicas. This patch fixes the bug by ignoring the previousSegmentId when skipLineageCheck is true.

Fixes #16587 Streaming ingestion tasks operate by allocating segments before ingesting rows. These allocations happen across replicas which may send different requests but must get the same segment id for a given (datasource, interval, version, sequenceName) across replicas. This patch fixes the bug by ignoring the previousSegmentId when skipLineageCheck is true. Co-authored-by: AmatyaAvadhanula <amatya.avadhanula@imply.io>

baaaaast added the Uncategorized problem report label Jun 12, 2024

asdf2014 added Bug Area - Ingestion and removed Uncategorized problem report labels Jun 13, 2024

kfaraz self-assigned this Jun 28, 2024

AmatyaAvadhanula mentioned this issue Oct 6, 2024

Fix batch segment allocation failure with replicas #17262

Merged

10 tasks

kfaraz closed this as completed in #17262 Oct 7, 2024

kfaraz mentioned this issue Oct 7, 2024

[Backport] Fix batch segment allocation failure with replicas (#17262) #17267

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kafka indexing service duplicate key exception in druid_pendingSegments table #16587

Kafka indexing service duplicate key exception in druid_pendingSegments table #16587

baaaaast commented Jun 12, 2024

zargor commented Jun 28, 2024

baaaaast commented Jun 28, 2024

zargor commented Jun 28, 2024

baaaaast commented Jun 28, 2024

kfaraz commented Jun 28, 2024

zargor commented Jun 28, 2024

zargor commented Jun 28, 2024

zargor commented Sep 25, 2024

AmatyaAvadhanula commented Oct 6, 2024

Kafka indexing service duplicate key exception in druid_pendingSegments table #16587

Kafka indexing service duplicate key exception in druid_pendingSegments table #16587

Comments

baaaaast commented Jun 12, 2024

Affected Version

Description

KIS configuration :

Particularity of the topic :

Problem

Overlord error

Postgresql error

Temporary solution

zargor commented Jun 28, 2024

baaaaast commented Jun 28, 2024

zargor commented Jun 28, 2024

baaaaast commented Jun 28, 2024

kfaraz commented Jun 28, 2024

zargor commented Jun 28, 2024

zargor commented Jun 28, 2024

zargor commented Sep 25, 2024

AmatyaAvadhanula commented Oct 6, 2024