Skip to content

fix: fix chat handler response parsing#19495

Merged
gianm merged 2 commits into
apache:masterfrom
jtuglu1:fix-task-chat-handler-response-parsing
May 22, 2026
Merged

fix: fix chat handler response parsing#19495
gianm merged 2 commits into
apache:masterfrom
jtuglu1:fix-task-chat-handler-response-parsing

Conversation

@jtuglu1
Copy link
Copy Markdown
Contributor

@jtuglu1 jtuglu1 commented May 21, 2026

Description

Overlord task report requests can sometimes be too eager during ingestion and ping a task before the http server servicing the chat requests has spun up. This causes 4xx/5xx to be returned, which are not correctly parsed by the chat client. While this doesn't explicitly fail the ingestion, it spams the logs and causes confusion.

This properly propagates the error code/body through to the response, so no parse exceptions occur.

Example:

2026-05-21T01:33:43,097 WARN [qtp1790748582-95] org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask - Encountered exception when getting live subtask report for task: task123
java.util.concurrent.ExecutionException: java.lang.RuntimeException: com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve subtype of [simple type, class org.apache.druid.indexer.report.TaskReport]: missing type id property 'type'
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 10] (through reference chain: org.apache.druid.indexer.report.TaskReport$ReportMap["error"])
	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:592) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:571) ~[guava-32.1.3-jre.jar:?]
	at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91) ~[guava-32.1.3-jre.jar:?]
	at org.apache.druid.common.guava.FutureUtils.get(FutureUtils.java:52) ~[druid-processing-34.0.0.jar:34.0.0]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.getTaskReport(ParallelIndexSupervisorTask.java:1881) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.getRowStatsAndUnparseableEventsForRunningTasks(ParallelIndexSupervisorTask.java:1711) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.doGetRowStatsAndUnparseableEventsParallelMultiPhase(ParallelIndexSupervisorTask.java:1688) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.doGetRowStatsAndUnparseableEvents(ParallelIndexSupervisorTask.java:1795) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.doGetLiveReports(ParallelIndexSupervisorTask.java:1838) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.getLiveReports(ParallelIndexSupervisorTask.java:1868) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[jersey-servlet-1.19.4.jar:1.19.4]
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.4.jar:1.19.4]
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[jersey-servlet-1.19.4.jar:1.19.4]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290) ~[guice-servlet-5.1.0.jar:?]
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280) ~[guice-servlet-5.1.0.jar:?]
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184) ~[guice-servlet-5.1.0.jar:?]
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89) ~[guice-servlet-5.1.0.jar:?]
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) ~[guice-servlet-5.1.0.jar:?]
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121) ~[guice-servlet-5.1.0.jar:?]
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133) ~[guice-servlet-5.1.0.jar:?]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.57.v20241219.jar:9.4.57.v20241219]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.57.v20241219.jar:9.4.57.v20241219]
	at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:84) ~[druid-server-34.0.0.jar:34.0.0]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.57.v20241219.jar:9.4.57.v20241219]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.57.v20241219.jar:9.4.57.v20241219]
	at org.apache.druid.server.initialization.jetty.StandardResponseHeaderFilterHolder$StandardResponseHeaderFilter.doFilter(StandardResponseHeaderFilterHolder.java:164) ~[druid-server-34.0.0.jar:34.0.0]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.57.v20241219.jar:9.4.57.v20241219]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.57.v20241219.jar:9.4.57.v20241219]
	at org.apache.druid.server.initialization.jetty.ResponseHeaderFilterHolder$ResponseHeaderFilter.doFilter(ResponseHeaderFilterHolder.java:100) ~[druid-server-34.0.0.jar:34.0.0]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.57.v20241219.jar:9.4.57.v20241219]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-...

Release note

Overlord task report requests can sometimes be too eager during ingestion and ping a task before the http server servicing the chat requests has spun up. This causes 4xx/5xx to be returned, which are not correctly parsed by the chat client. While this doesn't explicitly fail the ingestion, it spams the logs and causes confusion.

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@jtuglu1 jtuglu1 requested a review from gianm May 21, 2026 04:01
Copy link
Copy Markdown
Member

@FrankChen021 FrankChen021 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reviewed the code for correctness, edge cases, concurrency, and integration risks; no issues found.

Reviewed 5 of 5 changed files.


This is an automated review by Codex GPT-5.5

Copy link
Copy Markdown
Contributor

@gianm gianm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good to me. Could you update KubernetesTaskRunner too?

@jtuglu1 jtuglu1 force-pushed the fix-task-chat-handler-response-parsing branch from 4e272e2 to 690c6e6 Compare May 21, 2026 19:55
@jtuglu1 jtuglu1 requested a review from gianm May 22, 2026 03:39
@gianm gianm merged commit e5e31b7 into apache:master May 22, 2026
63 of 64 checks passed
@github-actions github-actions Bot added this to the 38.0.0 milestone May 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants