[Bug]: Kafka Consumer step fails when run inside async web service

[justin-tomlinson]

Apache Hop version?

2.17

Java version?

17.0.16

Operating system

Linux

What happened?

I have an async web service which starts a workflow. The workflow has a pipeline step that consumes from kafka using the kafka consumer (standard, not beam engine) the kafka consumer step has the child step defined and created.

Works ok in v2.15

Confirmed that this runs on Hop 2.17 & 2.16 with hop-run cmd and also with hop server call using executeWorkflow endpoint.

When running using asyncRun?service= in 2.17 or 2.16 it fails with errors as below:

2026/04/30 10:22:42 - touchpoint_sms_unsubscribe_members - Executing this pipeline using the Local Pipeline Engine with run configuration 'local'
2026/04/30 10:22:42 - touchpoint_sms_unsubscribe_members - Execution started for pipeline [touchpoint_sms_unsubscribe_members]
2026/04/30 10:22:42 - touchpoint_sms_unsubscribe_members_child - Executing this pipeline using the Local Pipeline Engine with run configuration 'Kafka'
2026/04/30 10:22:42 - touchpoint_sms_unsubscribe_members_child - Execution started for pipeline [touchpoint_sms_unsubscribe_members_child]
2026/04/30 10:22:42 - Kafka Consumer 2.0 - ERROR: Error initializing sub-transformation
2026/04/30 10:22:42 - Kafka Consumer 2.0 - ERROR: org.apache.hop.core.exception.HopException:
2026/04/30 10:22:42 - Kafka Consumer 2.0 - Unable to load and initialize sub pipeline
2026/04/30 10:22:42 - Kafka Consumer 2.0 -  at java.lang.Thread.run (Thread.java:840)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -  at org.apache.hop.pipeline.transform.TransformInitThread.run (TransformInitThread.java:66)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -  at org.apache.hop.pipeline.transforms.kafka.consumer.KafkaConsumerInput.init (KafkaConsumerInput.java:98)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -  at org.apache.hop.pipeline.transforms.kafka.consumer.KafkaConsumerInput.initSubPipeline (KafkaConsumerInput.java:135)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -  at org.apache.hop.pipeline.TransformWithMappingMeta.replaceVariableValues (TransformWithMappingMeta.java:316)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -  at org.apache.hop.pipeline.TransformWithMappingMeta.replaceVariableValues (TransformWithMappingMeta.java:309)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -  at org.apache.hop.pipeline.TransformWithMappingMeta.isInternalVariable (TransformWithMappingMeta.java:324)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -  at org.apache.hop.pipeline.TransformWithMappingMeta.isWorkflowInternalVariable (TransformWithMappingMeta.java:333)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -  at java.util.ImmutableCollections$SetN.contains (ImmutableCollections.java:937)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -  at java.util.Objects.requireNonNull (Objects.java:209)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -
2026/04/30 10:22:42 - Kafka Consumer 2.0 -      at org.apache.hop.pipeline.transforms.kafka.consumer.KafkaConsumerInput.initSubPipeline(KafkaConsumerInput.java:204)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -      at org.apache.hop.pipeline.transforms.kafka.consumer.KafkaConsumerInput.init(KafkaConsumerInput.java:98)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -      at org.apache.hop.pipeline.transform.TransformInitThread.run(TransformInitThread.java:66)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -      at java.base/java.lang.Thread.run(Thread.java:840)
2026/04/30 10:22:42 - Kafka Consumer 2.0 - Caused by: java.lang.NullPointerException
2026/04/30 10:22:42 - Kafka Consumer 2.0 -      at java.base/java.util.Objects.requireNonNull(Objects.java:209)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -      at java.base/java.util.ImmutableCollections$SetN.contains(ImmutableCollections.java:937)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -      at org.apache.hop.pipeline.TransformWithMappingMeta.isWorkflowInternalVariable(TransformWithMappingMeta.java:333)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -      at org.apache.hop.pipeline.TransformWithMappingMeta.isInternalVariable(TransformWithMappingMeta.java:324)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -      at org.apache.hop.pipeline.TransformWithMappingMeta.replaceVariableValues(TransformWithMappingMeta.java:309)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -      at org.apache.hop.pipeline.TransformWithMappingMeta.replaceVariableValues(TransformWithMappingMeta.java:316)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -      at org.apache.hop.pipeline.transforms.kafka.consumer.KafkaConsumerInput.initSubPipeline(KafkaConsumerInput.java:135)
2026/04/30 10:22:42 - Kafka Consumer 2.0 -      ... 3 more
2026/04/30 10:22:42 - Kafka Consumer 2.0 - ERROR: Error initializing transform [Kafka Consumer 2]
2026/04/30 10:22:42 - touchpoint_sms_unsubscribe_members - ERROR: Transform [Kafka Consumer 2.0] failed to initialize!

Issue Priority

Priority: 3

Issue Component

Component: Hop Server

[justin-tomlinson]

Current work around for this is to trigger the workflow via the sync call executeWorkflow

[petertsneg]

@justin-tomlinson I encountered the same issue before. I found that starting from version 2.16, a new 'request header content variable' field was added to the asynchronous web service. It must be configured for the service to function correctly. Hope this helps!