[Bug]: Remote Execution of workflows does not work because it tries to connect to 127.0.0.1

[usbrandon]

Apache Hop version?

2.4.0-SNAPSHOT (2023-03-08 07.37.34)

Java version?

11

Operating system

Windows

What happened?

I ran a simple workflow that runs a pipeline that outputs 10 rows to the logs. If you remote run the pipeline (export resources), it works fine. If you try to remote run the workflow, it seems to connect to the remote hop server, but when it tries to run the pipeline, for whatever reason the remote server tries to connect to itself using 127.0.0.1 instead of the IP address the server bound itself to.

If it could bind both to localhost and the host IP maybe this would be fine. Shouldn't hop-server try that to begin with? Alternatively, why wouldn't the server try to run whatever pipelines and workloads against the bound IP that was called to get things going?

2023/03/08 22:24:42 - workflow - Starting action [simple-pipe.hpl]
2023/03/08 22:24:42 - simple-pipe.hpl - Using run configuration [remote (ep-vdi-etl)]
2023/03/08 22:24:42 - simple-pipe - Executing this pipeline using the Remote Pipeline Engine with run configuration 'remote (ep-vdi-etl)'
2023/03/08 22:24:42 - simple-pipe.hpl - ERROR: Unable to prepare for execution of the pipeline
2023/03/08 22:24:42 - simple-pipe.hpl - ERROR: org.apache.hop.core.exception.HopException:
2023/03/08 22:24:42 - simple-pipe.hpl - Error preparing remote pipeline
2023/03/08 22:24:42 - simple-pipe.hpl -
2023/03/08 22:24:42 - simple-pipe.hpl - org.apache.http.conn.HttpHostConnectException: Connect to ep-etl-vdi:8999 [ep-etl-vdi/127.0.1.1] failed: Connection refused (Connection refused)
2023/03/08 22:24:42 - simple-pipe.hpl - Connect to ep-etl-vdi:8999 [ep-etl-vdi/127.0.1.1] failed: Connection refused (Connection refused)
2023/03/08 22:24:42 - simple-pipe.hpl -
2023/03/08 22:24:42 - simple-pipe.hpl -
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.hop.pipeline.engines.remote.RemotePipelineEngine.prepareExecution(RemotePipelineEngine.java:258)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.hop.pipeline.engines.remote.RemotePipelineEngine.execute(RemotePipelineEngine.java:597)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.hop.workflow.actions.pipeline.ActionPipeline.execute(ActionPipeline.java:609)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.hop.workflow.Workflow.executeFromStart(Workflow.java:655)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.hop.workflow.Workflow.executeFromStart(Workflow.java:795)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.hop.workflow.Workflow.executeFromStart(Workflow.java:439)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.hop.workflow.Workflow.startExecution(Workflow.java:300)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.hop.workflow.engines.local.LocalWorkflowEngine.startExecution(LocalWorkflowEngine.java:249)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.hop.www.StartWorkflowServlet.lambda$runWorkflow$0(StartWorkflowServlet.java:231)
2023/03/08 22:24:42 - simple-pipe.hpl - at java.base/java.lang.Thread.run(Thread.java:829)
2023/03/08 22:24:42 - simple-pipe.hpl - Caused by: org.apache.hop.core.exception.HopException:
2023/03/08 22:24:42 - simple-pipe.hpl - org.apache.http.conn.HttpHostConnectException: Connect to ep-etl-vdi:8999 [ep-etl-vdi/127.0.1.1] failed: Connection refused (Connection refused)
2023/03/08 22:24:42 - simple-pipe.hpl - Connect to ep-etl-vdi:8999 [ep-etl-vdi/127.0.1.1] failed: Connection refused (Connection refused)
2023/03/08 22:24:42 - simple-pipe.hpl -
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.hop.pipeline.engines.remote.RemotePipelineEngine.sendToHopServer(RemotePipelineEngine.java:398)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.hop.pipeline.engines.remote.RemotePipelineEngine.prepareExecution(RemotePipelineEngine.java:254)
2023/03/08 22:24:42 - simple-pipe.hpl - ... 9 more
2023/03/08 22:24:42 - simple-pipe.hpl - Caused by: org.apache.http.conn.HttpHostConnectException: Connect to ep-etl-vdi:8999 [ep-etl-vdi/127.0.1.1] failed: Connection refused (Connection refused)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.hop.server.HopServer.executeAuth(HopServer.java:678)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.hop.server.HopServer.sendXml(HopServer.java:545)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.hop.pipeline.engines.remote.RemotePipelineEngine.sendToHopServer(RemotePipelineEngine.java:362)
2023/03/08 22:24:42 - simple-pipe.hpl - ... 10 more
2023/03/08 22:24:42 - simple-pipe.hpl - Caused by: java.net.ConnectException: Connection refused (Connection refused)
2023/03/08 22:24:42 - simple-pipe.hpl - at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
2023/03/08 22:24:42 - simple-pipe.hpl - at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
2023/03/08 22:24:42 - simple-pipe.hpl - at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
2023/03/08 22:24:42 - simple-pipe.hpl - at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
2023/03/08 22:24:42 - simple-pipe.hpl - at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
2023/03/08 22:24:42 - simple-pipe.hpl - at java.base/java.net.Socket.connect(Socket.java:609)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
2023/03/08 22:24:42 - simple-pipe.hpl - at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
2023/03/08 22:24:42 - simple-pipe.hpl - ... 22 more
2023/03/08 22:24:42 - workflow - Finished action [simple-pipe.hpl] (result=[false])
2023/03/08 22:24:42 - workflow - Workflow execution finished
2023/03/08 22:24:42 - workflow - Workflow duration : 0.061 seconds [ 0.061" ]
2023/03/08 22:24:45 - Hop - Workflow execution has ended

Issue Priority

Priority: 2

Issue Component

Component: Hop Server

[usbrandon]

simple-pipe.zip
Workflow and pipeline for reproduction

[usbrandon]

I figured out this is a ME problem. Hop does not make it visible to users what pipelines on the workflow canvas will use to execute. When using a remote config for the workflow, all the pipelines may magically decide they will try to execute remotely too. In reality, we want to run the workflow remotely and when it lands, everything it does runs "local" to the remote server.