[Feature Request]: Support running multi-language pipelines locally without Docker

[chamikaramj]

What would you like to happen?

Currently, multi-language pipelines require Docker execution mode (Docker SDK harness containers when executing the pipeline). This means that multi-language pipelines may not work appropriately for environments where pipelines have to be started from Docker containers and run locally (for example, when running pipelines within a K8 cluster).

This is to track work related to supporting this potentially with new Prism runner.

cc: @robertwb @lostluck

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[lostluck]

Thank you, I'm planning on filling the litany of issues for Prism work After I Get Back From Vacation in July.

[lostluck]

I think I misunderstood this when originally filed, and didn't read "without docker".

But once my current prism task is complete technically this should function with prism, assuming each Xlang source is able to specify loopback mode. As it stands, that's always per SDK/per expansion service. I do know that combining Go SDK Loopback + Java Docker works as one would hope, as I just needed it...

Prism could in principle also support "process execution" mode to accomplish this as well, but I haven't looked at that as the Go SDK hasn't yet supported it. Otherwise I don't know how one would accomplish it.

[chamikaramj]

Thanks Rebo. That makes sense. I filed this as a feature request since some users wanted to run x-lang locally without Docker. Agree that there's more than runner support needed to address this.