[Task]: refactor: separate worker and pipeline construction module

[laysakura]

What needs to happen?

IMO, mod internals is a bad one.

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Kelvinyu1117]

Let me work on it. Can you provide more background information?

[laysakura]

@Kelvinyu1117 I appreciate your proposal! Although this is a refactoring issue, it requires knowledges about Beam internal.
You may want to check the post: #1 (comment)

My motivation here is that I want to separate client codes and worker (server-side) codes. Since Pipeline, PValue, and other structs with type parameters are used only from client side, I would like to separate them from the mirror types (ones without type parameters).

As you may feel, this issue is not well broken-down for now. If you feel this issue difficult, could you wait for me to create good-first issues?

[Kelvinyu1117]

Let me study the technical doc of Beam internal first then. For the server-side code, are you referring to the Fn API?

[laysakura]

@Kelvinyu1117

For the server-side code, are you referring to the Fn API?

Correct.

---

After I talked with several people in the Beam Summit 2023, I realized we would be re-write the existing Rust codes.
It would be more important to add features than making locally-optimal refactors.
Then, let me close this issue, sorry.

---

For Beam internal docs, you may find some interesting ones from https://cwiki.apache.org/confluence/display/BEAM/Design+Documents.

I'm happy if you can start from #46, #47, or even add new tasks in https://github.com/laysakura/beam/wiki/Task-dependencies-graph.