[multistage] On Multistage Engine Query Parallelism

[ankitsultana]

MSE Query Parallelism as it stands is quite complicated for users to understand. There are some other issues with it too:

• Options like stageParallelism can or cannot apply based on Runtime conditions which are complicated to understand for users.
• There are overlapping responsibilities. e.g. partition_parallelism is largely the same as stageParallelism.
• Names are confusing. e.g. "partition_size" controls the number of workers in the leaf stage.
• ... etc.

This is covered in detail in the following doc which also proposes a new behavior that the new optimizer will follow: https://docs.google.com/document/d/1h_IgCiUU4u0xQQ6lz4ZhzH0htd6NaJV8lFBE59MeQZA/edit?tab=t.0#heading=h.aq4kaj522uyn

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!