The query_compiler.merge reconstructs the Right dataframe for every partition of Left Dataframe

[arunjose696]

The query_compiler.merge reconstructs the Right dataframe from its partitions for every partition of Left Dataframe, The concat operation results in higher memory consumption when the size of right dataframe is large.

A possible option is to combine the right Dataframe partitions to a single partition dataframe by calling a remote function. This single partiotion dataframe is then passed to each partition of left dataframe thus avoiding the reconstruction in every worker while doing merge.