No objects to concatenate issue with Fugue

[fdosani]

@goodwanghan Just an FYI, seems like there might be a corner case here. Appreciate your insights.

In the example bellow we have 2 dataframes with exclusive join values (x). So nothing should match. Forcing anything more than 2 for parallelism throws ValueError: No objects to concatenate.

from datacompy import Compare, is_match, report, unq_columns
import pandas as pd
import numpy as np

df1 = pd.DataFrame(np.random.randint(0, 5, size=(10000, 2)), columns=["x", "y"])
df2 = pd.DataFrame(np.random.randint(6, 11, size=(10000, 2)), columns=["x", "y"])

print(report(df1, df2, join_columns=["x"], parallelism=4))

throws an error:

File .../datacompy/datacompy/fugue.py:556, in _distributed_compare.<locals>._comp(df)
    554 def _comp(df: List[Dict[str, Any]]) -> List[List[Any]]:
    555     df1 = (
--> 556         pd.concat([pickle.loads(r["data"]) for r in df if r["left"]])
    557         .sort_values(df1_cols)
    558         .reset_index(drop=True)
    559     )
    560     df2 = (
    561         pd.concat([pickle.loads(r["data"]) for r in df if not r["left"]])
    562         .sort_values(df2_cols)
    563         .reset_index(drop=True)
    564     )
    565     comp = Compare(
    566         df1=df1,
    567         df2=df2,
   (...)
    573         cast_column_names_lower=False,
    574     )

File .../python3.9/site-packages/pandas/core/reshape/concat.py:425, in _Concatenator.__init__(self, objs, axis, join, keys, levels, names, ignore_index, verify_integrity, copy, sort)
    422     objs = list(objs)
    424 if len(objs) == 0:
--> 425     raise ValueError("No objects to concatenate")
    427 if keys is None:
    428     objs = list(com.not_none(*objs))

ValueError: No objects to concatenate

[fdosani]

@goodwanghan Just want to double check is it ok for me to assign this to you? only if you have time. No pressure.

[goodwanghan]

Yes I will start working on it this week

[fdosani]

Fixed. Closing