pd.DataFrame don't always return the same dtype as in pandas

[c3-cjazra]

System information

modin 0.12.0
Ray 1.7.1
python 3.9.7

Describe the problem

import modin.pandas as modin_pd
import pandas as pandas_pd


df_modin = modin_pd.DataFrame({"b": modin_pd.Series([], dtype='bool'),
                "a": modin_pd.Series([], dtype='int64')
            }, index=[])

df_pandas = pandas_pd.DataFrame({"b": pandas_pd.Series([], dtype='bool'),
                "a": pandas_pd.Series([], dtype='int64')
            }, index=[])

print(df_pandas.dtypes)
print(df_modin.dtypes)

---- df_pandas.dtypes
b     bool
a    int64
dtype: object

----df_modin.dtypes
b    object. <-- not like pandas
a    object. <-- 
dtype: object

also _to_pandas() seems to modify the dtypes to object. For exmaple

json ='{"schema":{"fields":[{"name":"index","type":"string"},{"name":"b","type":"boolean"},{"name":"a","type":"integer"}],"primaryKey":["index"],"pandas_version":"0.20.0"},"data":[]}'
df_modin = modin_pd.read_json(a, orient='table')
df_pandas = pandas_pd.read_json(a, orient='table')

assert df_pandas.dtypes.equals(df_modin.dtypes) <-- true
 
print(df_pandas.dtypes)
print(df_modin._to_pandas().dtypes) <-- _to_pandas changes the dtype to object

--- df_pandas.dtypes
b     bool 
a    int64
dtype: object

---- df_modin._to_pandas().dtypes
b    object
a    object
dtype: object

[mvashishtha]

@c3-cjazra Thank you for reporting this issue! I've massaged your code snippets a little to reproduce each issue. The first is:

import modin.pandas as pd
import pandas

df_modin = pd.DataFrame({"b": pd.Series([], dtype='bool'),
                "a": pd.Series([], dtype='int64')
            }, index=[])

df_pandas = pandas.DataFrame({"b": pandas.Series([], dtype='bool'),
                "a": pandas.Series([], dtype='int64')
            }, index=[])

print(df_pandas.dtypes)
print(df_modin.dtypes)

And the second snippet:

import modin.pandas as pd
import pandas

json ='{"schema":{"fields":[{"name":"index","type":"string"},{"name":"b","type":"boolean"},{"name":"a","type":"integer"}],"primaryKey":["index"],"pandas_version":"0.20.0"},"data":[]}'
df_modin = pd.read_json(json, orient='table')
df_pandas = pandas.read_json(json, orient='table')

assert df_pandas.dtypes.equals(df_modin.dtypes) 
 
print(df_pandas.dtypes)
print(df_modin._to_pandas().dtypes) 

[mvashishtha]

Per the discussion here in my attempted fix #4108 , we don't yet know how to fix this.

[anmyachev]

The problem is fixed on the current master - 947e06b.