TypeError: type of argument "df" must be one of (pandas.core.frame.DataFrame, pyspark.sql.dataframe.DataFrame); got NoneType insteadBug Report

[calculus-ask]

Current Behaviour

python version: 3.9.6
ydata_profiling: 4.0.0 (install using PIP)

import pandas as pd
from ydata_profiling import ProfileReport, compare

df1 = pd.read_csv("C:/Downloads/new.csv")
df2 = pd.read_csv("C:/Downloads/new1.csv")

df1_rep = ProfileReport(df1, title="One")
df2_rep = ProfileReport(df2, title="two")

comparison_report = compare([df1_rep, df2_rep])
also tried with
comparison_report = df1_rep.compare(df2_rep)

TypeError: type of argument "df" must be one of (pandas.core.frame.DataFrame, pyspark.sql.dataframe.DataFrame); got NoneType instead

The Same code works fine in pandas_profiling package.
Please let me know if you need additional information.

Expected Behaviour

Reports needs to generated

Data Description

name, age, country, state
kris, 20, usa, texas

Code that reproduces the bug

import pandas as pd
from ydata_profiling import ProfileReport, compare

df1 = pd.read_csv("C:/Downloads/new.csv")
df2 = pd.read_csv("C:/Downloads/new1.csv")

df1_rep = ProfileReport(df1, title="One")
df2_rep = ProfileReport(df2, title="two")

comparison_report = compare([df1_rep, df2_rep])
also tried with
comparison_report = df1_rep.compare(df2_rep)

pandas-profiling version

4.0.0

Dependencies

numpy: 1.22.4
pandas: 1.5.3

OS

Window 10

Checklist

• There is not yet another bug report for this issue in the issue tracker
• The problem is reproducible from this bug report. This guide can help to craft a minimal bug report.
• The issue has not been resolved by the entries listed under Common Issues.

[fabclmnt]

Hi @calculus-ask,

I might need more information from your side, as after replicating your environment I'm able to calculate the compare report. Please check the screenshots below:

Can you check whether pandas have successfully read your datasets?

I'll be happy to assist you more closely - please join us in Discord, I'll be around.

[calculus-ask]

Hi @fabclmnt

Thanks for your reply. Pandas read csv files correctly into dataframes.

print(df1.shape)
(24, 6)
print(df2.shape)
(136, 4)

df1_rep = ProfileReport(df1, title="One")
df_bk = df1_rep.df
print(df_bk.shape)
(24, 6)

df2_rep = ProfileReport(df2, title="two")
df_bk2 = df2_rep.df
print(df_bk2.shape)
(136, 4)

More information on the error:

comparison_report = compare([df1_rep, df2_rep])
C:\Users--\AppData\Local\Programs\Python\Python39\lib\site-packages\ydata_profiling\compare_reports.py:182: UserWarning: The datasets being profiled have a different set of columns. Only the left side profile will be calculated.
warnings.warn(
Summarize dataset: 0%| | 0/5 [00:00<?, ?it/s]
Traceback (most recent call last):

Cell In[15], line 1
comparison_report = compare([df1_rep, df2_rep])

File ~\AppData\Local\Programs\Python\Python39\lib\site-packages\ydata_profiling\compare_reports.py:309 in compare
profile = ProfileReport(None, config=_config)

File ~\AppData\Local\Programs\Python\Python39\lib\site-packages\typeguard_init_.py:1033 in wrapper
retval = func(*args, **kwargs)

File ~\AppData\Local\Programs\Python\Python39\lib\site-packages\ydata_profiling\profile_report.py:94 in init
self.__validate_inputs(df, minimal, tsmode, config_file, lazy)

File ~\AppData\Local\Programs\Python\Python39\lib\site-packages\typeguard_init_.py:1032 in wrapper
check_argument_types(memo)

File ~\AppData\Local\Programs\Python\Python39\lib\site-packages\typeguard_init_.py:875 in check_argument_types
raise TypeError(*exc.args) from None

TypeError: type of argument "df" must be one of (pandas.core.frame.DataFrame, pyspark.sql.dataframe.DataFrame); got NoneType instead

[fabclmnt]

@calculus-ask it seems that is related to your data, for some reason one of the data frames seems to be empty.

Is there any possibility that you can share your data with us? Or perhaps jump into a call?

Feel free to reach me out in Discord, I'll be around.

[sightama]

Hey I don't believe this is due to the data. I have the same problem occurring on my end with same steps above to reproduce. When I delved into the ydata/pandas-profiling code I can see where its occurring.
ydata-profiling -> compare_reports.py -> line 309
** profile = ProfileReport(None, config=_config)**

I'm assuming that strictness has been applied to ProfileReport and that's causing this, but idk past that.

To clarify I am passing in two ProfileReports that both have valid dataframes; I am not passing in a config so it uses default config setup.

Cheers

EDIT:
I also noticed if I instead pass in the summary obtained from report.get_description(), it fails at line 170 in compare_Reports because the summary does not contain a config key. Hope that helps!

More background: I'm executing this not in a notebook, but in code workflow in Pycharm on a Mac m2 silicon with python 3.10

[orsho]

I have the same issue as described above

[fabclmnt]

@sightama unfortunately I can't replicate the behavior. There is not need or hard requirement on passing the config. I can generate a report for 2 datasets that are the same, with no need to set a config.

The bug exists, but in the absence of being able to reproduce or more details on the code is provided, it will be hard for us to understand what is happening.

If someone can provide more details so we can replicate it, that would be highly helpful

[orsho]

Hi @fabclmnt, I think I found the solution for this bug. I don't know why it's happened but it relates to the pyspark library
pip uninstall pyspark fixed the issue for me

[fabclmnt]

@orsho thank you for that detail!
That might help to traceback the cause!
This shouldn't happen, if pandas DataFrames are used, even if PySpark is installed. We will have a look into it!

[EdwardMarsDS]

Hi @fabclmnt I stumbled across this issue this week, similar to the OP.

I've never had pyspark on my current machine, but I tried @orsho suggestion of uninstalling anyways, confirming that I did not have a sneaky install of pyspark on my machine.

I can likely reproducible dataset and code if necessary.