How to speed the data in lot of data

[Benjamin-zhangjb]

No description provided.

[Benjamin-zhangjb]

plot={'histogram':{'bayesian_blocks_bins': False}} Where it should be added

[neomatrix369]

plot={'histogram':{'bayesian_blocks_bins': False}} Where it should be added

Here is an example:

training_profile = train.profile_report(title='Pandas Profiling on training set', 
plot={'histogram': {'bins': 8}}, 
style={'full_width': True}, minify_html=True, pool_size=no_processors)

Thats how you use the parameters.

[rmokros]

df.shape (6.370.599, 33)
profile = ProfileReport( df,

plot={'histogram': {'bins': None}},

                    plot={'histogram':{'bayesian_blocks_bins': False}},
                    check_correlation_pearson=False,
                    correlations={
                        "pearson": False,
                        "spearman": False,
                        "kendall": False,
                        "phi_k": False,
                        "cramers": False,
                        "recoded":False})

AWS ml.m5.24xlarge (384 GB memory)
Result:
with 4 floats memory error convert this float to string and runs ok .....

[sbrugman]

The minimal mode uses even less computation as of v2.8.0. The release before that contained numerous performance optimizations as well, included disabling bayesian_blocks by default.

[mthomp89]

Running minimal=True on dataframe with shape (326878, 38) with profile report dumping out to html file. The profile report does not produce. Executed 5% sample of original dataframe and the profile report produced in 2-3 minutes. What other configuration settings can be turned off?

pandas_profiling v2.4
pandas v1.0.1

[sbrugman]

Running minimal=True on dataframe with shape (326878, 38) with profile report dumping out to html file. The profile report does not produce. Executed 5% sample of original dataframe and the profile report produced in 2-3 minutes. What other configuration settings can be turned off?

pandas_profiling v2.4
pandas v1.0.1

Upgrading to the latest version will speed up significantly.