pandarallel_apply crashes with OverflowError: int too big to convert

[yeus]

Hi everyone,

I am getting this error here using parallel_apply in pandas:

  File "extract_specifications.py", line 156, in <module>
    extracted_data = df.parallel_apply(extract_raw_infos, axis=1)
  File "/home/tom/.local/lib/python3.6/site-packages/pandarallel/pandarallel.py", line 367, in closure
    kwargs,
  File "/home/tom/.local/lib/python3.6/site-packages/pandarallel/pandarallel.py", line 239, in get_workers_args
    zip(input_files, output_files, chunk_lengths)
  File "/home/tom/.local/lib/python3.6/site-packages/pandarallel/pandarallel.py", line 238, in <listcomp>
    for index, (input_file, output_file, chunk_length) in enumerate(
  File "/home/tom/.local/lib/python3.6/site-packages/pandarallel/pandarallel.py", line 169, in wrapper
    time=time,
  File "/home/tom/.local/lib/python3.6/site-packages/pandarallel/utils/inliner.py", line 34, in wrapper
    return function(*args, **kwargs)
  File "/home/tom/.local/lib/python3.6/site-packages/pandarallel/utils/inliner.py", line 464, in inline
    func_instructions, len(b"".join(pinned_pre_func_instructions_without_return))
  File "/home/tom/.local/lib/python3.6/site-packages/pandarallel/utils/inliner.py", line 34, in wrapper
    return function(*args, **kwargs)
  File "/home/tom/.local/lib/python3.6/site-packages/pandarallel/utils/inliner.py", line 314, in shift_instructions
    for instruction in instructions
  File "/home/tom/.local/lib/python3.6/site-packages/pandarallel/utils/inliner.py", line 314, in <genexpr>
    for instruction in instructions
  File "/home/tom/.local/lib/python3.6/site-packages/pandarallel/utils/inliner.py", line 34, in wrapper
    return function(*args, **kwargs)
  File "/home/tom/.local/lib/python3.6/site-packages/pandarallel/utils/inliner.py", line 293, in shift_instruction
    return bytes((operation,)) + int2python_bytes(python_ints2int(values) + qty)
  File "/home/tom/.local/lib/python3.6/site-packages/pandarallel/utils/inliner.py", line 34, in wrapper
    return function(*args, **kwargs)
  File "/home/tom/.local/lib/python3.6/site-packages/pandarallel/utils/inliner.py", line 71, in int2python_bytes
    return int.to_bytes(item, nb_bytes, "little")
OverflowError: int too big to convert

I am using

pandarallel == 1.4.2
pandas == 0.24.2
python == 3.6.9

Any idea how to proceed from here? I have basically no idea what could cause this bug. I suspect it might be related to the size of the data I have in one column (I save html from web pages in there). But otherwise no idea. I would help removing this bug(?) if I had some guidance here. Thx for helping.

[nalepae]

Could you please try without the progress bar and/or with only a small part of your dataset and tell me the result ?

Could you also print the result of : len(df) ?

[yeus]

Could you please try without the progress bar and/or with only a small part of your dataset and tell me the result ?

Could you also print the result of : len(df) ?

thx. So without the progress bar it seemed to work. Thx for the hint. As it is a quiet long computation it would be nice to have it though...

here is some more information about my dataset. it includes a lot of python objects holding quiet large strings (largest one being html_content)

len(df) == 4737

# columns: 
df.shape = (4737, 21)

# memory consumption of columns
df.memory_usage(deep=True) == 
Index             201736
url               524872
depth              37896
counter            37896
time              336327
filename          486537
encoding          297990
file_path         548118
tables            246232
main_content    13186430
html_content    22344093
language          279483
lists             243120
list_score         37896
description     16791103
table0d           227376
table1d           259776
tablexd           232944
table_len          37896
table_num          37896
table_score        37896
co_score           37896

[bensdm]

same here, any way to make it work with progress bar?

[MarvinT]

Why not just use the parallel version of tqdm instead of trying to build your own progress bar?

[orena1]

I'm using the parallel version of tqdm and I still get this error...

[pacman100]

I am also getting the same error. Trying with the progress bar set to False seems to work. It would be better if this would have worked with the progress bar.

[nbrosse]

Same error also. It woud be nice to have the progress bar.

[Collonville]

Are there any updates for this issue? I have the same problem and can't solve it.

[aburneraccount]

I get the same error no matter what size df I use or how many cores I use, on python 3.8. I checked and the value being passed to int.to_bytes in inliner.py is 308

[wangchong666]

just edit PYTHON_PATH/python3.6/site-packages/pandarallel/utils/inliner.py line 71

retrun int.to_bytes(item%(1<<nb_bytes*8), nb_bytes, "little")

[yeus]

just edit PYTHON_PATH/python3.6/site-packages/pandarallel/utils/inliner.py line 71

retrun int.to_bytes(item%(1<<nb_bytes*8), nb_bytes, "little")

could we have this as a patch? Or does it interfere with some other stuff?

[yechafengyun]

Same Problem

[yechafengyun]

Same Problem

i check my apply function work correctly when progress_bar is set False. It seems to be something relating to the version stuff.
Pls fix it

[RobSpectre]

Same happening here. @wangchong666's fix does work - would love to see this released in the module.

[s-arjun]

Same here. @wangchong666 's fix initiated the progressbar, but no progress was made. it was just stuck, even for a small dataset.
Any fix/patch would be helpful

[RobSpectre]

Confirming this bug still persists in 1.5.4.

[nalepae]

On it.

[park]

Same here. Waiting for a fix. Thanks.

[dmytrostriletskyi]

Same here. Waiting for a fix. Thanks.

[nalepae]

Workaround before the real fix: If this happens, just remove the progress bar. Le mar. 8 févr. 2022 à 10:48, Dmytro Striletskyi ***@***.***> a écrit :

…

Same here. Waiting for a fix. Thanks. — Reply to this email directly, view it on GitHub <#63 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABFW7VX2LE5PQOXATFYLAELU2DRIBANCNFSM4J6OC2TA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[nalepae]

On Pandarallel v1.5.6, I temporarily deactivated the inliner, which causes this issue.

Positive impact: This bug is solve.
Negative impact: In some cases, progress bars themselves will slow down computation.