OverflowError

[lfdversluis]

Using python 3.8, I am getting an OverflowError running apply_parallel:

OverflowError                             Traceback (most recent call last)
<ipython-input-5-a78fd5119887> in <module>
     37     grouped_df = df.groupby("id")
     38 
---> 39     grouped_df.parallel_apply(lookahead) \
     40         .to_parquet(output_location_look_ahead, compression='snappy', engine='pyarrow')
     41 

../miniconda3/lib/python3.8/site-packages/pandarallel/pandarallel.py in closure(data, func, *args, **kwargs)
    429         queue = manager.Queue()
    430 
--> 431         workers_args, chunk_lengths, input_files, output_files = get_workers_args(
    432             use_memory_fs,
    433             nb_requested_workers,

../miniconda3/lib/python3.8/site-packages/pandarallel/pandarallel.py in get_workers_args(use_memory_fs, nb_workers, progress_bar, chunks, worker_meta_args, queue, func, args, kwargs)
    284             raise OSError(msg)
    285 
--> 286         workers_args = [
    287             (
    288                 input_file.name,

../miniconda3/lib/python3.8/site-packages/pandarallel/pandarallel.py in <listcomp>(.0)
    293                 progress_bar == PROGRESS_IN_WORKER,
    294                 dill.dumps(
--> 295                     progress_wrapper(
    296                         progress_bar >= PROGRESS_IN_FUNC, queue, index, chunk_length
    297                     )(func)

../miniconda3/lib/python3.8/site-packages/pandarallel/pandarallel.py in wrapper(func)
    203     def wrapper(func):
    204         if progress_bar:
--> 205             wrapped_func = inline(
    206                 progress_pre_func,
    207                 func,

../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in wrapper(*args, **kwargs)
     32             raise SystemError("Python version should be 3.{5, 6, 7, 8}")
     33 
---> 34         return function(*args, **kwargs)
     35 
     36     return wrapper

../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in inline(pre_func, func, pre_func_arguments)
    485 
    486     func_instructions = tuple(get_instructions(func))
--> 487     shifted_func_instructions = shift_instructions(
    488         func_instructions, len(b"".join(pinned_pre_func_instructions_without_return))
    489     )

../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in wrapper(*args, **kwargs)
     32             raise SystemError("Python version should be 3.{5, 6, 7, 8}")
     33 
---> 34         return function(*args, **kwargs)
     35 
     36     return wrapper

../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in shift_instructions(instructions, qty)
    301     If Python version not in 3.{5, 6, 7}, a SystemError is raised.
    302     """
--> 303     return tuple(
    304         shift_instruction(instruction, qty)
    305         if bytes((instruction[0],))

../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in <genexpr>(.0)
    302     """
    303     return tuple(
--> 304         shift_instruction(instruction, qty)
    305         if bytes((instruction[0],))
    306         in (

../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in wrapper(*args, **kwargs)
     32             raise SystemError("Python version should be 3.{5, 6, 7, 8}")
     33 
---> 34         return function(*args, **kwargs)
     35 
     36     return wrapper

../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in shift_instruction(instruction, qty)
    291     """
    292     operation, *values = instruction
--> 293     return bytes((operation,)) + int2python_bytes(python_ints2int(values) + qty)
    294 
    295 

../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in wrapper(*args, **kwargs)
     32             raise SystemError("Python version should be 3.{5, 6, 7, 8}")
     33 
---> 34         return function(*args, **kwargs)
     35 
     36     return wrapper

../miniconda3/lib/python3.8/site-packages/pandarallel/utils/inliner.py in int2python_bytes(item)
     69 
     70     nb_bytes = 2 if python_version.minor == 5 else 1
---> 71     return int.to_bytes(item, nb_bytes, "little")
     72 
     73 

OverflowError: int too big to convert

[philwilkes]

Me too! pandarallel 1.5.2 and Python 3.8.1
It occurs when progress_bar=True

Also got the same error with Python 3.7.6

[lschneider-apixio]

any solution for this?

[robin-loche]

Same problem with python 3.7.5, only when progress_bar=True

[frava91]

Same issue with python 3.7.9, only when progress_bar=True

[sekeratib]

same issue here

[nalepae]

On it.

[nalepae]

On Pandarallel v1.5.6, I temporarily deactivated the inliner, which causes this issue.

Positive impact: This bug is solve.
Negative impact: In some cases, progress bars themselves will slow down computation.