OverflowError in 1 Preprocess Data

[briandw]

I'm trying to run the notebooks in my own python 3.6 conda environment.

I'm running into a problem when running this code:
pairs = flattenlist(apply_parallel(get_function_docstring_pairs_list, df.content.tolist(), cpu_cores=4))

I see the following traceback:

---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/multiprocess/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/multiprocess/pool.py", line 44, in mapstar
    return list(map(*args))
  File "<ipython-input-16-3f34f247210c>", line 40, in get_function_docstring_pairs_list
    return [get_function_docstring_pairs(b) for b in blob_list]
  File "<ipython-input-16-3f34f247210c>", line 40, in <listcomp>
    return [get_function_docstring_pairs(b) for b in blob_list]
  File "<ipython-input-16-3f34f247210c>", line 23, in get_function_docstring_pairs
    source = astor.to_source(f)
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/code_gen.py", line 52, in to_source
    generator.result.append('\n')
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/node_util.py", line 143, in visit
    return visitor(node)
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/code_gen.py", line 320, in visit_FunctionDef
    if not self.indentation:
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/code_gen.py", line 218, in body
    self.indentation -= 1
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/code_gen.py", line 168, in write
    elif callable(item):
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/node_util.py", line 143, in visit
    return visitor(node)
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/code_gen.py", line 472, in visit_Return
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/code_gen.py", line 206, in conditional_write
    # Inform the caller that we wrote
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/code_gen.py", line 168, in write
    elif callable(item):
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/node_util.py", line 143, in visit
    return visitor(node)
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/code_gen.py", line 659, in visit_Tuple
    with self.delimit(node, op) as delimiters:
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/code_gen.py", line 268, in comma_list
    self.write(',' if trailing else '')
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/code_gen.py", line 168, in write
    elif callable(item):
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/node_util.py", line 143, in visit
    return visitor(node)
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/code_gen.py", line 627, in visit_Num
    delimiters.discard = delimiters.pp != pow_lhs
  File "/home/brian/.conda/envs/tmp/lib/python3.6/site-packages/astor/code_gen.py", line 619, in part
    self.write(s)
OverflowError: int too large to convert to float
"""

The above exception was the direct cause of the following exception:

OverflowError                             Traceback (most recent call last)
/media/HDD/brian/code_search/notebooks/general_utils.py in apply_parallel(func, data, cpu_cores)
     75         pool = Pool(cpu_cores)
---> 76         transformed_data = pool.map(func, chunked(data, chunk_size), chunksize=1)
     77     finally:

~/.conda/envs/tmp/lib/python3.6/site-packages/multiprocess/pool.py in map(self, func, iterable, chunksize)
    265         '''
--> 266         return self._map_async(func, iterable, mapstar, chunksize).get()
    267 

~/.conda/envs/tmp/lib/python3.6/site-packages/multiprocess/pool.py in get(self, timeout)
    643         else:
--> 644             raise self._value
    645 

OverflowError: int too large to convert to float

I'm not sure whats going on here. Any help appreciated.

[hamelsmu]

Are you using the docker container associated with this tutorial? Also make sure you use the exact copy of the fastai library in this repo

[briandw]

I wasn't using the docker. I prefer to have the ability to reproduce the environment that the code runs on myself. Thanks.

[shengyfu]

@briandw, I got the same error as you did, how did you resolve the issue?

[briandw]

I didn't get it resolved. I've moved on to making my own project to train sentence vectors. Check it out if you are interested. It should be easy to run in a condo environment with Fastai and pytorch 0.4.1
https://github.com/briandw/SiameseULMFiT

[hamelsmu]

Just so I understand what is the downside of using Docker for this tutorial? The web of dependencies can be hard to manage with just requirements.txt

[Laksh47]

Like @hamelsmu suggests, using Docker is the best thing to do! However, this issue is just a floating point arithmetic issue. One can workaround it by editing the library file code_gen.py

Just change the line "import math" to "from mpmath import mp as math" @briandw
Refer: http://mpmath.org/

[hydeparksnow]

I changed the following line in function get_function_docstring_pairs, worked for me
#except (AssertionError, MemoryError, SyntaxError, UnicodeEncodeError):
to
except:

[gauravkoradiya]

I also get same error :
RemoteTraceback Traceback (most recent call last)
RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\multiprocess\pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\multiprocess\pool.py", line 44, in mapstar
return list(map(*args))
File "E:\Mindtree_IDP\general_utils.py", line 184, in get_function_docstring_pairs_list
return [get_function_docstring_pairs(b) for b in blob_list]
File "E:\Mindtree_IDP\general_utils.py", line 184, in
return [get_function_docstring_pairs(b) for b in blob_list]
File "E:\Mindtree_IDP\general_utils.py", line 167, in get_function_docstring_pairs
source = astor.to_source(f)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\code_gen.py", line 52, in to_source
generator.visit(node)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\node_util.py", line 143, in visit
return visitor(node)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\code_gen.py", line 320, in visit_FunctionDef
self.body(node.body)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\code_gen.py", line 218, in body
self.write(*statements)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\code_gen.py", line 168, in write
visit(item)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\node_util.py", line 143, in visit
return visitor(node)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\code_gen.py", line 356, in visit_If
self.body(node.body)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\code_gen.py", line 218, in body
self.write(*statements)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\code_gen.py", line 168, in write
visit(item)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\node_util.py", line 143, in visit
return visitor(node)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\code_gen.py", line 472, in visit_Return
self.conditional_write(' ', node.value)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\code_gen.py", line 206, in conditional_write
self.write(*stuff)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\code_gen.py", line 168, in write
visit(item)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\node_util.py", line 143, in visit
return visitor(node)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\code_gen.py", line 627, in visit_Num
real = part(x.real if isinstance(x, complex) else x, imaginary=False)
File "C:\Users\GK\AppData\Local\Continuum\anaconda3\envs\mindtree\lib\site-packages\astor\code_gen.py", line 619, in part
if math.isinf(p):
OverflowError: int too large to convert to float
"""