MemoryError channel recv

[andresriancho]

While running some scans using w3af, which uses pebble for parsing HTML documents I get MemoryError exceptions:

Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/eth/tools/virtualenvs/w3af/local/lib/python2.7/site-packages/pebble/pool/process.py", line 383, in worker_process
    for task in worker_get_next_task(channel, params.max_tasks):
  File "/home/eth/tools/virtualenvs/w3af/local/lib/python2.7/site-packages/pebble/pool/process.py", line 398, in worker_get_next_task
    yield fetch_task(channel)
  File "/home/eth/tools/virtualenvs/w3af/local/lib/python2.7/site-packages/pebble/pool/process.py", line 404, in fetch_task
    return task_transaction(channel)
  File "/home/eth/tools/virtualenvs/w3af/local/lib/python2.7/site-packages/pebble/pool/process.py", line 413, in task_transaction
    task = channel.recv()
  File "/home/eth/tools/virtualenvs/w3af/local/lib/python2.7/site-packages/pebble/pool/channel.py", line 90, in recv
    return self.reader.recv()
MemoryError

Any ideas what these could be? My research leads me to believe that this could be because of large HTML files being sent through the Pipe, but I was unable to confirm it.

Also, memory usage shouldn't be an issue, I have 64G of RAM and 64G of swap.

[andresriancho]

Run this test and it seems to be working well, this doesn't seem to reproduce the issue 👎

from pebble import ProcessPool
from concurrent.futures import TimeoutError


def function(foo):
    return foo


def task_done(future):
    try:
        result = future.result()  # blocks until results are ready
    except TimeoutError as error:
        print("Function took longer than %d seconds" % error.args[1])
    except Exception as error:
        print("Function raised %s" % error)
        print(error.traceback)  # traceback of the function
    else:
        print(len(result))


pool = ProcessPool(max_workers=5, max_tasks=10)

for i in xrange(10):
    future = pool.schedule(function, args=['A' * (10 ** i)], timeout=20)
    future.add_done_callback(task_done)

pool.close()
pool.join()

[andresriancho]

Potential root cause for this issue?

https://github.com/python/cpython/blob/2.7/Modules/_multiprocessing/pipe_connection.c#L70-L74

[noxdafox]

Yep, that's likely to be the culprit. The longer anser is in issue #29.

[noxdafox]

Closing this as duplicate of #29.