Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeError: cannot serialize 'greenlet.greenlet' object #57

Open
mokawi opened this issue May 8, 2017 · 1 comment
Open

TypeError: cannot serialize 'greenlet.greenlet' object #57

mokawi opened this issue May 8, 2017 · 1 comment

Comments

@mokawi
Copy link

mokawi commented May 8, 2017

For some reason, I'm getting an error because python is trying to serialize greenlets:

Traceback (most recent call last):
  File "/home/apps/Logiciels/Python/python-3.5.1/lib/python3.5/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/apps/Logiciels/Python/python-3.5.1/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 302, in <module>
    b.main()
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 92, in main
    self.run()
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 290, in run
    futures_startup()
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 271, in futures_startup
    run_name="__main__"
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/futures.py", line 64, in _startup
    result = _controller.switch(rootFuture, *args, **kargs)
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/_control.py", line 230, in runController
    execQueue.sendResult(future)
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/_types.py", line 384, in sendResult
    self.socket.sendResult(future)
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/_comm/scoopzmq.py", line 319, in sendResult
    pickle.HIGHEST_PROTOCOL,
TypeError: cannot serialize 'greenlet.greenlet' object

I'm not using greenlets, so I'm not sure why I'm getting this. I'm really trying to get started with scoop—I've never used mapreduce nor done parallelized code before, so I might also be making an obvious error. Here is my code:

import os
import re

from scoop import futures as fut

def get_sideeffect_sentences(f):
    sideeffect = re.compile(r"\bside\s+effects?\b")
    tagsplit =  re.compile(r"\s*<[hp]>\s*")
    for ln in open(f):
        for sent in tagsplit.split(ln):
            if sideeffect.search(sent.lower()):
                yield sent

def writeout(x, y):
    print(y, file=outf)

if __name__ == "__main__":
    scratch = os.getenv("SCRATCH")
    outf = open(scratch + "/now/sideeffect_sents.txt", "w")

    lsfiles = [ scratch + "/now/text/" + f for f in os.listdir(scratch + "/now/text")  ]
    fut.mapReduce(get_sideeffect_sentences, writeout, lsfiles)

    outf.close()

The command is "python -m scoop -n 8". I noticed there are Québécois-sounding last names in the docs, so perhaps it might help to say I've been trying that on Briarée (Calcul Québec computer).

@yolsever
Copy link

I am getting the same error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants