You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For some reason, I'm getting an error because python is trying to serialize greenlets:
Traceback (most recent call last):
File "/home/apps/Logiciels/Python/python-3.5.1/lib/python3.5/runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "/home/apps/Logiciels/Python/python-3.5.1/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 302, in <module>
b.main()
File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 92, in main
self.run()
File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 290, in run
futures_startup()
File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 271, in futures_startup
run_name="__main__"
File "/home/lochar/python/lib/python3.5/site-packages/scoop/futures.py", line 64, in _startup
result = _controller.switch(rootFuture, *args, **kargs)
File "/home/lochar/python/lib/python3.5/site-packages/scoop/_control.py", line 230, in runController
execQueue.sendResult(future)
File "/home/lochar/python/lib/python3.5/site-packages/scoop/_types.py", line 384, in sendResult
self.socket.sendResult(future)
File "/home/lochar/python/lib/python3.5/site-packages/scoop/_comm/scoopzmq.py", line 319, in sendResult
pickle.HIGHEST_PROTOCOL,
TypeError: cannot serialize 'greenlet.greenlet' object
I'm not using greenlets, so I'm not sure why I'm getting this. I'm really trying to get started with scoop—I've never used mapreduce nor done parallelized code before, so I might also be making an obvious error. Here is my code:
import os
import re
from scoop import futures as fut
def get_sideeffect_sentences(f):
sideeffect = re.compile(r"\bside\s+effects?\b")
tagsplit = re.compile(r"\s*<[hp]>\s*")
for ln in open(f):
for sent in tagsplit.split(ln):
if sideeffect.search(sent.lower()):
yield sent
def writeout(x, y):
print(y, file=outf)
if __name__ == "__main__":
scratch = os.getenv("SCRATCH")
outf = open(scratch + "/now/sideeffect_sents.txt", "w")
lsfiles = [ scratch + "/now/text/" + f for f in os.listdir(scratch + "/now/text") ]
fut.mapReduce(get_sideeffect_sentences, writeout, lsfiles)
outf.close()
The command is "python -m scoop -n 8". I noticed there are Québécois-sounding last names in the docs, so perhaps it might help to say I've been trying that on Briarée (Calcul Québec computer).
The text was updated successfully, but these errors were encountered:
For some reason, I'm getting an error because python is trying to serialize greenlets:
I'm not using greenlets, so I'm not sure why I'm getting this. I'm really trying to get started with scoop—I've never used mapreduce nor done parallelized code before, so I might also be making an obvious error. Here is my code:
The command is "python -m scoop -n 8". I noticed there are Québécois-sounding last names in the docs, so perhaps it might help to say I've been trying that on Briarée (Calcul Québec computer).
The text was updated successfully, but these errors were encountered: