UnicodeEncodeError

[anndro]

Hello,

When I try to run elephas I got some errors.

Command:
../spark/bin/spark-submit --driver-memory 2G --executor-memory 30G ./run_distributed.py -v

Output:
172.20.20.111 - - [14/Dec/2015 17:48:18] "GET /parameters HTTP/1.1" 200 -
15/12/14 17:48:56 WARN TaskSetManager: Lost task 7.0 in stage 1.0 (TID 43, 172.20.20.107): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/home/torque/mucahit/spark-1.5.2-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/worker.py", line 111, in main
process()
File "/home/torque/mucahit/spark-1.5.2-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/worker.py", line 106, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/home/torque/mucahit/spark-1.5.2-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/serializers.py", line 263, in dump_stream
vs = list(itertools.islice(iterator, batch))
File "/home/torque/mucahit/keras_venv/local/lib/python2.7/site-packages/elephas/spark_model.py", line 222, in train
put_deltas_to_server(deltas, self.master_url)
File "/home/torque/mucahit/keras_venv/local/lib/python2.7/site-packages/elephas/spark_model.py", line 36, in put_deltas_to_server
return urllib2.urlopen(request).read()
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 401, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 419, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
result = func(_args)
File "/usr/lib/python2.7/urllib2.py", line 1211, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1178, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "/home/torque/mucahit/keras_venv/lib/python2.7/httplib.py", line 963, in request
File "/home/torque/mucahit/keras_venv/lib/python2.7/httplib.py", line 997, in _send_request
File "/home/torque/mucahit/keras_venv/lib/python2.7/httplib.py", line 959, in endheaders
File "/home/torque/mucahit/keras_venv/lib/python2.7/httplib.py", line 819, in _send_output
File "/home/torque/mucahit/keras_venv/lib/python2.7/httplib.py", line 795, in send
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(_args)
UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 183: ordinal not in range(128)

at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)
at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:207)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

15/12/14 17:48:56 INFO TaskSetManager: Starting task 7

[maxpumperla]

Hi, thanks for reporting this. The error indicates that this question mark symbol below appears in either HTTP header or content, when updating parameters back from executers to master (while getting parameters apparently works.)

http://graphemica.com/%EF%BF%BD

Option 1: Master URL is broken. This is somewhat unlikely, as in line 215 it gets parameters from there. Can you check or even send me your master URL?
Option 2: For some reason the pickled parameters contain this symbol. This can actually only happen if something goes wrong with training, maybe no data on the node?! Hard to remote debug this one.

Could you tell me more about your setup? Python 2 or 3 (urllib has been moved)? Does it work in local mode? Does it break all the time or just here and there? Does it work with exactly one executer?

I would gladly discuss this further! Thanks.

[maxpumperla]

Hey @anndro, do you have any feedback for me? These serialization errors are fairly common with Spark and currently it's unclear to me if this is due to elephas or your setup.

[anndro]

Hello, Sorry for the delay I was working last 2 days to fix issue. I figured out the error was because of the python version.
I was using debian7 and python version was 2.7.3 Then I updated my enviroment to python 3.4. Now it's working good.
Thanks for help and this good library.

[maxpumperla]

Cool, glad to hear that. In case you have an interesting application / example to add, please let me know.