-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pydoop submit script fails #366
Comments
To see what went wrong you have to check the individual task logs. You can access them via the Hadoop web UI. |
After tyring multiple times, the console gives me these messages:
I opened "sys logs" from web UI and could not find any error or even warning messages, but "stderr" data is like this:
I searched for the message "Container exited with a non-zero exit code 143" and found that it may be related to the garbage collector or other memory allocation issues. If this is the case, how the default script Pydoop works with no problems! |
I see. Try tweaking the memory settings and good luck :) |
I am running my Hadoop in my single machine on VM installed with 10 GB RAM and 2 processing cores, Centos 7 yarn-site.xml
mapred-site.xml
|
That depends on many factors, including the Hadoop version you're running. You can try asking on the Hadoop mailing lists. In the Docker images we use for testing, the configuration is rather minimal. If you want, you can check it out here. |
I have tried to run the code for wordCount example linked here https://crs4.github.io/pydoop/tutorial/pydoop_script.html using the
pydoop script script.py hdfs_input hdfs_output
and it worked fine for me and I could see the results from HDFS. However when I try to run the full-featured version of the program using "Pydoop submit" linked here https://crs4.github.io/pydoop/tutorial/mapred_api.html#api-tutorial usingpydoop submit --upload-file-to-cache wc.py wc input output
it takes too much time while running without getting any response or result, also the map-reduce job looks like it got stuck and always get something like this in the terminal:Map-Reduce job fails when using "Pydoop submit"!!
What could cause the problem and how to solve it?
The text was updated successfully, but these errors were encountered: