ingest accepts image but then fails with ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)

[ghost]

Using Cloudera QuickStart CDH 5.3 VirtualBox, I was able to build the project for CDH5.3 and yarn, but cannot ingest an image (about 450MB geotif). The ingest command says "accepted", but the YARN job fails:

[cloudera@quickstart mrgeo]$ mrgeo-cmd/src/main/scripts/mrgeo ingest River09Q100B.tif/River09Q100B.tif --output River09Q100B.pyramid
/home/cloudera/git/mrgeo/mrgeo-cmd/mrgeo-cmd-distribution/target:/home/cloudera/git/mrgeo/mrgeo-cmd/mrgeo-cmd-distribution/target/lib/*
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/cloudera/git/mrgeo/mrgeo-cmd/mrgeo-cmd-distribution/target/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
*** checking River09Q100B.tif/River09Q100B.tif accepted ***
15/03/24 05:27:34 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
15/03/24 05:31:29 ERROR ingest.IngestImage: IngestImage exited with error
[cloudera@quickstart mrgeo]$ 

The verbose mode has some additional info:

[cloudera@quickstart mrgeo]$ mrgeo-cmd/src/main/scripts/mrgeo ingest River09Q100B.tif/River09Q100B.tif --output River09Q100B.pyramid --verbose
/home/cloudera/git/mrgeo/mrgeo-cmd/mrgeo-cmd-distribution/target:/home/cloudera/git/mrgeo/mrgeo-cmd/mrgeo-cmd-distribution/target/lib/*
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/cloudera/git/mrgeo/mrgeo-cmd/mrgeo-cmd-distribution/target/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
*** checking River09Q100B.tif/River09Q100B.tif15/03/27 02:17:37 INFO geotools.GeotoolsRasterUtils: Loading missing epsg codes
15/03/27 02:17:41 INFO data.DataProviderFactory: Skipping image ingest data provider org.mrgeo.data.accumulo.ingest.AccumuloImageIngestDataProviderFactory because isValid returned false
15/03/27 02:17:41 INFO data.DataProviderFactory: Skipping mrs image data provider org.mrgeo.data.accumulo.image.AccumuloMrsImageDataProviderFactory because isValid returned false
15/03/27 02:17:42 WARN imageio.gdalframework: Failed to load the GDAL native libs. This is not a problem unless you need to use the GDAL plugins: they won't be enabled.
java.lang.UnsatisfiedLinkError: no gdaljni in java.library.path
15/03/27 02:21:00 INFO HSQLDB4C5A88DA50.ENGINE: dataFileCache open start
15/03/27 02:21:00 INFO HSQLDB4C5A88DA50.ENGINE: Checkpoint start
15/03/27 02:21:00 INFO HSQLDB4C5A88DA50.ENGINE: Checkpoint end
 accepted ***
15/03/27 02:24:25 INFO ingest.IngestImage: Ingest inputs (1)
15/03/27 02:24:25 INFO ingest.IngestImage:    River09Q100B.tif/River09Q100B.tif
15/03/27 02:24:26 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
15/03/27 02:24:26 INFO client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
15/03/27 02:24:29 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
15/03/27 02:24:30 INFO Configuration.deprecation: io.sort.spill.percent is deprecated. Instead, use mapreduce.map.sort.spill.percent
15/03/27 02:24:30 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
15/03/27 02:24:30 INFO format.IngestImageSplittingInputFormat: Spill size for splitting is: 79691776b
15/03/27 02:24:30 INFO format.IngestImageSplittingInputFormat:   reading: hdfs://quickstart.cloudera:8020/user/cloudera/River09Q100B.tif/River09Q100B.tif
15/03/27 02:24:30 INFO format.IngestImageSplittingInputFormat:     zoomlevel: 14
15/03/27 02:28:40 INFO mapreduce.JobSubmitter: number of splits:2450
15/03/27 02:28:40 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1427193016034_0007
15/03/27 02:28:41 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
15/03/27 02:28:41 INFO impl.YarnClientImpl: Submitted application application_1427193016034_0007
15/03/27 02:28:41 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1427193016034_0007/
15/03/27 02:28:41 INFO mapreduce.Job: Running job: job_1427193016034_0007
15/03/27 02:29:06 INFO mapreduce.Job: Job job_1427193016034_0007 running in uber mode : false
15/03/27 02:29:06 INFO mapreduce.Job:  map 0% reduce 0%
15/03/27 02:29:06 INFO mapreduce.Job: Job job_1427193016034_0007 failed with state FAILED due to: Application application_1427193016034_0007 failed 2 times due to AM Container for appattempt_1427193016034_0007_000002 exited with  exitCode: 1 due to: Exception from container-launch.
Container id: container_1427193016034_0007_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
    at org.apache.hadoop.util.Shell.run(Shell.java:455)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
15/03/27 02:29:06 INFO mapreduce.Job: Counters: 0
15/03/27 02:29:06 ERROR ingest.IngestImage: IngestImage exited with error

I am not sure if my build is not correct (I had to make many tweaks to get it done) or if there is something wrong with my Hadoop config (see "No job jar file set. " warning). Anybody there who managed to get this done with CDH 5.3 and YARN? As I said I am using QS VirtualBox with no extra settings, running as user cloudera with sudo rights, etc.

[ttislerdg]

I am running successfully on CDH5.3.2 yarn in pseudo-distributed mode. One suggestion is to make sure you are building mrgeo with yarn support: ./build cdh532 --yarn

If that still doesn't work, it looks like a map/reduce job was started, but failed in the early stages. If you look in one of the container logs it may give more information.