Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

[MaruthiD]

Hello Everybody
I am trying to running oryx 2 in Cloudera Express 5.3.0, with a single node but when i execute the BatchLayer, this return me the next warning

"WARN YarnClient Cluster Scheduler:71 Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory"

I look at my yarn, It has sufficient memory of 8GB, and 1Vcore.

I changed my conf file accordingly:

Number of executors to start

num-executors = 1

# Cores per executor
executor-cores = 1

# Memory per executor
executor-memory = "512m"

# Heap size for the Speed driver process
driver-memory = "512m"

How can i solve this problem?

Thanks,
Maruthi.

[srowen]

(Consider using the mailing list for questions? https://groups.google.com/a/cloudera.org/forum/#!forum/oryx-user This is more for handling pull requests and reporting specific issues.)

That does typically mean that YARN thinks it does not have enough resource to fulfill the request. First, check the YARN Resource Manager to verify how much RAM / cores it thinks it has available. It may not be managing all of your machine's resource, though, I imagine the default config does.

Can you show more of your config file? from the comment above it looks like you are configuring the speed instead of batch layer.

[MaruthiD]

Here i am trying als-example.conf file.

My config file is:

kafka-brokers = "localhost:9092"
zk-servers = "localhost:2181"
hdfs-base = "hdfs:///user/example/Oryx"
oryx {
input-topic {
broker = ${kafka-brokers}
lock = {
master = ${zk-servers}
}
}
update-topic {
broker = ${kafka-brokers}
lock = {
master = ${zk-servers}
}
}
batch {
streaming {
generation-interval-sec = 300
block-interval-sec = 5
num-executors = 1
executor-cores = 1
executor-memory = "1g"
driver-memory = "1g"
}
update-class = "com.cloudera.oryx.app.mllib.als.ALSUpdate"
storage {
data-dir = ${hdfs-base}"/data/"
model-dir = ${hdfs-base}"/model/"
}
ui {
port = 5040
}
}
speed {
model-manager-class = "com.cloudera.oryx.app.speed.als.ALSSpeedModelManager"
ui {
port = 5041
}
}
serving {
model-manager-class = "com.cloudera.oryx.app.serving.als.model.ALSServingModelManager"
application-resources = "com.cloudera.oryx.app.serving,com.cloudera.oryx.app.serving.als"
api {
port = 5080
}
}
}

[MaruthiD]

My yarn has 8GB of memory and 1vcore.

[srowen]

OK. Hm but with just 1 core, there isn't enough to run both the application manager and the app, both of which need at least 1 core. In fact, a Spark Streaming app needs at least 2 executor slots, since 1 will be taken by the receiver. I think you'd have to have at least 3 vcores to run any Spark Streaming app. Can you simply over-commit YARN and tell it that it should allocate at least 3 vcores?

If I'm right about this, I'll definitely document it.

[MaruthiD]

HI,
I will try to increase the vcores in yarn and let you know if it runs.

[srowen]

OK, reopen this if this turns out to not be the explanation.

[MaruthiD]

HI,
I tried to change the vcores. Now i have 8vcores. When i run the batch file. Here is my log:
2015-04-16 07:58:36,857 INFO BatchLayer:164 Creating message stream from topic
2015-04-16 07:58:36,938 INFO BatchLayer:201 Starting Spark Streaming
2015-04-16 07:58:37,060 INFO BatchLayer:208 Spark Streaming is running
2015-04-16 08:00:00,048 INFO BatchUpdateFunction:87 No data in current generation's RDD; nothing to do
2015-04-16 08:05:45,102 WARN AmIpFilter:157 Could not find proxy-user cookie, so user will not be set
2015-04-16 08:05:45,121 WARN AmIpFilter:157 Could not find proxy-user cookie, so user will not be set

While this batch layer is running, I started speed layer, this is accepted by yarn but it is in pending state not running till i stop the batch layer. How to make changes to yarn so that more than one job runs at a time?

Thanks,
Maruthi.

[MaruthiD]

HI,
I tried to change the vcores. Now i have 8vcores. When i run the batch file. Here is my log:
2015-04-16 07:58:36,857 INFO BatchLayer:164 Creating message stream from topic
2015-04-16 07:58:36,938 INFO BatchLayer:201 Starting Spark Streaming
2015-04-16 07:58:37,060 INFO BatchLayer:208 Spark Streaming is running
2015-04-16 08:00:00,048 INFO BatchUpdateFunction:87 No data in current generation's RDD; nothing to do
2015-04-16 08:05:45,102 WARN AmIpFilter:157 Could not find proxy-user cookie, so user will not be set
2015-04-16 08:05:45,121 WARN AmIpFilter:157 Could not find proxy-user cookie, so user will not be set

While this batch layer is running, I started speed layer, this is accepted by yarn but it is in pending state not running till i stop the batch layer. How to make changes to yarn so that more than one job runs at a time?

Thanks,
Maruthi.?

Maruthi Donthi
Java Developer
[aeverie-logo-med-res- signature size]
250 Parkway Drive Suite 150
Lincolnshire, Illinois 60069
203-218-6949(M)
maruthi@aeverie.commailto:rajesh@aeverie.com
http://www.aeverie.com/

---

From: Sean Owen notifications@github.com
Sent: Tuesday, April 14, 2015 10:15 AM
To: OryxProject/oryx
Cc: Maruthi@aeverie.com
Subject: Re: [oryx] Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory (#204)

OK, reopen this if this turns out to not be the explanation.

Reply to this email directly or view it on GitHubhttps://github.com//issues/204#issuecomment-92899641.

[srowen]

It still sounds like you are asking for more resource than your YARN cluster provisions. You can turn down the resource you ask for or increase the resource YARN provides. But that's the issue and solution.