error ./prepare.sh: line 25: INPUT_HDFS: unbound variable #93

mittalsumeet · 2015-05-15T00:57:22Z

Hi
I am trying to use HiBench 4.0 for some performance benchmarking of HDP Hadoop 2.6.0.2.2.4.2-2
I have set the following properties (I dont have spark, I just want to test mapreduce)
hibench.hadoop.home /usr/lib/hadoop
hibench.spark.home /PATH/TO/YOUR/SPARK/ROOT
hibench.hdfs.master hdfs://10.0.2.15:50070

But i get the following error:
Any idea why? What configuration am I missing? The README in HiBench 4.0 doesnt say about any other configurations.

root@sandbox prepare]# ./prepare.sh
Parsing conf: /root/HiBench/conf/00-default-properties.conf
Parsing conf: /root/HiBench/conf/10-data-scale-profile.conf
Parsing conf: /root/HiBench/conf/99-user_defined_properties.conf
Parsing conf: /root/HiBench/workloads/sort/conf/00-sort-default.conf
Parsing conf: /root/HiBench/workloads/sort/conf/10-sort-userdefine.conf
Traceback (most recent call last):
File "/root/HiBench/bin/functions/load-config.py", line 440, in
load_config(conf_root, workload_root, workload_folder)
File "/root/HiBench/bin/functions/load-config.py", line 154, in load_config
generate_optional_value()
File "/root/HiBench/bin/functions/load-config.py", line 209, in generate_optional_value
if hadoop_version[0] != '1': # hadoop2? or CDH's MR1?
IndexError: string index out of range
/root/HiBench/bin/functions/workload-functions.sh: line 33: .: filename argument required
.: usage: . filename [arguments]
start HadoopPrepareSort bench
./prepare.sh: line 25: INPUT_HDFS: unbound variable

JessyLiao · 2015-05-22T08:42:16Z

Hi
I had same issue. My env is Cloudera Manager CDH 5.4.0
My properties is

hibench.hadoop.home /opt/cloudera/parcels/CDH/lib/hadoop
hibench.spark.home /opt/cloudera/parcels/CDH/lib/spark
hibench.hdfs.master hdfs://master1:50070/

My error message ~
[root@master1 conf]# /HiBench-master/workloads/pagerank/prepare/prepare.sh
Parsing conf: /HiBench-master/conf/00-default-properties.conf
Parsing conf: /HiBench-master/conf/10-data-scale-profile.conf
Parsing conf: /HiBench-master/conf/99-user_defined_properties.conf
Parsing conf: /HiBench-master/workloads/pagerank/conf/00-pagerank-default.conf
Parsing conf: /HiBench-master/workloads/pagerank/conf/10-pagerank-userdefine.conf
Parsing conf: /HiBench-master/workloads/pagerank/conf/properties.conf
Traceback (most recent call last):
File "/HiBench-master/bin/functions/load-config.py", line 440, in
load_config(conf_root, workload_root, workload_folder)
File "/HiBench-master/bin/functions/load-config.py", line 154, in load_config
generate_optional_value()
File "/HiBench-master/bin/functions/load-config.py", line 209, in generate_optional_value
if hadoop_version[0] != '1': # hadoop2? or CDH's MR1?
IndexError: string index out of range
/HiBench-master/bin/functions/workload-functions.sh: line 33: .: filename argument required
.: usage: . filename [arguments]
start HadoopPreparePagerank bench
/HiBench-master/workloads/pagerank/prepare/prepare.sh: line 25: INPUT_HDFS: unbound variable

GoodAsh · 2015-05-22T17:47:29Z

Receiving the same error with hdp 2.2.0.0-2041 here, has anyone solved the problem by adjusting configurations or changing the code?

GraceH · 2015-05-22T23:48:56Z

yes. we will take a look at this. thanks for reporting that.

-Grace
sent from mobile phone

-------- �� --------
��Re: [HiBench] error ./prepare.sh: line 25: INPUT_HDFS: unbound variable (#93)
��GoodAsh
��intel-hadoop/HiBench
��

Receiving the same error here, has anyone solved the problem by adjusting configurations or changing the code?

��
Reply to this email directly or view it on GitHubhttps://github.com//issues/93#issuecomment-104725998.

lvsoft · 2015-05-23T02:00:45Z

Seems like it failed to detect hadoop version.
The script'll execute <hadoop_bin>/hadoop version | head -1 | cut -d \ -f 2 to get your hadoop dist info.
According to the trace log, the shell cmd seems returned nothing.
Could you paste the result of <hadoop_bin>/hadoop version for reference?

Besides, you can set hibench.hadoop.version and hibench.hadoop.release in your 99-user_defined_properties.conf to bypass this probe procedures. For example:

     hibench.hadoop.version     hadoop2
     hibench.hadoop.release     cdh5

Note: The value of hibench.hadoop.version could be hadoop1 or hadoop2, and The value of hibench.hadoop.release could be cdh4, cdh5 and apache. Please set properly according to your environment.

JessyLiao · 2015-05-24T14:47:36Z

I try to set hibench.hadoop.version and hibench.hadoop.release in my 99-user_defined_properties.conf.
hibench.hadoop.version hadoop2
hibench.hadoop.release cdh5

----My 99-user_defined_properties.conf----
hibench.hadoop.home /opt/cloudera/parcels/CDH/
hibench.spark.home /opt/cloudera/parcels/CDH/
hibench.hdfs.master hdfs://master1:50070/

hibench.spark.master yarn-client
hibench.hadoop.executable ${hibench.hadoop.home}/bin/hadoop
hibench.hadoop.version hadoop2
hibench.hadoop.release cdh5
hibench.spark.version spark1.3

====Run prepare.sh ====
[root@master1 conf]# /HiBench-master/workloads/join/prepare/prepare.sh
Parsing conf: /HiBench-master/conf/00-default-properties.conf
Parsing conf: /HiBench-master/conf/10-data-scale-profile.conf
Parsing conf: /HiBench-master/conf/99-user_defined_properties.conf
Parsing conf: /HiBench-master/workloads/join/conf/00-join-default.conf
Parsing conf: /HiBench-master/workloads/join/conf/10-join-userdefine.conf
This filename pattern "/opt/cloudera/parcels/CDH//share/hadoop/mapreduce2/hadoop-mapreduce-examples-.jar" is required to match only one file.
However, there's no file found, please fix it.
Traceback (most recent call last):
File "/HiBench-master/bin/functions/load-config.py", line 440, in
load_config(conf_root, workload_root, workload_folder)
File "/HiBench-master/bin/functions/load-config.py", line 154, in load_config
generate_optional_value()
File "/HiBench-master/bin/functions/load-config.py", line 272, in generate_optional_value
HibenchConf["hibench.hadoop.examples.jar"] = OneAndOnlyOneFile(HibenchConf['hibench.hadoop.home'] + "/share/hadoop/mapreduce2/hadoop-mapreduce-examples-.jar")
File "/HiBench-master/bin/functions/load-config.py", line 113, in OneAndOnlyOneFile
raise Exception("Need to match one and only one file!")
Exception: Need to match one and only one file!
/HiBench-master/bin/functions/workload-functions.sh: line 33: .: filename argument required
.: usage: . filename [arguments]
start HadoopPrepareJoin bench
/HiBench-master/workloads/join/prepare/prepare.sh: line 25: INPUT_HDFS: unbound variable

lvsoft · 2015-05-24T16:35:52Z

Can you go to your folder /opt/cloudera/parcels/CDH//share/hadoop/mapreduce2/, and take a look if there's one and only one jar file exists which matches hadoop-mapreduce-examples-*.jar?

It seems the init script can't find the hadoop-mapreduce-examples-*.jar.
And, if the file is placed elsewhere, you'll need to set hibench.hadoop.examples.jar to the file path in your 99-user_defined_properties.conf to bypass this auto configure step.

TYoung1221 · 2015-05-26T11:09:54Z

I am running a test using spark 1.3 and hadoop 2.6
It seems that I had a same problem. But I think I fix it.

//My error msg is:
Parsing conf: /home/yang/HiBench/conf/00-default-properties.conf
Parsing conf: /home/yang/HiBench/conf/10-data-scale-profile.conf
Parsing conf: /home/yang/HiBench/conf/99-user_defined_properties.conf
Parsing conf: /home/yang/HiBench/workloads/wordcount/conf/00-wordcount-default.conf
Parsing conf: /home/yang/HiBench/workloads/wordcount/conf/10-wordcount-userdefine.conf
Probing spark verison, may last long at first time...
spark://localhost:7077 localhost
Traceback (most recent call last):
File "/home/yang/HiBench/bin/functions/load-config.py", line 440, in
load_config(conf_root, workload_root, workload_folder)
File "/home/yang/HiBench/bin/functions/load-config.py", line 154, in load_config
generate_optional_value()
File "/home/yang/HiBench/bin/functions/load-config.py", line 338, in generate_optional_value
assert 0, "Get workers from spark master's web UI page failed, reason:%s\nPlease check your configurations, network settings, proxy settings, or set hibench.masters.hostnames and hibench.slaves.hostnames manually to bypass auto-probe" % e
AssertionError: Get workers from spark master's web UI page failed, reason:[Errno socket error] [Errno 111] Connection refused
Please check your configurations, network settings, proxy settings, or set hibench.masters.hostnames and hibench.slaves.hostnames manually to bypass auto-probe
/home/yang/HiBench/bin/functions/workload-functions.sh: line 33: .: filename argument required
.: usage: . filename [arguments]
start HadoopPrepareWordcount bench
./prepare.sh: line 25: INPUT_HDFS: unbound variable

Because there is an socket error, so I modify the "hibench.spark.master" in "99-user_defined_properties.conf" file as "http://hostname:4040". And the error didn't occur.

I think the problem is that, you should make sure to export HADOOP_HOME and SPARK_HOME first.

lvsoft · 2015-05-26T11:21:59Z

@TYoung1221 actually, this should not be right...
HiBench will try to access your spark master's 8080 port, to fetch the slave node lists from web page. In most default conf, 8080 will be the port of Spark web UI. However, this may failed if you set web UI to another port number.

As the assertion information suggests, you can set hibench.masters.hostnames and hibench.slaves.hostnames(space separated if your have multiple slave nodes) in your 99-user_defined_properties.conf manually to bypass this auto probe step.

chinadragon0515 · 2015-06-05T08:48:25Z

Anyone has a solution for this error,
I set these properties,

Hadoop home

hibench.hadoop.home /opt/mapr/hadoop/hadoop-2.5.1/

Spark home

hibench.spark.home /opt/mapr/spark/spark-1.2.1/

HDFS master

hibench.hdfs.master maprfs:///
hibench.hadoop.version hadoop2./bin/run-all.sh
Prepare wordcount ...
Exec script: /mnt/mapr/HiBench-master/workloads/wordcount/prepare/prepare.sh
Parsing conf: /mnt/mapr/HiBench-master/conf/00-default-properties.conf
Parsing conf: /mnt/mapr/HiBench-master/conf/10-data-scale-profile.conf
Parsing conf: /mnt/mapr/HiBench-master/conf/99-user_defined_properties.conf
Parsing conf: /mnt/mapr/HiBench-master/workloads/wordcount/conf/00-wordcount-default.conf
Parsing conf: /mnt/mapr/HiBench-master/workloads/wordcount/conf/10-wordcount-userdefine.conf
Traceback (most recent call last):
File "/mnt/mapr/HiBench-master/bin/functions/load-config.py", line 440, in
load_config(conf_root, workload_root, workload_folder)
File "/mnt/mapr/HiBench-master/bin/functions/load-config.py", line 154, in load_config
generate_optional_value()
File "/mnt/mapr/HiBench-master/bin/functions/load-config.py", line 299, in generate_optional_value
HibenchConf["hibench.sleep.job.jar"] = HibenchConf['hibench.hadoop.examples.test.jar']
KeyError: 'hibench.hadoop.examples.test.jar'
/mnt/mapr/HiBench-master/bin/functions/workload-functions.sh: line 33: .: filename argument required
.: usage: . filename [arguments]
start HadoopPrepareWordcount bench
/mnt/mapr/HiBench-master/workloads/wordcount/prepare/prepare.sh: line 25: INPUT_HDFS: unbound variable
ERROR: wordcount prepare failed!
Run all done!
hibench.hadoop.release mapr
hibench.masters.hostnames h07
hibench.slaves.hostnames h06 h07 h08 h09

And run the test case wordcount in mapreduce only, I still get error

lvsoft · 2015-06-11T05:05:50Z

I've confirmed this is a bug and is related to #102

- Skip hostname probe mechanism if masters/slaves hostname is defined in custom user configuration.

PtdThant · 2015-12-14T05:40:34Z

Hello,
I am now using apache hadoop 2.7.1 and try to run HiBench benchmark on that Hadoop cluster. I have a problem with the following error (unbound variable). I am just a beginner so could you help me how to work on it? Thank you very much in advance.

[phyo@rain29 terasort]$ ./prepare/prepare.sh
Parsing conf: /home/phyo/HADOOP_ALLINONE/HiBench-HiBench-4.0/conf/00-default-properties.conf
Parsing conf: /home/phyo/HADOOP_ALLINONE/HiBench-HiBench-4.0/conf/10-data-scale-profile.conf
Parsing conf: /home/phyo/HADOOP_ALLINONE/HiBench-HiBench-4.0/conf/99-user_defined_properties.conf
Parsing conf: /home/phyo/HADOOP_ALLINONE/HiBench-HiBench-4.0/workloads/terasort/conf/00-terasort-default.conf
Parsing conf: /home/phyo/HADOOP_ALLINONE/HiBench-HiBench-4.0/workloads/terasort/conf/10-terasort-userdefine.conf
ERROR, execute cmd: '( /home/phyo/HADOOP_ALLINONE/hadoop-2.7.1/bin/yarn node -list 2> /dev/null | grep RUNNING )' timedout.
STDOUT:

STDERR:

Please check!
Traceback (most recent call last):
File "/home/phyo/HADOOP_ALLINONE/HiBench-HiBench-4.0/bin/functions/load-config.py", line 447, in
load_config(conf_root, workload_root, workload_folder)
File "/home/phyo/HADOOP_ALLINONE/HiBench-HiBench-4.0/bin/functions/load-config.py", line 154, in load_config
generate_optional_value()
File "/home/phyo/HADOOP_ALLINONE/HiBench-HiBench-4.0/bin/functions/load-config.py", line 358, in generate_optional_value
assert 0, "Get workers from spark master's web UI page failed, reason:%s\nplease set hibench.masters.hostnames and hibench.slaves.hostnames manually" % e
AssertionError: Get workers from spark master's web UI page failed, reason:( /home/phyo/HADOOP_ALLINONE/hadoop-2.7.1/bin/yarn node -list 2> /dev/null | grep RUNNING ) executed timedout for 100 seconds
please set hibench.masters.hostnames and hibench.slaves.hostnames manually
/home/phyo/HADOOP_ALLINONE/HiBench-HiBench-4.0/bin/functions/workload-functions.sh: line 33: .: filename argument required
.: usage: . filename [arguments]
start HadoopPrepareTerasort bench
./prepare/prepare.sh: line 25: INPUT_HDFS: unbound variable

CocoaWang · 2016-04-12T08:06:25Z

I am just like you @PtdThant .Hava you got any solutions?Thanks!

PtdThant · 2016-04-13T06:34:29Z

Hello Mr Wang Shaui, Nice to meet you! Thank you very much for your kindness! I have finished that one. And I am just a beginner trying to do research using Hadoop. And if I have any problems, may I ask you later. Sincerely, Phyo

CocoaWang · 2016-04-14T11:37:09Z

Nice to meet you too!
I am a beginner too.I just completed wordcount in single mode.Hadoop-hdfs2.7+spark1.5 +hibench .
I am happy to learn hadoop with you together!
You can share your problems with me.

asarraf · 2017-08-29T22:01:57Z

The way this issue was solved for me was by exporting the JAVA_HOME and adding it into the PATH in the .bashrc for the hdfs user (I was running with HDFS user).

lvsoft added the duplicate label Jun 11, 2015

lvsoft pushed a commit to lvsoft/HiBench that referenced this issue Jun 11, 2015

fix Intel-bigdata#102 and Intel-bigdata#93

1cab31c

- Skip hostname probe mechanism if masters/slaves hostname is defined in custom user configuration.

lvsoft mentioned this issue Jun 11, 2015

Hotfix 102 and others #103

Merged

lvsoft closed this as completed in #103 Jul 2, 2015

lvsoft pushed a commit to lvsoft/HiBench that referenced this issue Oct 15, 2015

fix Intel-bigdata#102 and Intel-bigdata#93

597d590

- Skip hostname probe mechanism if masters/slaves hostname is defined in custom user configuration.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error ./prepare.sh: line 25: INPUT_HDFS: unbound variable #93

error ./prepare.sh: line 25: INPUT_HDFS: unbound variable #93

mittalsumeet commented May 15, 2015

JessyLiao commented May 22, 2015

GoodAsh commented May 22, 2015

GraceH commented May 22, 2015

lvsoft commented May 23, 2015

JessyLiao commented May 24, 2015

lvsoft commented May 24, 2015

TYoung1221 commented May 26, 2015

lvsoft commented May 26, 2015

chinadragon0515 commented Jun 5, 2015

lvsoft commented Jun 11, 2015

PtdThant commented Dec 14, 2015

CocoaWang commented Apr 12, 2016

PtdThant commented Apr 13, 2016 via email

CocoaWang commented Apr 14, 2016

asarraf commented Aug 29, 2017

error ./prepare.sh: line 25: INPUT_HDFS: unbound variable #93

error ./prepare.sh: line 25: INPUT_HDFS: unbound variable #93

Comments

mittalsumeet commented May 15, 2015

JessyLiao commented May 22, 2015

GoodAsh commented May 22, 2015

GraceH commented May 22, 2015

lvsoft commented May 23, 2015

JessyLiao commented May 24, 2015

lvsoft commented May 24, 2015

TYoung1221 commented May 26, 2015

lvsoft commented May 26, 2015

chinadragon0515 commented Jun 5, 2015

Hadoop home

Spark home

HDFS master

lvsoft commented Jun 11, 2015

PtdThant commented Dec 14, 2015

CocoaWang commented Apr 12, 2016

PtdThant commented Apr 13, 2016 via email

CocoaWang commented Apr 14, 2016

asarraf commented Aug 29, 2017