can not start the windowCli

[java8964]

Tested with CDH3 Update 3 (cdh3u3)

Here is error log, any idea what I did wrong?

~/hive/bin> hive --service windowingCli -w com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar
Hive history file=/tmp/yzhang/hive_job_log_yzhang_201202112135_1368506989.txt
Exception in thread "main" com.sap.hadoop.windowing.WindowingException: java.lang.NumberFormatException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:54)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190)
at com.sap.hadoop.windowing.cli.WindowingClient.connect(WindowingClient.groovy:126)
at com.sap.hadoop.windowing.cli.WindowingClient.(WindowingClient.groovy:38)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:54)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:198)
at com.sap.hadoop.windowing.WindowingHiveCliDriver.setupClient(WindowingHiveCliDriver.groovy:64)
at com.sap.hadoop.windowing.WindowingHiveCliDriver$setupClient.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
at com.sap.hadoop.windowing.WindowingHiveCliDriver.main(WindowingHiveCliDriver.groovy:221)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
Caused by: java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:417)
at java.lang.Integer.parseInt(Integer.java:499)
at java_lang_Integer$parseInt.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at com.sap.hadoop.windowing.cli.WindowingClient.readPort(WindowingClient.groovy:110)
at com.sap.hadoop.windowing.cli.WindowingClient.connect(WindowingClient.groovy:117)
... 21 more

[hbutani]

Looks like some issue with communication from the CLI to the WIndowingServer. So from the CLI a process is spawned to run the WindowingEngine. The first thing this does is send the CLI the port to commute on; this initial communication is failing.
Can you post the windowingCli.sh you are using.

[achaubal]

THISSERVICE=windowingCli
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

windowingCli () {
CLASS=com.sap.hadoop.windowing.WindowingHiveCliDriver

JAR=/tmp/windowing/com.sap.hadoop.windowing-.jar
#JAR=${HIVE_LIB}/com.sap.hadoop.windowing-.jar
exec $HADOOP jar $JAR $CLASS "$@"

}
windowingCli_help () {
windowingCli "--help"
}

[hbutani]

Ameet, are you also having this issue?

[achaubal]

sorry, didn't see the thread started by someone else. No, it works fine if I use metaservice. but not without it

[java8964]

The .sh file is the same as mine. How and what is metaservice? I didn't use that.
Yong

Date: Sun, 12 Feb 2012 18:28:59 -0800
From: reply@reply.github.com
To: java8964@hotmail.com
Subject: Re: [SQLWindowing] can not start the windowCli (#11)

sorry, didn't see the thread started by someone else. No, it works fine if I use metaservice. but not without it

---

Reply to this email directly or view it on GitHub:
#11 (comment)

[hbutani]

He is talking about the hive metastore server.
But your stacktrace indicates that you didn't get that far.

Can you check the following things

• Your .sh points to com.sap.hadoop.windowing-0.0.1-SNAPSHOT.jar
• your -w option points to com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar

[achaubal]

you need a metaservice to use this jar,

1. on another machine, start the metaservice, : hive --service metastore
2. on the first machine, change the hive-site.xml to reflect a connection to that metaservice,
   add these 2 sections,
   hive.metastore.local
   false
   controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM

hive.metastore.uris thrift://secondserver_where_you_started_metaservice:9083

[java8964]

It is fresh installed hive + hadoop. The only thing I changed is to use the mysql as the metastore in the hive-site.xml
~/hive/bin/ext> more windowCli.sh # Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements. See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License. You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.
THISSERVICE=windowingCliexport SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "
windowingCli () { CLASS=com.sap.hadoop.windowing.WindowingHiveCliDriver JAR=${HIVE_LIB}/com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar exec $HADOOP jar $JAR $CLASS "$@"}
windowingCli_help() { windowingCli "--help"}
~/hive/bin> ./hive --service windowingCli -w com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jarHive history file=/tmp/yzhang/hive_job_log_yzhang_201202141330_1899362727.txtException in thread "main" com.sap.hadoop.windowing.WindowingException: java.lang.NumberFormatException: null at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77) at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:54) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190) at com.sap.hadoop.windowing.cli.WindowingClient.connect(WindowingClient.groovy:126) at com.sap.hadoop.windowing.cli.WindowingClient.(WindowingClient.groovy:38) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77) at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:54) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:198) at com.sap.hadoop.windowing.WindowingHiveCliDriver.setupClient(WindowingHiveCliDriver.groovy:64) at com.sap.hadoop.windowing.WindowingHiveCliDriver$setupClient.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120) at com.sap.hadoop.windowing.WindowingHiveCliDriver.main(WindowingHiveCliDriver.groovy:221) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197)Caused by: java.lang.NumberFormatException: null at java.lang.Integer.parseInt(Integer.java:417) at java.lang.Integer.parseInt(Integer.java:499) at java_lang_Integer$parseInt.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) at com.sap.hadoop.windowing.cli.WindowingClient.readPort(WindowingClient.groovy:110) at com.sap.hadoop.windowing.cli.WindowingClient.connect(WindowingClient.groovy:117) ... 21 more

Date: Mon, 13 Feb 2012 17:37:30 -0800
From: reply@reply.github.com
To: java8964@hotmail.com
Subject: Re: [SQLWindowing] can not start the windowCli (#11)

He is talking about the hive metastore server.
But your stacktrace indicates that you didn't get that far.

Can you check the following things

• Your .sh points to com.sap.hadoop.windowing-0.0.1-SNAPSHOT.jar
• your -w option points to com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar

---

Reply to this email directly or view it on GitHub:
#11 (comment)

[achaubal]

it's not enough to use mySQL as a metastore. You need to use a metastore service.
this is done by starting a TCP daemon which listens on 9083 port.
it's done by : hive --service metastore
please check the above post with configs for hive-site.xml

[hbutani]

1. The script should be:
   THISSERVICE=windowingCliexport SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "
   windowingCli () { CLASS=com.sap.hadoop.windowing.WindowingHiveCliDriver JAR=${HIVE_LIB}/com.sap.hadoop.windowing-0.0.1-SNAPSHOT.jar exec $HADOOP jar $JAR $CLASS "$@"}
   windowingCli_help() { windowingCli "--help"}
   Don't use the jar-with-dependencies in the script.
2. On the command line give the full path to the 'com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar'
3. as Ameet says you have to start hive metastore service. I haven't tested windowingCli directly connecting to mysql.

Hey Yong, maybe a quick call might help with resolving these issues. Let me know if you want to do this & when. Send me a private message.

[java8964]

OK. I started again to try this software. It is a very good feature, and I really want to make it work in my environment.

I got your 2 jar files. Without dependence one is put in $HIVE_HOME/lib, With-dependence one is put in $MY_HOME/lib
Here is the error with as detail information I can give you:

yzhang@bsdhcp15755:~> ls -ls
0 lrwxrwxrwx 1 yzhang users 23 Mar 22 10:57 hadoop -> ./hadoop-0.20.2-cdh3u3/
4 drwxr-xr-x 17 yzhang users 4096 Mar 22 14:14 hadoop-0.20.2-cdh3u3
0 lrwxrwxrwx 1 yzhang users 20 Mar 22 10:57 hive -> ./hive-0.7.1-cdh3u3/
4 drwxr-xr-x 9 yzhang users 4096 Mar 22 10:56 hive-0.7.1-cdh3u3

yzhang@bsdhcp15755:> cd hive/bin/ext/
yzhang@bsdhcp15755:/hive/bin/ext> ls
cli.sh help.sh hiveserver.sh hwi.sh jar.sh lineage.sh metastore.sh rcfilecat.sh util windowCli.sh

yzhang@bsdhcp15755:~/hive/bin/ext> more windowCli.sh

THISSERVICE=windowingCli
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

windowingCli () {
CLASS=com.sap.hadoop.windowing.WindowingHiveCliDriver
JAR=${HIVE_LIB}/com.sap.hadoop.windowing-0.0.1-SNAPSHOT.jar
exec $HADOOP jar $JAR $CLASS "$@"
}

windowingCli_help () {
windowingCli "--help"
}

yzhang@bsdhcp15755:~/hive/bin/ext> ls ~/hive/lib/com.sap.hadoop.windowing-0.0.1-SNAPSHOT.jar
/home/yzhang/hive/lib/com.sap.hadoop.windowing-0.0.1-SNAPSHOT.jar

yzhang@bsdhcp15755:~/hive/bin/ext> ls ~/lib/com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar
/home/yzhang/lib/com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar

yzhang@bsdhcp15755:~> hive --service windowingCli -w /home/yzhang/lib/com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar
Exception in thread "main" java.lang.NoClassDefFoundError: groovy/lang/GroovyObject
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:295)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.util.RunJar.main(RunJar.java:190)
Caused by: java.lang.ClassNotFoundException: groovy.lang.GroovyObject
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 16 more

I can see the class groovy/lang/GroovyObject is in /home/yzhang/lib/com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar file. Any idea why?

[hbutani]

Yes, thanks for catching this.
You have to place the groovy jar in $HIVE_HOME/lib also.
Have updated the README.txt and uploaded the groovy jar

[hbutani]

Same as Issue 14