Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
sql query on Adam's flattened file -> Non-local session path expected to be non-null; #141
That's what I tried ....
scala> val sqlRDD2 = sqlContext.parquetFile("hdfs:///user/ec2-user/1kg/chr1.adam_flatten”)
so far – so nice – but when I want “select “ something ….
scala> val countResult = sqlContext.sql("SELECT COUNT(*) FROM sqlRDD2)").collect()
any idea ? Found some Internet links with "Non-local session path expected to be non-null;" but unfortunately no answer ...
I'm not sure what the error is—I haven't seen that error myself—but, does the error reproduce if you use Spark SQL without flattening the Parquet file? You need to flatten the Parquet file if you are using Hive or Impala, but Spark SQL knows how to process nested schemas.
thanks for looking into this !
yes - same Problem ..
scala> val sqlRDD = sqlContext.parquetFile("hdfs:///user/ec2-user/1kg/chr1.adam")
); la> val countResult = sqlContext.sql("SELECT COUNT(*) FROM sqlRDD)").collect(
Hi, maybe that the Eggo SQL issue is related to Hive ?!
when I do ...
scala> val sqlRDD2 = sqlContext.parquetFile("hdfs:///user/ec2-user/1kg/chr1.adam_flatten")
this fails with an error, too:
15/11/18 01:31:06 INFO ObjectStore: ObjectStore, initialize called
Google-ing "Error creating transactional connection factory" points to misconfigured "hive".
but Hive itself comes up on the master node:
[ec2-user@ip-10-1-1-239 ~]$ hive
Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.4.8-1.cdh5.4.8.p0.4/jars/hive-common-1.1.0-cdh5.4.8.jar!/hive-log4j.properties
Maybe someone can send me some SQL commands, which are working on his/her Eggo Installation ?!
My project does highly depend on SQL.