ERROR ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. #150
Comments
|
Hi @charles138, " |
|
I used the latter method and it's working now. Thank you. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I tried your spark-shell example in "Unable to save data at HBase #69 (#69)" and I got an error: "ERROR ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper." during the save.
--==============================================================
-- My VM Info:
--==============================================================
Hortonwork Sandbox: HDP_2.6_vmware_19_04_2017_20_25_43_hdp_ambari_2_5_0_5_1
Spark Version: 2.1.1
Scala Version: 2.11.8
HBase Version: 1.1.2.2
--==============================================================
-- I have sucessfully completed the following in my sandbox:
--==============================================================
a) completed the lab: "Spark HBase - A DataFrame Based Connector" (https://hortonworks.com/blog/spark-hbase-dataframe-based-hbase-connector/)
b) downloaded and compiled https://github.com/hortonworks-spark/shc
c) ran ./bin/spark-submit --verbose --class org.apache.spark.sql.execution.datasources.hbase.examples.HBaseSource --master yarn-cluster --packages com.hortonworks:shc-core:1.1.1-2.1-s_2.11 --repositories http://repo.hortonworks.com/content/groups/public/ --files /usr/hdp/current/hbase-client/conf/hbase-site.xml shc-examples-1.1.1-2.1-s_2.11-SNAPSHOT.jar
d) checked tables shcExampleTable and shcExampleTable1 in HBase and no errors
When I tried to run your spark-shell example, I got an error during save to HBase. Any insights?
Thank you in advance for your reply.
--==============================================================
-- the following is the capture from my screen:
--==============================================================
[root@sandbox ~]# export SPARK_HOME=/usr/hdp/current/spark2-client
[root@sandbox ~]# export SPARK_MAJOR_VERSION=2
[root@sandbox ~]# cd /usr/hdp/current/spark2-client
[root@sandbox spark2-client]#
[root@sandbox spark2-client]# netstat -lnap | grep 4040
tcp 0 0 0.0.0.0:4040 0.0.0.0:* LISTEN 2967/java
[root@sandbox spark2-client]# kill -9 2967
[root@sandbox spark2-client]#
[root@sandbox spark2-client]# netstat -lnap | grep 4040
[root@sandbox spark2-client]# spark-shell --master yarn-client --packages com.hortonworks:shc-core:1.1.1-2.1-s_2.11 --num-executors 4 --driver-memory 512m --executor-memory 512m --executor-cores 1 --repositories http://nexus-private.hortonworks.com/nexus/content/repositories/IN-QA/
SPARK_MAJOR_VERSION is set to 2, using Spark2
Warning: Master yarn-client is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
http://nexus-private.hortonworks.com/nexus/content/repositories/IN-QA/ added as a remote repository with the name: repo-1
:: loading settings :: url = jar:file:/usr/hdp/2.6.0.3-8/spark2/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.hortonworks#shc-core added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found com.hortonworks#shc-core;1.1.1-2.1-s_2.11 in repo-1
found org.apache.hbase#hbase-server;1.1.2 in central
found org.apache.hbase#hbase-protocol;1.1.2 in central
found org.apache.hbase#hbase-annotations;1.1.2 in central
found com.github.stephenc.findbugs#findbugs-annotations;1.3.9-1 in central
found log4j#log4j;1.2.17 in central
found junit#junit;4.11 in central
found org.hamcrest#hamcrest-core;1.3 in central
found com.google.protobuf#protobuf-java;2.5.0 in central
found org.apache.hbase#hbase-procedure;1.1.2 in central
found com.google.guava#guava;12.0.1 in central
found com.google.code.findbugs#jsr305;1.3.9 in local-m2-cache
found org.apache.hbase#hbase-client;1.1.2 in central
found commons-codec#commons-codec;1.9 in central
found commons-io#commons-io;2.4 in central
found commons-lang#commons-lang;2.6 in central
found io.netty#netty-all;4.0.23.Final in central
found org.apache.zookeeper#zookeeper;3.4.6 in central
found org.slf4j#slf4j-api;1.7.7 in central
found org.slf4j#slf4j-log4j12;1.6.1 in central
found org.apache.htrace#htrace-core;3.1.0-incubating in central
found org.codehaus.jackson#jackson-mapper-asl;1.9.13 in central
found org.codehaus.jackson#jackson-core-asl;1.9.13 in central
found org.jruby.jcodings#jcodings;1.0.8 in central
found org.jruby.joni#joni;2.1.2 in central
found commons-httpclient#commons-httpclient;3.1 in central
found commons-collections#commons-collections;3.2.1 in central
found com.yammer.metrics#metrics-core;2.2.0 in central
found com.sun.jersey#jersey-core;1.9 in central
found com.sun.jersey#jersey-server;1.9 in central
found commons-cli#commons-cli;1.2 in local-m2-cache
found org.apache.commons#commons-math;2.2 in central
found org.mortbay.jetty#jetty;6.1.26 in local-m2-cache
found org.mortbay.jetty#jetty-util;6.1.26 in local-m2-cache
found org.mortbay.jetty#jetty-sslengine;6.1.26 in central
found org.mortbay.jetty#jsp-2.1;6.1.14 in central
found org.mortbay.jetty#jsp-api-2.1;6.1.14 in central
found org.mortbay.jetty#servlet-api-2.5;6.1.14 in central
found org.codehaus.jackson#jackson-jaxrs;1.9.13 in central
found tomcat#jasper-compiler;5.5.23 in central
found org.jamon#jamon-runtime;2.3.1 in central
found com.lmax#disruptor;3.3.0 in central
found org.apache.hbase#hbase-prefix-tree;1.1.2 in central
found org.mortbay.jetty#servlet-api;2.5-20081211 in local-m2-cache
found tomcat#jasper-runtime;5.5.23 in central
found commons-el#commons-el;1.0 in central
found org.apache.hbase#hbase-common;1.1.2 in central
found org.apache.phoenix#phoenix-core;4.9.0-HBase-1.1 in repo-1
found org.apache.tephra#tephra-api;0.9.0-incubating in repo-1
found org.apache.tephra#tephra-hbase-compat-1.1;0.9.0-incubating in repo-1
found org.apache.tephra#tephra-core;0.9.0-incubating in repo-1
found com.google.code.gson#gson;2.2.4 in repo-1
found com.google.guava#guava;13.0.1 in repo-1
found com.google.inject#guice;3.0 in repo-1
found javax.inject#javax.inject;1 in repo-1
found aopalliance#aopalliance;1.0 in repo-1
found org.sonatype.sisu.inject#cglib;2.2.1-v20090111 in repo-1
found asm#asm;3.1 in repo-1
found com.google.inject.extensions#guice-assistedinject;3.0 in repo-1
found ch.qos.logback#logback-classic;1.0.9 in repo-1
found ch.qos.logback#logback-core;1.0.9 in repo-1
found org.apache.thrift#libthrift;0.9.0 in repo-1
found org.apache.httpcomponents#httpcore;4.1.3 in repo-1
found it.unimi.dsi#fastutil;6.5.6 in repo-1
found org.apache.twill#twill-common;0.6.0-incubating in repo-1
found com.google.code.findbugs#jsr305;2.0.1 in repo-1
found org.apache.twill#twill-core;0.6.0-incubating in repo-1
found org.apache.twill#twill-api;0.6.0-incubating in repo-1
found org.apache.twill#twill-discovery-api;0.6.0-incubating in repo-1
found org.apache.twill#twill-zookeeper;0.6.0-incubating in repo-1
found org.apache.twill#twill-discovery-core;0.6.0-incubating in repo-1
found org.ow2.asm#asm-all;5.0.2 in repo-1
found io.dropwizard.metrics#metrics-core;3.1.0 in repo-1
found org.antlr#antlr-runtime;3.5.2 in repo-1
found jline#jline;2.11 in repo-1
found sqlline#sqlline;1.2.0 in repo-1
found joda-time#joda-time;1.6 in repo-1
found com.github.stephenc.jcip#jcip-annotations;1.0-1 in repo-1
found junit#junit;4.12 in repo-1
found org.apache.httpcomponents#httpclient;4.0.1 in repo-1
found commons-logging#commons-logging;1.2 in repo-1
found org.iq80.snappy#snappy;0.3 in repo-1
found commons-collections#commons-collections;3.2.2 in repo-1
found org.apache.commons#commons-csv;1.0 in repo-1
found org.apache.hbase#hbase-annotations;1.1.3 in central
found org.apache.hbase#hbase-protocol;1.1.3 in central
found org.apache.hadoop#hadoop-common;2.7.1 in repo-1
found org.apache.hadoop#hadoop-annotations;2.7.1 in repo-1
found org.apache.commons#commons-math3;3.1.1 in repo-1
found xmlenc#xmlenc;0.52 in repo-1
found commons-net#commons-net;3.1 in repo-1
found javax.servlet#servlet-api;2.5 in repo-1
found com.sun.jersey#jersey-json;1.9 in repo-1
found org.codehaus.jettison#jettison;1.1 in repo-1
found com.sun.xml.bind#jaxb-impl;2.2.3-1 in repo-1
found javax.xml.bind#jaxb-api;2.2.2 in repo-1
found javax.xml.stream#stax-api;1.0-2 in repo-1
found javax.activation#activation;1.1 in repo-1
found org.codehaus.jackson#jackson-xc;1.9.2 in repo-1
found net.java.dev.jets3t#jets3t;0.9.0 in repo-1
found org.apache.httpcomponents#httpcore;4.2.5 in repo-1
found com.jamesmurty.utils#java-xmlbuilder;0.4 in repo-1
found commons-configuration#commons-configuration;1.6 in repo-1
found commons-digester#commons-digester;1.8 in repo-1
found commons-beanutils#commons-beanutils;1.7.0 in repo-1
found commons-beanutils#commons-beanutils-core;1.8.0 in repo-1
found org.apache.hadoop#hadoop-auth;2.7.1 in repo-1
found org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15 in repo-1
found org.apache.directory.server#apacheds-i18n;2.0.0-M15 in repo-1
found org.apache.directory.api#api-asn1-api;1.0.0-M20 in repo-1
found org.apache.directory.api#api-util;1.0.0-M20 in repo-1
found org.apache.curator#curator-framework;2.7.1 in repo-1
found org.apache.curator#curator-client;2.7.1 in repo-1
found com.jcraft#jsch;0.1.42 in repo-1
found org.apache.curator#curator-recipes;2.7.1 in repo-1
found org.apache.commons#commons-compress;1.4.1 in central
found org.tukaani#xz;1.0 in central
found org.apache.hadoop#hadoop-mapreduce-client-core;2.7.1 in repo-1
found org.apache.hadoop#hadoop-yarn-common;2.7.1 in repo-1
found org.apache.hadoop#hadoop-yarn-api;2.7.1 in repo-1
found com.sun.jersey#jersey-client;1.9 in repo-1
found com.google.inject.extensions#guice-servlet;3.0 in repo-1
found com.sun.jersey.contribs#jersey-guice;1.9 in repo-1
found org.slf4j#slf4j-log4j12;1.7.10 in repo-1
found io.netty#netty;3.6.2.Final in repo-1
found javax.servlet.jsp#jsp-api;2.1 in repo-1
found org.apache.avro#avro;1.7.6 in central
found com.thoughtworks.paranamer#paranamer;2.3 in central
found org.xerial.snappy#snappy-java;1.0.5 in central
:: resolution report :: resolve 10884ms :: artifacts dl 109ms
:: modules in use:
aopalliance#aopalliance;1.0 from repo-1 in [default]
asm#asm;3.1 from repo-1 in [default]
ch.qos.logback#logback-classic;1.0.9 from repo-1 in [default]
ch.qos.logback#logback-core;1.0.9 from repo-1 in [default]
com.github.stephenc.findbugs#findbugs-annotations;1.3.9-1 from central in [default]
com.github.stephenc.jcip#jcip-annotations;1.0-1 from repo-1 in [default]
com.google.code.findbugs#jsr305;2.0.1 from repo-1 in [default]
com.google.code.gson#gson;2.2.4 from repo-1 in [default]
com.google.guava#guava;13.0.1 from repo-1 in [default]
com.google.inject#guice;3.0 from repo-1 in [default]
com.google.inject.extensions#guice-assistedinject;3.0 from repo-1 in [default]
com.google.inject.extensions#guice-servlet;3.0 from repo-1 in [default]
com.google.protobuf#protobuf-java;2.5.0 from central in [default]
com.hortonworks#shc-core;1.1.1-2.1-s_2.11 from repo-1 in [default]
com.jamesmurty.utils#java-xmlbuilder;0.4 from repo-1 in [default]
com.jcraft#jsch;0.1.42 from repo-1 in [default]
com.lmax#disruptor;3.3.0 from central in [default]
com.sun.jersey#jersey-client;1.9 from repo-1 in [default]
com.sun.jersey#jersey-core;1.9 from central in [default]
com.sun.jersey#jersey-json;1.9 from repo-1 in [default]
com.sun.jersey#jersey-server;1.9 from central in [default]
com.sun.jersey.contribs#jersey-guice;1.9 from repo-1 in [default]
com.sun.xml.bind#jaxb-impl;2.2.3-1 from repo-1 in [default]
com.thoughtworks.paranamer#paranamer;2.3 from central in [default]
com.yammer.metrics#metrics-core;2.2.0 from central in [default]
commons-beanutils#commons-beanutils;1.7.0 from repo-1 in [default]
commons-beanutils#commons-beanutils-core;1.8.0 from repo-1 in [default]
commons-cli#commons-cli;1.2 from local-m2-cache in [default]
commons-codec#commons-codec;1.9 from central in [default]
commons-collections#commons-collections;3.2.2 from repo-1 in [default]
commons-configuration#commons-configuration;1.6 from repo-1 in [default]
commons-digester#commons-digester;1.8 from repo-1 in [default]
commons-el#commons-el;1.0 from central in [default]
commons-httpclient#commons-httpclient;3.1 from central in [default]
commons-io#commons-io;2.4 from central in [default]
commons-lang#commons-lang;2.6 from central in [default]
commons-logging#commons-logging;1.2 from repo-1 in [default]
commons-net#commons-net;3.1 from repo-1 in [default]
io.dropwizard.metrics#metrics-core;3.1.0 from repo-1 in [default]
io.netty#netty;3.6.2.Final from repo-1 in [default]
io.netty#netty-all;4.0.23.Final from central in [default]
it.unimi.dsi#fastutil;6.5.6 from repo-1 in [default]
javax.activation#activation;1.1 from repo-1 in [default]
javax.inject#javax.inject;1 from repo-1 in [default]
javax.servlet#servlet-api;2.5 from repo-1 in [default]
javax.servlet.jsp#jsp-api;2.1 from repo-1 in [default]
javax.xml.bind#jaxb-api;2.2.2 from repo-1 in [default]
javax.xml.stream#stax-api;1.0-2 from repo-1 in [default]
jline#jline;2.11 from repo-1 in [default]
joda-time#joda-time;1.6 from repo-1 in [default]
junit#junit;4.12 from repo-1 in [default]
log4j#log4j;1.2.17 from central in [default]
net.java.dev.jets3t#jets3t;0.9.0 from repo-1 in [default]
org.antlr#antlr-runtime;3.5.2 from repo-1 in [default]
org.apache.avro#avro;1.7.6 from central in [default]
org.apache.commons#commons-compress;1.4.1 from central in [default]
org.apache.commons#commons-csv;1.0 from repo-1 in [default]
org.apache.commons#commons-math;2.2 from central in [default]
org.apache.commons#commons-math3;3.1.1 from repo-1 in [default]
org.apache.curator#curator-client;2.7.1 from repo-1 in [default]
org.apache.curator#curator-framework;2.7.1 from repo-1 in [default]
org.apache.curator#curator-recipes;2.7.1 from repo-1 in [default]
org.apache.directory.api#api-asn1-api;1.0.0-M20 from repo-1 in [default]
org.apache.directory.api#api-util;1.0.0-M20 from repo-1 in [default]
org.apache.directory.server#apacheds-i18n;2.0.0-M15 from repo-1 in [default]
org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15 from repo-1 in [default]
org.apache.hadoop#hadoop-annotations;2.7.1 from repo-1 in [default]
org.apache.hadoop#hadoop-auth;2.7.1 from repo-1 in [default]
org.apache.hadoop#hadoop-common;2.7.1 from repo-1 in [default]
org.apache.hadoop#hadoop-mapreduce-client-core;2.7.1 from repo-1 in [default]
org.apache.hadoop#hadoop-yarn-api;2.7.1 from repo-1 in [default]
org.apache.hadoop#hadoop-yarn-common;2.7.1 from repo-1 in [default]
org.apache.hbase#hbase-annotations;1.1.3 from central in [default]
org.apache.hbase#hbase-client;1.1.2 from central in [default]
org.apache.hbase#hbase-common;1.1.2 from central in [default]
org.apache.hbase#hbase-prefix-tree;1.1.2 from central in [default]
org.apache.hbase#hbase-procedure;1.1.2 from central in [default]
org.apache.hbase#hbase-protocol;1.1.3 from central in [default]
org.apache.hbase#hbase-server;1.1.2 from central in [default]
org.apache.htrace#htrace-core;3.1.0-incubating from central in [default]
org.apache.httpcomponents#httpclient;4.0.1 from repo-1 in [default]
org.apache.httpcomponents#httpcore;4.2.5 from repo-1 in [default]
org.apache.phoenix#phoenix-core;4.9.0-HBase-1.1 from repo-1 in [default]
org.apache.tephra#tephra-api;0.9.0-incubating from repo-1 in [default]
org.apache.tephra#tephra-core;0.9.0-incubating from repo-1 in [default]
org.apache.tephra#tephra-hbase-compat-1.1;0.9.0-incubating from repo-1 in [default]
org.apache.thrift#libthrift;0.9.0 from repo-1 in [default]
org.apache.twill#twill-api;0.6.0-incubating from repo-1 in [default]
org.apache.twill#twill-common;0.6.0-incubating from repo-1 in [default]
org.apache.twill#twill-core;0.6.0-incubating from repo-1 in [default]
org.apache.twill#twill-discovery-api;0.6.0-incubating from repo-1 in [default]
org.apache.twill#twill-discovery-core;0.6.0-incubating from repo-1 in [default]
org.apache.twill#twill-zookeeper;0.6.0-incubating from repo-1 in [default]
org.apache.zookeeper#zookeeper;3.4.6 from central in [default]
org.codehaus.jackson#jackson-core-asl;1.9.13 from central in [default]
org.codehaus.jackson#jackson-jaxrs;1.9.13 from central in [default]
org.codehaus.jackson#jackson-mapper-asl;1.9.13 from central in [default]
org.codehaus.jackson#jackson-xc;1.9.2 from repo-1 in [default]
org.codehaus.jettison#jettison;1.1 from repo-1 in [default]
org.hamcrest#hamcrest-core;1.3 from central in [default]
org.iq80.snappy#snappy;0.3 from repo-1 in [default]
org.jamon#jamon-runtime;2.3.1 from central in [default]
org.jruby.jcodings#jcodings;1.0.8 from central in [default]
org.jruby.joni#joni;2.1.2 from central in [default]
org.mortbay.jetty#jetty;6.1.26 from local-m2-cache in [default]
org.mortbay.jetty#jetty-sslengine;6.1.26 from central in [default]
org.mortbay.jetty#jetty-util;6.1.26 from local-m2-cache in [default]
org.mortbay.jetty#jsp-2.1;6.1.14 from central in [default]
org.mortbay.jetty#jsp-api-2.1;6.1.14 from central in [default]
org.mortbay.jetty#servlet-api;2.5-20081211 from local-m2-cache in [default]
org.mortbay.jetty#servlet-api-2.5;6.1.14 from central in [default]
org.ow2.asm#asm-all;5.0.2 from repo-1 in [default]
org.slf4j#slf4j-api;1.7.7 from central in [default]
org.slf4j#slf4j-log4j12;1.7.10 from repo-1 in [default]
org.sonatype.sisu.inject#cglib;2.2.1-v20090111 from repo-1 in [default]
org.tukaani#xz;1.0 from central in [default]
org.xerial.snappy#snappy-java;1.0.5 from central in [default]
sqlline#sqlline;1.2.0 from repo-1 in [default]
tomcat#jasper-compiler;5.5.23 from central in [default]
tomcat#jasper-runtime;5.5.23 from central in [default]
xmlenc#xmlenc;0.52 from repo-1 in [default]
:: evicted modules:
org.apache.hbase#hbase-protocol;1.1.2 by [org.apache.hbase#hbase-protocol;1.1.3] in [default]
org.apache.hbase#hbase-annotations;1.1.2 by [org.apache.hbase#hbase-annotations;1.1.3] in [default]
junit#junit;4.11 by [junit#junit;4.12] in [default]
com.google.guava#guava;12.0.1 by [com.google.guava#guava;13.0.1] in [default]
com.google.code.findbugs#jsr305;1.3.9 by [com.google.code.findbugs#jsr305;2.0.1] in [default]
org.slf4j#slf4j-log4j12;1.6.1 by [org.slf4j#slf4j-log4j12;1.7.10] in [default]
commons-collections#commons-collections;3.2.1 by [commons-collections#commons-collections;3.2.2] in [default]
org.slf4j#slf4j-api;1.6.4 by [org.slf4j#slf4j-api;1.7.7] in [default]
commons-lang#commons-lang;2.5 by [commons-lang#commons-lang;2.6] in [default]
org.apache.httpcomponents#httpclient;4.1.3 by [org.apache.httpcomponents#httpclient;4.0.1] in [default]
org.apache.httpcomponents#httpcore;4.1.3 by [org.apache.httpcomponents#httpcore;4.2.5] in [default]
org.apache.zookeeper#zookeeper;3.4.5 by [org.apache.zookeeper#zookeeper;3.4.6] in [default]
org.codehaus.jackson#jackson-core-asl;1.9.2 by [org.codehaus.jackson#jackson-core-asl;1.9.13] in [default]
org.codehaus.jackson#jackson-mapper-asl;1.9.2 by [org.codehaus.jackson#jackson-mapper-asl;1.9.13] in [default]
org.apache.httpcomponents#httpcore;4.0.1 by [org.apache.httpcomponents#httpcore;4.2.5] in [default]
commons-codec#commons-codec;1.7 by [commons-codec#commons-codec;1.9] in [default]
org.codehaus.jackson#jackson-jaxrs;1.9.2 by [org.codehaus.jackson#jackson-jaxrs;1.9.13] in [default]
org.apache.httpcomponents#httpclient;4.2.5 by [org.apache.httpcomponents#httpclient;4.0.1] in [default]
org.apache.avro#avro;1.7.4 by [org.apache.avro#avro;1.7.6] in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 140 | 9 | 9 | 19 || 121 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
0 artifacts copied, 121 already retrieved (0kB/43ms)
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://172.17.0.2:4040
Spark context available as 'sc' (master = yarn, app id = application_1498343037765_0001).
Spark session available as 'spark'.
Welcome to
____ __
/ / ___ / /
\ / _ / _ `/ __/ '/
// .__/_,// //_\ version 2.1.0.2.6.0.3-8
//
Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_121)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val sc = spark.sparkContext
sc: org.apache.spark.SparkContext = org.apache.spark.SparkContext@5c94d4b8
scala> val sqlContext = spark.sqlContext
sqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@102dff25
scala> import sqlContext.implicits._
import sqlContext.implicits._
scala> import spark.sql
import spark.sql
scala>
scala> val cat = s"""{
| "table":{"namespace":"default", "name":"shcExampleTable", "tableCoder":"PrimitiveType"},
| "rowkey":"key",
| "columns":{
| "key":{"cf":"rowkey", "col":"key", "type":"int"},
| "col1":{"cf":"cf1", "col":"col1", "type":"boolean"},
| "col2":{"cf":"cf2", "col":"col2", "type":"double"},
| "col3":{"cf":"cf3", "col":"col3", "type":"float"}
| }
| }""".stripMargin
cat: String =
{
"table":{"namespace":"default", "name":"shcExampleTable", "tableCoder":"PrimitiveType"},
"rowkey":"key",
"columns":{
"key":{"cf":"rowkey", "col":"key", "type":"int"},
"col1":{"cf":"cf1", "col":"col1", "type":"boolean"},
"col2":{"cf":"cf2", "col":"col2", "type":"double"},
"col3":{"cf":"cf3", "col":"col3", "type":"float"}
}
}
scala>
scala> import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.sql.{DataFrame, SparkSession}
scala> import org.apache.spark.sql.execution.datasources.hbase
import org.apache.spark.sql.execution.datasources.hbase
scala>
scala> def withCatalog(cat: String): DataFrame = {
| sqlContext
| .read
| .options(Map(org.apache.spark.sql.execution.datasources.hbase.HBaseTableCatalog.tableCatalog->cat))
| .format("org.apache.spark.sql.execution.datasources.hbase")
| .load()
| }
withCatalog: (cat: String)org.apache.spark.sql.DataFrame
scala>
scala> case class LRRecord(
| key: Int,
| col1: Boolean,
| col2: Double,
| col3: Float)
defined class LRRecord
scala>
scala> val data = Seq(LRRecord(1,false, 1.0, 1.0.toFloat), LRRecord(2,true, 1.0, 1.0.toFloat))
data: Seq[LRRecord] = List(LRRecord(1,false,1.0,1.0), LRRecord(2,true,1.0,1.0))
scala> sc.parallelize(data).toDF.write.options(Map(org.apache.spark.sql.execution.datasources.hbase.HBaseTableCatalog.tableCatalog -> cat, org.apache.spark.sql.execution.datasources.hbase.HBaseTableCatalog.newTable -> "5")).format("org.apache.spark.sql.execution.datasources.hbase").save()
17/06/24 22:34:19 ERROR ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
17/06/24 22:34:19 ERROR ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
17/06/24 22:34:20 ERROR ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
17/06/24 22:34:20 ERROR ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
17/06/24 22:34:21 ERROR ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
17/06/24 22:34:23 ERROR ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
17/06/24 22:34:27 ERROR ConnectionManager$HConnectionImplementation: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.
[root@sandbox spark2-client]#
The text was updated successfully, but these errors were encountered: