-
Notifications
You must be signed in to change notification settings - Fork 2.8k
Closed
Description
I'm tring to write data using DataStream api, which the table was created by spark with catalog.type=hive.
My hdfs cluster was configed HA.
But when I run my code in IDEA, I got ERROR: Caused by: java.net.UnknownHostException: nsdev
this is my code:
Map hiveMetaInfo = new HashMap();
hiveMetaInfo.put("uri", "thrift://host-10-0-105-244:9083,thrift://host-10-0-105-248:9083");
hiveMetaInfo.put("warehouse", "hdfs://host-10-0-105-244:8020/warehouse/tablespace/managed/hive/");
Configuration conf = new Configuration(false);
conf.set("fs.defaultFS", "hdfs://nsdev");
conf.set("dfs.ha.namenodes.nsdev", "nn1,nn2");
conf.set("dfs.internal.nameservices", "nsdev");
conf.set("dfs.namenode.rpc-address.nsdev.nn1", "host-10-0-105-248:8020");
conf.set("dfs.namenode.rpc-address.nsdev.nn2", "host-10-0-105-244:8020");
TableLoader loader = TableLoader.fromCatalog(CatalogLoader.hive("hive_catalog",conf,hiveMetaInfo),TableIdentifier.of("test.radius_log"));
I also tried to config the environment variable :HADOOP_CONF_DIR
both dosn't work.
How to resolve it? thanks
Metadata
Metadata
Assignees
Labels
No labels