Skip to content

iceberg.flink.TableLoader from hive , can't resolve hdfs nameservice #2839

@lordk911

Description

@lordk911

I'm tring to write data using DataStream api, which the table was created by spark with catalog.type=hive.
My hdfs cluster was configed HA.
But when I run my code in IDEA, I got ERROR: Caused by: java.net.UnknownHostException: nsdev

this is my code:

Map hiveMetaInfo = new HashMap();
hiveMetaInfo.put("uri", "thrift://host-10-0-105-244:9083,thrift://host-10-0-105-248:9083");
hiveMetaInfo.put("warehouse", "hdfs://host-10-0-105-244:8020/warehouse/tablespace/managed/hive/");

Configuration conf = new Configuration(false);
conf.set("fs.defaultFS", "hdfs://nsdev");
conf.set("dfs.ha.namenodes.nsdev", "nn1,nn2");
conf.set("dfs.internal.nameservices", "nsdev");
conf.set("dfs.namenode.rpc-address.nsdev.nn1", "host-10-0-105-248:8020");
conf.set("dfs.namenode.rpc-address.nsdev.nn2", "host-10-0-105-244:8020");

TableLoader loader = TableLoader.fromCatalog(CatalogLoader.hive("hive_catalog",conf,hiveMetaInfo),TableIdentifier.of("test.radius_log"));

I also tried to config the environment variable :HADOOP_CONF_DIR

both dosn't work.
How to resolve it? thanks

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions