Skip to content

[SUPPORT] sparkSql create table sync metastore Failed #9575

@viverlxl

Description

@viverlxl

Tips before filing an issue

  • Have you gone through our FAQs?

  • Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

  • If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced

A clear and concise description of the problem.

To Reproduce

Steps to reproduce the behavior:

  1. SparkSql Client: xxx/spark-3.1.2-bin-hadoop3.2/bin/spark-sql --jars XX/hudi-spark-bundle_2.11-0.10.0.jar --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'
  2. create table hudi_mor_tbl (
    id int,
    name string,
    price double,
    ts bigint
    ) using hudi
    tblproperties (
    type = 'mor',
    primaryKey = 'id',
    preCombineField = 'ts'
    );
  3. WARN [main] config.DFSPropertiesConfiguration (DFSPropertiesConfiguration.java:getConfPathFromEnv(216)) - Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
    2023-08-30 16:12:37,279 WARN [main] config.DFSPropertiesConfiguration (DFSPropertiesConfiguration.java:addPropsFromFile(150)) - Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
    2023-08-30 16:12:40,660 WARN [main] command.CreateHoodieTableCommand (Logging.scala:logWarning(69)) - Failed to create catalog table in metastore: org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat

Expected behavior
hive can find table which create

A clear and concise description of what you expected to happen.

Environment Description

  • Hudi version : 0.10.0

  • Spark version : 3.1.2

  • Hive version : 2.1.1

  • Hadoop version : chd-6.3

  • Storage (HDFS/S3/GCS..) : hdfs

  • Running on Docker? (yes/no) : no

Additional context

Add any other context about the problem here.

Stacktrace

Add the stacktrace of the error.

related: #6128

had add jar to hive but no work

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions