Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SUPPORT]Compile Error of hudi-0.13.0 "org.apache.hadoop.hive.common.type can not be converted to java.sql.Date" #8216

Closed
gamblewin opened this issue Mar 17, 2023 · 8 comments
Closed

[SUPPORT]Compile Error of hudi-0.13.0 "org.apache.hadoop.hive.common.type can not be converted to java.sql.Date" #8216

gamblewin opened this issue Mar 17, 2023 · 8 comments
Labels
priority:major degraded perf; unable to move forward; potential bugs project-build

Comments

@gamblewin
Copy link

When compiling source code of hudi-0.13.0, report compile error:
"HiveAvroSerializer.java[302, 93]: org.apache.hadoop.hive.common.type can not be converted to java.sql.Date"

  • Hudi version :0.13.0
  • Hive version :3.1.3
  • Hadoop version :3.0.0
  • Running on Docker? (yes/no) :no
@lokeshj1703
Copy link
Contributor

@gamblewin Can you share the stacktrace of the error?
Also share the maven version, java version and the command used to compile. Please also mention the branch you are building against.

@xicm
Copy link
Contributor

xicm commented Mar 21, 2023

you can try #7173

@gamblewin
Copy link
Author

@gamblewin Can you share the stacktrace of the error?
Also share the maven version, java version and the command used to compile. Please also mention the branch you are building against.

@lokeshj1703 Thx for replying.

  • Maven Version: 3.9.0

  • Java Version: 1.8.0_331

  • Hudi Branch: 0.13.0

  • Change I did to pom.xml
    <hadoop.version>3.0.0</hadoop.version>
    <hive.version>3.1.3</hive.version>

  • Compile Comand: mvn clean package -DskipTests -Dflink1.13 -Dscala-2.12 -Dhadoop.version=3.0.0 -Pflink-bundle-shade-hive3

  • Stacktrace
    [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project hudi-hadoop-mr: Compilation failure: Compilation failure:
    [ERROR] /Users/gamblewin/bigdata/hudi-0.13.0-test/hudi-0.13.0/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveAvroSerializer.java:[302,93] 不兼容的类型: org.apache.hadoop.hive.common.type.Date无法转换为java.sql.Date
    [ERROR] /Users/gamblewin/bigdata/hudi-0.13.0-test/hudi-0.13.0/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveAvroSerializer.java:[305,72] 不兼容的类型: org.apache.hadoop.hive.common.type.Timestamp无法转换为java.sql.Timestamp
    [ERROR] /Users/gamblewin/bigdata/hudi-0.13.0-test/hudi-0.13.0/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HiveAvroSerializer.java:[309,98] 不兼容的类型: org.apache.hadoop.hive.common.type.Date无法转换为java.sql.Date
    [ERROR] -> [Help 1]
    org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project hudi-hadoop-mr: Compilation failure
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:347)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:260)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:172)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:100)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:821)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:270)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
    Caused by: org.apache.maven.plugin.compiler.CompilationFailureException: Compilation failure
    at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute (AbstractCompilerMojo.java:1215)
    at org.apache.maven.plugin.compiler.CompilerMojo.execute (CompilerMojo.java:188)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:342)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:260)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:172)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:100)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:821)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:270)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
    [ERROR]
    [ERROR] Re-run Maven using the -X switch to enable full debug logging.
    [ERROR]
    [ERROR] For more information about the errors and possible solutions, please read the following articles:
    [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
    [ERROR]
    [ERROR] After correcting the problems, you can resume the build with the command
    [ERROR] mvn -rf :hudi-hadoop-mr

  • Update
    If I compile with command mvn clean package -DskipTests, everything works fine.

@codope codope added priority:major degraded perf; unable to move forward; potential bugs project-build labels Mar 22, 2023
@ad1happy2go
Copy link
Collaborator

@gamblewin I am able to reproduce the error . Any specific reason why you updating hive version to 3.1.3.
When I tried to set hive version to 3.0.0, the hudi-hadoop-mr build got succeeded.

@gamblewin
Copy link
Author

you can try #7173

thx, this solved my problem. Btw, line 312 also needs to be changed.

@gamblewin
Copy link
Author

@gamblewin I am able to reproduce the error . Any specific reason why you updating hive version to 3.1.3.
When I tried to set hive version to 3.0.0, the hudi-hadoop-mr build got succeeded.

I've solved this problem according to #7173. Actually, it's because someone else has already installed hive 3.1.3 in the environment.

@IntronsYanh
Copy link

you can try #7173

thx, this solved my problem. Btw, line 312 also needs to be changed.

Hi gamblewin, may I have your modified for line 312?

@gamblewin
Copy link
Author

you can try #7173

thx, this solved my problem. Btw, line 312 also needs to be changed.

Hi gamblewin, may I have your modified for line 312?

case INT:
        if (schema.getLogicalType() != null && schema.getLogicalType().getName().equals("date")) {
          //  - return DateWritable.dateToDays(new WritableDateObjectInspector().getPrimitiveJavaObject(structFieldData));
          return new DateWritable((DateWritable)structFieldData).getDays();
        }

but I don't know if I did the right change, cuz after this alteration, compilation succeeded. But when I want to insert data into a hudi cdc COW table, it will report mistake.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority:major degraded perf; unable to move forward; potential bugs project-build
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@lokeshj1703 @IntronsYanh @codope @xicm @gamblewin @ad1happy2go