Skip to content

[HUDI-4383]Make hudi-flink-bundle module compile with the correct flink version#6084

Closed
xiaozhch5 wants to merge 1 commit intoapache:masterfrom
xiaozhch5:HUDI-4383
Closed

[HUDI-4383]Make hudi-flink-bundle module compile with the correct flink version#6084
xiaozhch5 wants to merge 1 commit intoapache:masterfrom
xiaozhch5:HUDI-4383

Conversation

@xiaozhch5
Copy link
Contributor

@xiaozhch5 xiaozhch5 commented Jul 12, 2022

Tips

What is the purpose of the pull request

(For example: This pull request adds quick-start document.)

Brief change log

Fix compile bug.

At present, when I compile hudi with the parameter of "-Pflink1.15", the flink version of hudi-flink-bundle is still 1.14.

the output is list below:

[root@hadoop3 hudi-master]# mvn clean package -DskipTests -Dhive.version=3.1.2 -Dscala.version=2.12.12 -Dscala.binary.version=2.12 -Pflink-bundle-shade-hive3 -Pflink1.15 -Dcheckstyle.skip=true -Dspark3.2 -Dscala-2.12
[INFO] Scanning for projects...
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for org.apache.hudi:hudi-common:jar:0.12.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.logging.log4j:log4j-core:jar -> version ${log4j2.version} vs ${log4j.test.version} @ org.apache.hudi:hudi:0.12.0-SNAPSHOT, /data/hudi-master/pom.xml, line 1175, column 19
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.logging.log4j:log4j-api:jar -> version ${log4j2.version} vs ${log4j.test.version} @ org.apache.hudi:hudi:0.12.0-SNAPSHOT, /data/hudi-master/pom.xml, line 1182, column 19
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for org.apache.hudi:hudi-cli:jar:0.12.0-SNAPSHOT
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.logging.log4j:log4j-core:jar -> duplicate declaration of version (?) @ line 216, column 17
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.logging.log4j:log4j-api:jar -> duplicate declaration of version (?) @ line 222, column 17
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for org.apache.hudi:hudi-spark-common_2.12:jar:0.12.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.hudi:hudi-spark-common_${scala.binary.version}:0.12.0-SNAPSHOT, /data/hudi-master/hudi-spark-datasource/hudi-spark-common/pom.xml, line 24, column 15
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for org.apache.hudi:hudi-spark_2.12:jar:0.12.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.hudi:hudi-spark_${scala.binary.version}:0.12.0-SNAPSHOT, /data/hudi-master/hudi-spark-datasource/hudi-spark/pom.xml, line 26, column 15
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for org.apache.hudi:hudi-utilities_2.12:jar:0.12.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.hudi:hudi-utilities_${scala.binary.version}:0.12.0-SNAPSHOT, /data/hudi-master/hudi-utilities/pom.xml, line 26, column 15
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.logging.log4j:log4j-core:jar -> duplicate declaration of version (?) @ org.apache.hudi:hudi-utilities_${scala.binary.version}:0.12.0-SNAPSHOT, /data/hudi-master/hudi-utilities/pom.xml, line 501, column 17
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for org.apache.hudi:hudi-hive-sync:jar:0.12.0-SNAPSHOT
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.logging.log4j:log4j-core:jar -> duplicate declaration of version (?) @ line 162, column 17
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for org.apache.hudi:hudi-spark3.2-bundle_2.12:jar:0.12.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.hudi:hudi-spark${sparkbundle.version}-bundle_${scala.binary.version}:0.12.0-SNAPSHOT, /data/hudi-master/packaging/hudi-spark-bundle/pom.xml, line 26, column 15
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for org.apache.hudi:hudi-utilities-bundle_2.12:jar:0.12.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.hudi:hudi-utilities-bundle_${scala.binary.version}:0.12.0-SNAPSHOT, /data/hudi-master/packaging/hudi-utilities-bundle/pom.xml, line 26, column 15
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for org.apache.hudi:hudi-utilities-slim-bundle_2.12:jar:0.12.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.hudi:hudi-utilities-slim-bundle_${scala.binary.version}:0.12.0-SNAPSHOT, /data/hudi-master/packaging/hudi-utilities-slim-bundle/pom.xml, line 26, column 15
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for org.apache.hudi:hudi-flink1.14-bundle:jar:0.12.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.hudi:hudi-flink${flink.bundle.version}-bundle:0.12.0-SNAPSHOT, /data/hudi-master/packaging/hudi-flink-bundle/pom.xml, line 28, column 15
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for org.apache.hudi:hudi-spark2_2.12:jar:0.12.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.hudi:hudi-spark2_${scala.binary.version}:0.12.0-SNAPSHOT, /data/hudi-master/hudi-spark-datasource/hudi-spark2/pom.xml, line 24, column 15
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for org.apache.hudi:hudi:pom:0.12.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.logging.log4j:log4j-core:jar -> version ${log4j2.version} vs ${log4j.test.version} @ line 1175, column 19
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.logging.log4j:log4j-api:jar -> version ${log4j2.version} vs ${log4j.test.version} @ line 1182, column 19
[WARNING] 
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING] 
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING] 
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Hudi                                                               [pom]
[INFO] hudi-common                                                        [jar]
[INFO] hudi-hadoop-mr                                                     [jar]
[INFO] hudi-sync-common                                                   [jar]
[INFO] hudi-hive-sync                                                     [jar]
[INFO] hudi-aws                                                           [jar]
[INFO] hudi-timeline-service                                              [jar]
[INFO] hudi-client                                                        [pom]
[INFO] hudi-client-common                                                 [jar]
[INFO] hudi-spark-client                                                  [jar]
[INFO] hudi-spark-datasource                                              [pom]
[INFO] hudi-spark-common_2.12                                             [jar]
[INFO] hudi-spark3-common                                                 [jar]
[INFO] hudi-spark3_2.12                                                   [jar]
[INFO] hudi-java-client                                                   [jar]
[INFO] hudi-spark_2.12                                                    [jar]
[INFO] hudi-utilities_2.12                                                [jar]
[INFO] hudi-utilities-bundle_2.12                                         [jar]
[INFO] hudi-cli                                                           [jar]
[INFO] hudi-flink-client                                                  [jar]
[INFO] hudi-gcp                                                           [jar]
[INFO] hudi-datahub-sync                                                  [jar]
[INFO] hudi-adb-sync                                                      [jar]
[INFO] hudi-sync                                                          [pom]
[INFO] hudi-hadoop-mr-bundle                                              [jar]
[INFO] hudi-datahub-sync-bundle                                           [jar]
[INFO] hudi-hive-sync-bundle                                              [jar]
[INFO] hudi-aws-bundle                                                    [jar]
[INFO] hudi-gcp-bundle                                                    [jar]
[INFO] hudi-spark3.2-bundle_2.12                                          [jar]
[INFO] hudi-presto-bundle                                                 [jar]
[INFO] hudi-utilities-slim-bundle_2.12                                    [jar]
[INFO] hudi-timeline-server-bundle                                        [jar]
[INFO] hudi-trino-bundle                                                  [jar]
[INFO] hudi-examples                                                      [pom]
[INFO] hudi-examples-common                                               [jar]
[INFO] hudi-examples-spark                                                [jar]
[INFO] hudi-flink-datasource                                              [pom]
[INFO] hudi-flink1.14.x                                                   [jar]
[INFO] hudi-flink                                                         [jar]
[INFO] hudi-examples-flink                                                [jar]
[INFO] hudi-examples-java                                                 [jar]
[INFO] hudi-flink1.13.x                                                   [jar]
[INFO] hudi-flink1.15.x                                                   [jar]
[INFO] hudi-kafka-connect                                                 [jar]
[INFO] hudi-flink1.14-bundle                                              [jar]
[INFO] hudi-kafka-connect-bundle                                          [jar]
[INFO] hudi-spark2_2.12                                                   [jar]
[INFO] hudi-spark2-common                                                 [jar]
[INFO] 
[INFO] ------------------------< org.apache.hudi:hudi >------------------------
[INFO] Building Hudi 0.12.0-SNAPSHOT                                     [1/49]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ hudi ---
[INFO] Deleting /data/hudi-master/target
[INFO] 
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-maven-version) @ hudi ---
[INFO] 
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-versions) @ hudi ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) @ hudi ---
[INFO] 
[INFO] --- maven-checkstyle-plugin:3.0.0:check (default) @ hudi ---
[INFO] 
[INFO] --- maven-source-plugin:2.2.1:jar-no-fork (attach-sources) @ hudi ---
[INFO] 
[INFO] --- maven-site-plugin:3.7.1:attach-descriptor (attach-descriptor) @ hudi ---
[INFO] No site descriptor found: nothing to attach.
......

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end.
  • Added HoodieClientWriteTest to verify the change.
  • Manually verified the change by running a job locally.

Committer checklist

  • Has a corresponding JIRA in PR title & commit

  • Commit message is descriptive of the change

  • CI is green

  • Necessary doc changes done or have another open PR

  • For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@hudi-bot
Copy link
Collaborator

hudi-bot commented Jul 12, 2022

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@xushiyan xushiyan added priority:medium Moderate impact; usability gaps engine:flink Flink integration labels Jul 12, 2022
@yihua yihua added the dependencies Dependency updates label Sep 13, 2022
@nsivabalan nsivabalan added the release-0.12.2 Patches targetted for 0.12.2 label Dec 6, 2022
codope
codope reviewed Dec 7, 2022
View reviewed changes
Copy link
Member

@codope codope left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.
@danny0405 Can you please review this patch?

@codope codope removed the release-0.12.2 Patches targetted for 0.12.2 label Dec 7, 2022
@bvaradar
Copy link
Contributor

bvaradar commented Feb 22, 2023

@danny0405 : Is this PR still valid to be landed ?

@bvaradar bvaradar self-assigned this Feb 22, 2023
@danny0405
Copy link
Contributor

danny0405 commented Feb 22, 2023

Not anymore, close it.

@danny0405 danny0405 closed this Feb 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@codope codope codope left review comments

Assignees

@bvaradar bvaradar

@danny0405 danny0405

Labels

dependencies Dependency updates engine:flink Flink integration priority:medium Moderate impact; usability gaps

Projects

Hudi Issue Support
Status: 🚧 Needs Repro
Hudi PR Support
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@xiaozhch5 @hudi-bot @bvaradar @danny0405 @codope @nsivabalan @yihua @xushiyan