[HUDI-6220] Add HUDI code version to commit files and hoodie.properties. by prashantwason · Pull Request #8724 · apache/hudi

prashantwason · 2023-05-16T07:45:02Z

[HUDI-6220] Add HUDI code version to commit files and hoodie.properties.

Change Logs

Added a class HUDIVersion which can be used to get HUDI version
Added HUDI version and other engine specific information to HoodieCOmmitMetadata as extraMetadata
Added HUDI version to the file comment for hoodie.properties

Impact

More debugging information available

Risk level (write none, low medium or high below)

None

Documentation Update

None

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

danny0405

cc @bvaradar Can you help the review?

bvaradar · 2023-05-17T08:18:23Z

will review this tomorrow.

bvaradar

@prashantwason : Overall, looks good. Few questions.

bvaradar · 2023-05-17T23:28:30Z

...udi-client-common/src/main/java/org/apache/hudi/client/embedded/EmbeddedTimelineService.java

    } else {
      LOG.warn("Unable to find driver bind address from spark config");
-      this.hostAddr = NetworkUtils.getHostname();
+      this.hostAddr = NetworkUtils.getHostAddr();


What is the rational for this change ?

The NetworkUtils.getHostname function is returning the IP address. So renamed it to getHostAddr which is more appropriate.

Also, I added the getHostName function which returns the hostname

bvaradar · 2023-05-17T23:37:28Z

...nt/hudi-java-client/src/main/java/org/apache/hudi/client/common/HoodieJavaEngineContext.java

+  @Override
+  public Map<String, String> getInfo() {
+    final Map<String, String> info = new HashMap<>();
+    System.getProperties().stringPropertyNames().forEach(property -> info.put(property, System.getProperty(property)));


Instead of blindly copying all properties, we should do this selectively, as there may be sensitive information here.

yes. we should be judicious here

These files are within the dataset itself. The dataset would contain sensitive information too. So I dont see the issue.

Anyways, I will filter them.

bvaradar · 2023-05-17T23:45:16Z

hudi-common/src/main/java/org/apache/hudi/HoodieVersion.java

+    return parts;
+  }
+
+  public static void main(String[] args) {


Can you remove this

nsivabalan · 2023-05-23T00:02:05Z

...nt/hudi-java-client/src/main/java/org/apache/hudi/client/common/HoodieJavaEngineContext.java

+  @Override
+  public Map<String, String> getInfo() {
+    final Map<String, String> info = new HashMap<>();
+    System.getProperties().stringPropertyNames().forEach(property -> info.put(property, System.getProperty(property)));


yes. we should be judicious here

nsivabalan · 2023-05-23T00:04:14Z

.../hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java

+    info.put("spark.defaultParallelism", String.valueOf(javaSparkContext.defaultParallelism()));
+    info.put("spark.defaultMinPartitions", String.valueOf(javaSparkContext.defaultMinPartitions()));
+    info.put("spark.executor.instances", String.valueOf(javaSparkContext.getConf().get("spark.executor.instances")));
+    return info;


should we also let users configure any hoodie write configs to be added to the extra metadata. since we don't serialize them anywhere, sometimes might come in handy during investigations. by default we don't need to add any hoodie write configs. but if user configures them, we can add them as well

Simply easier to add the entire hoodie config (without schema).

nsivabalan · 2023-05-23T00:06:42Z

hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java

-    if (isValidChecksum(props)) {
+    final boolean isValidChecksum = isValidChecksum(props);
+    final String comment = String.format("Date=%s, host=%s, #properties=%d, hudi_version=%s",
+        Instant.now(), NetworkUtils.getHostname(), isValidChecksum ? props.size() : props.size() + 1, HoodieVersion.get());


if you modified NetworkUtils.getHostname() just for this purpose, lets keep the existing method and add a new one.

NetworkUtils.getHostname() does not return hostname - it returns IP address. So this is a fix.

bvaradar

LGTM

yihua · 2023-05-30T21:44:44Z

Hi @prashantwason could you check the build failure?

prashantwason · 2023-06-05T19:55:52Z

@hudi-bot run azure

prashantwason · 2023-06-21T20:24:39Z

@hudi-bot run azure

… in unit tests.

prashantwason · 2023-07-18T11:19:43Z

@hudi-bot run azure

hudi-bot · 2023-07-18T14:28:25Z

CI report:

e557752 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

bvaradar · 2023-10-04T17:27:54Z

@prashantwason : Can you fix the conflictx and test so that we can land this ?

danny0405 reviewed May 16, 2023

View reviewed changes

bvaradar self-assigned this May 17, 2023

bvaradar reviewed May 18, 2023

View reviewed changes

nsivabalan self-assigned this May 23, 2023

nsivabalan reviewed May 23, 2023

View reviewed changes

bvaradar approved these changes May 24, 2023

View reviewed changes

nsivabalan added the priority:critical Production degraded; pipelines stalled label Jun 5, 2023

prashantwason force-pushed the pw_hudi_version branch from 92d5c03 to 315d808 Compare June 6, 2023 06:32

codope force-pushed the pw_hudi_version branch from dd2b117 to 7b99dac Compare June 19, 2023 08:35

prashantwason added the release-0.14.0 label Jun 21, 2023

nsivabalan added priority:high Significant impact; potential bugs and removed priority:critical Production degraded; pipelines stalled labels Jun 21, 2023

prashantwason and others added 5 commits July 17, 2023 10:52

[HUDI-6220] Add HUDI code version to commit files and hoodie.properties.

ec16c22

Addressed review comments

c96988c

Fixed mockito version

8fe933c

Fix dependencies and cleanup

9a91e2f

Rebased on master

8525623

prashantwason force-pushed the pw_hudi_version branch from 7b99dac to 8525623 Compare July 17, 2023 22:03

Added defaults for the Spark conf as they might not have valid values…

e557752

… in unit tests.

codope removed the release-0.14.0 label Aug 4, 2023

github-actions bot added the size:L PR with lines of changes in (300, 1000] label Feb 26, 2024

hudi-bot mentioned this pull request Dec 9, 2025

Add HUDI code version to commit files and hoodie.properties #15958

Open

Conversation

prashantwason commented May 16, 2023

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Uh oh!

danny0405 left a comment

Choose a reason for hiding this comment

Uh oh!

bvaradar commented May 17, 2023

Uh oh!

bvaradar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bvaradar left a comment

Choose a reason for hiding this comment

Uh oh!

yihua commented May 30, 2023

Uh oh!

prashantwason commented Jun 5, 2023

Uh oh!

prashantwason commented Jun 21, 2023

Uh oh!

prashantwason commented Jul 18, 2023

Uh oh!

hudi-bot commented Jul 18, 2023

CI report:

Uh oh!

bvaradar commented Oct 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants