Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HDFS-16464. Create only libhdfspp static libraries for Windows #4571

Merged
merged 4 commits into from
Jul 19, 2022
Merged

HDFS-16464. Create only libhdfspp static libraries for Windows #4571

merged 4 commits into from
Jul 19, 2022

Conversation

GauthamBanasandra
Copy link
Member

Description of PR

While building dynamic libraries (.dll) on Windows, there's a constraint that all the dependent libraries be linked dynamically. This poses an issue since Protobuf (which is an HDFS native client dependency) runs into build issues when linked dynamically. There are a few warning notes on the Protobuf repository's build instructions page as well.

Thus, to keep things simple, we can resort to do only static linking and thereby only produce statically linked libraries on Windows. In summary, we'll be providing only Hadoop .lib files initially. We can aim to produce Hadoop DLL on Windows eventually once we're able to resolve Protobuf's DLL issues.

How was this patch tested?

Hadoop Jenkins CI validation ensures that the Linux components aren't affected. Verified that the static library is produced as a result of successful build on my local Windows system.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@GauthamBanasandra
HDFS-16464. Create only libhdfspp static libraries for Windows
f8dbc88 
* Added the appropriate CMake flags and
  commands to enable only statically
  linked libraries and executables to
  be built on Windows.
@GauthamBanasandra
Run CI
e7c98b1
@GauthamBanasandra
Revert "Run CI"
a6c1a45 
This reverts commit e7c98b1.
@GauthamBanasandra GauthamBanasandra self-assigned this Jul 16, 2022
@hadoop-yetus

This comment was marked as outdated.

Sign in to view
@GauthamBanasandra
Add Windows flags only for MSVC
9dc5cd0 
* Also, removed Boost library
  linkage. Will create a separate
  PR for this.
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 44s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 40m 3s trunk passed
+1 💚 compile 4m 11s trunk passed
+1 💚 mvnsite 0m 45s trunk passed
+1 💚 shadedclient 65m 33s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 22s the patch passed
+1 💚 compile 3m 31s the patch passed
+1 💚 cc 3m 31s the patch passed
+1 💚 golang 3m 31s the patch passed
+1 💚 javac 3m 31s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 27s the patch passed
+1 💚 shadedclient 19m 6s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 36m 19s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 0m 53s The patch does not generate ASF License warnings.
129m 21s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/artifact/out/Dockerfile
GITHUB PR #4571
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang
uname Linux e4ee9e2b0bc6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 9dc5cd0
Default Java Red Hat, Inc.-1.8.0_332-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/testReport/
Max. process+thread count 704 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/console
versions git=2.9.5 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 21m 27s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 22m 36s trunk passed
+1 💚 compile 4m 6s trunk passed
+1 💚 mvnsite 1m 4s trunk passed
+1 💚 shadedclient 47m 33s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 31s the patch passed
+1 💚 compile 3m 43s the patch passed
+1 💚 cc 3m 43s the patch passed
+1 💚 golang 3m 43s the patch passed
+1 💚 javac 3m 43s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 34s the patch passed
+1 💚 shadedclient 19m 10s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 33m 5s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 1m 2s The patch does not generate ASF License warnings.
129m 44s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/artifact/out/Dockerfile
GITHUB PR #4571
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang
uname Linux 5dd21a11e1e5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 9dc5cd0
Default Java Red Hat, Inc.-1.8.0_312-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/testReport/
Max. process+thread count 680 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/console
versions git=2.27.0 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 21m 2s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 24m 53s trunk passed
+1 💚 compile 3m 34s trunk passed
+1 💚 mvnsite 0m 44s trunk passed
+1 💚 shadedclient 56m 15s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 25s the patch passed
+1 💚 compile 3m 17s the patch passed
+1 💚 cc 3m 17s the patch passed
+1 💚 golang 3m 17s the patch passed
+1 💚 javac 3m 17s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 27s the patch passed
+1 💚 shadedclient 26m 18s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 32m 22s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 0m 47s The patch does not generate ASF License warnings.
143m 19s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/artifact/out/Dockerfile
GITHUB PR #4571
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang
uname Linux 1237e12f1dd5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 9dc5cd0
Default Java Debian-11.0.15+10-post-Debian-1deb10u1
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/testReport/
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/console
versions git=2.20.1 maven=3.6.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 44s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 21m 34s trunk passed
+1 💚 compile 3m 58s trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 compile 4m 4s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 mvnsite 0m 44s trunk passed
+1 💚 shadedclient 50m 32s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 24s the patch passed
+1 💚 compile 3m 43s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 cc 3m 43s the patch passed
+1 💚 golang 3m 43s the patch passed
+1 💚 javac 3m 43s the patch passed
+1 💚 compile 3m 43s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 cc 3m 43s the patch passed
+1 💚 golang 3m 43s the patch passed
+1 💚 javac 3m 43s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 0m 28s the patch passed
+1 💚 shadedclient 19m 51s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 33m 9s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 0m 47s The patch does not generate ASF License warnings.
115m 43s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/artifact/out/Dockerfile
GITHUB PR #4571
Optional Tests dupname asflicense compile cc mvnsite javac unit codespell detsecrets golang
uname Linux 06912fdbae28 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 9dc5cd0
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/testReport/
Max. process+thread count 683 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4571/2/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@GauthamBanasandra GauthamBanasandra marked this pull request as ready for review July 17, 2022 15:15
@GauthamBanasandra GauthamBanasandra merged commit 4fb799e into apache:trunk Jul 19, 2022
@GauthamBanasandra GauthamBanasandra deleted the msvc-static-lib branch July 19, 2022 16:07
HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022
@GauthamBanasandra
HDFS-16464. Create only libhdfspp static libraries for Windows (apach…
0038f7b 
…e#4571)

* Added the appropriate CMake flags and
  commands to enable only statically
  linked libraries and executables to
  be built on Windows.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@goiri goiri goiri approved these changes

Assignees

@GauthamBanasandra GauthamBanasandra

Labels
HDFS
Projects
None yet
Milestone
No milestone
3 participants
@GauthamBanasandra @hadoop-yetus @goiri