Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HADOOP-18357. Retarget solution file to VS2019 #4616

Closed
wants to merge 4 commits into from
Closed

HADOOP-18357. Retarget solution file to VS2019 #4616

wants to merge 4 commits into from

Conversation

GauthamBanasandra
Copy link
Member

@GauthamBanasandra GauthamBanasandra commented Jul 23, 2022 •
edited

Description of PR

The Visual Studio version used by winutils and native components in Hadoop common are quite old. We need to retarget the solution and vcxproj files to use the latest version (Visual Studio 2019 as of this writing).

With this PR, we can finally build Hadoop on Windows without any hacks.

Hadoop tar

image

Hadoop distribution

image

Hadoop native libraries

image

How was this patch tested?

  1. Tested by building locally on my Windows 10 PC.
  2. Hadoop Jenkins CI validation.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@GauthamBanasandra
HADOOP-18357. Retarget solution file to VS2019
33a7947 
* This PR retargets the Visual Studio solution
  and vcxproj files to use the latest version
  of Visual Studio (2019 as of this writing).
@GauthamBanasandra
Run CI
8093b32
@GauthamBanasandra
Revert "Run CI"
62f6e66 
This reverts commit 8093b32.
@GauthamBanasandra GauthamBanasandra self-assigned this Jul 23, 2022
@GauthamBanasandra GauthamBanasandra changed the title Retarget vs HADOOP-18357. Retarget solution file to VS2019 Jul 23, 2022
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 39s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 15m 17s Maven dependency ordering for branch
+1 💚 mvninstall 25m 41s trunk passed
+1 💚 compile 23m 20s trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 compile 20m 50s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 mvnsite 3m 26s trunk passed
+1 💚 javadoc 2m 50s trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 2m 26s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 shadedclient 115m 14s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 35s Maven dependency ordering for patch
+1 💚 mvninstall 1m 31s the patch passed
+1 💚 compile 22m 28s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 cc 22m 28s the patch passed
+1 💚 golang 22m 28s the patch passed
+1 💚 javac 22m 28s the patch passed
+1 💚 compile 20m 45s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 cc 20m 45s the patch passed
+1 💚 golang 20m 45s the patch passed
+1 💚 javac 20m 45s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 3m 24s the patch passed
+1 💚 javadoc 2m 39s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 2m 28s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 shadedclient 33m 35s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 18m 46s hadoop-common in the patch passed.
+1 💚 unit 33m 34s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 1m 34s The patch does not generate ASF License warnings.
250m 54s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4616/1/artifact/out/Dockerfile
GITHUB PR #4616
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets cc golang
uname Linux 410758acc5e5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 62f6e66
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4616/1/testReport/
Max. process+thread count 3158 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-native-client U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4616/1/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@GauthamBanasandra GauthamBanasandra marked this pull request as ready for review July 24, 2022 12:22
goiri
goiri reviewed Jul 24, 2022
View reviewed changes
hadoop-common-project/hadoop-common/src/main/native/native.vcxproj
@@ -39,12 +37,14 @@
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
<PlatformToolset>v142</PlatformToolset>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What was the old number and where was it defined?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the old number should've been v100. I'm not sure why it wasn't defined. Visual Studio automatically added this when I ran the command to retarget the project files.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we check if it was inherited from somewhere else?
This may trigger inconsistencies.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@goiri I checked on the current trunk. There's no PlatformToolset or v100 defined anywhere -

image

image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So how does it know which VS version to be scoped to?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran the retarget command from VS2019 command prompt, thus the retarget tool added the PlatformToolset version corresponding to VS2019.
The retarget command would take care of inserting the right version based on Visual Studio command prompt that we choose. Please also note that the subsequent mvn install command must be run from the same command prompt.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that we didn't have the target before; why would we add it now?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why would we add it now?

The compilation fails without the PlatformToolset when compiled with Visual Studio 2019 (I'm not sure about VS2010). I'm fine with documenting running the retarget command as a pre-requisite step prior to running the mvn install command, if we don't want to add the PlatformToolset here.

Copy link
Member

@goiri goiri Jul 27, 2022 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take a look at HADOOP-14667
Specially: MSBuild Solution files are converted to the version of VS at build time
If this makes sense, we go with this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is amazing. Thanks for sharing this @goiri 😊. I'm able to confirm that while building the hadoop-common-project module, the vcxproj and solution files automatically get upgraded to the appropriate Visual Studio version. All thanks to

hadoop/hadoop-common-project/hadoop-common/pom.xml

Lines 844 to 923 in 95a8587

<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<executions>
<execution>
<id>convert-ms-winutils</id>
<phase>generate-sources</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>${basedir}\..\..\dev-support\bin\win-vs-upgrade.cmd</executable>
<arguments>
<argument>${basedir}\src\main\winutils</argument>
<argument>${project.build.directory}</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>compile-ms-winutils</id>
<phase>compile</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>msbuild</executable>
<arguments>
<argument>${basedir}/src/main/winutils/winutils.sln</argument>
<argument>/nologo</argument>
<argument>/p:Configuration=Release</argument>
<argument>/p:OutDir=${project.build.directory}/bin/</argument>
<argument>/p:IntermediateOutputPath=${project.build.directory}/winutils/</argument>
<argument>/p:WsceConfigDir=${wsce.config.dir}</argument>
<argument>/p:WsceConfigFile=${wsce.config.file}</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>convert-ms-native-dll</id>
<phase>generate-sources</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>${basedir}\..\..\dev-support\bin\win-vs-upgrade.cmd</executable>
<arguments>
<argument>${basedir}\src\main\native</argument>
<argument>${project.build.directory}</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>compile-ms-native-dll</id>
<phase>compile</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>msbuild</executable>
<arguments>
<argument>${basedir}/src/main/native/native.sln</argument>
<argument>/nologo</argument>
<argument>/p:Configuration=Release</argument>
<argument>/p:OutDir=${project.build.directory}/bin/</argument>
<argument>/p:CustomZstdPrefix=${zstd.prefix}</argument>
<argument>/p:CustomZstdLib=${zstd.lib}</argument>
<argument>/p:CustomZstdInclude=${zstd.include}</argument>
<argument>/p:RequireZstd=${require.zstd}</argument>
<argument>/p:CustomOpensslPrefix=${openssl.prefix}</argument>
<argument>/p:CustomOpensslLib=${openssl.lib}</argument>
<argument>/p:CustomOpensslInclude=${openssl.include}</argument>
<argument>/p:RequireOpenssl=${require.openssl}</argument>
<argument>/p:RequireIsal=${require.isal}</argument>
<argument>/p:CustomIsalPrefix=${isal.prefix}</argument>
<argument>/p:CustomIsalLib=${isal.lib}</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin> 

[INFO] --- exec-maven-plugin:1.3.1:exec (convert-ms-native-dll) @ hadoop-common ---
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\devenv.com
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\devenv.exe

Microsoft Visual Studio 2019 Version 16.11.9.
Copyright (C) Microsoft Corp. All rights reserved.
Upgrading project 'native'...
        Configuration 'Release|Win32': changing Platform Toolset to 'v142' (was 'v100').
        Configuration 'Release|x64': changing Platform Toolset to 'v142' (was 'v100').

Migration completed successfully, but some warnings were detected during migration.
For more information, see the migration report:
E:\hadoop-common-project\hadoop-common\src\main\native\UpgradeLog.htm

I'm able to fully build the trunk as well -
image

I'll abandon this PR since I'm able to build the trunk on Windows without these changes.

hadoop-common-project/hadoop-common/src/main/native/native.vcxproj Outdated Show resolved Hide resolved
hadoop-common-project/hadoop-common/src/main/native/native.vcxproj.filters Show resolved Hide resolved
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 47s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 14m 54s Maven dependency ordering for branch
+1 💚 mvninstall 28m 37s trunk passed
+1 💚 compile 25m 8s trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 compile 21m 48s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 mvnsite 2m 53s trunk passed
+1 💚 javadoc 2m 19s trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 1m 52s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 shadedclient 121m 47s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 28s Maven dependency ordering for patch
+1 💚 mvninstall 1m 29s the patch passed
+1 💚 compile 24m 40s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 cc 24m 40s the patch passed
+1 💚 golang 24m 40s the patch passed
+1 💚 javac 24m 40s the patch passed
+1 💚 compile 21m 46s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 cc 21m 46s the patch passed
+1 💚 golang 21m 46s the patch passed
+1 💚 javac 21m 46s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 2m 51s the patch passed
+1 💚 javadoc 2m 7s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 1m 52s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 shadedclient 36m 16s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 18m 48s hadoop-common in the patch passed.
+1 💚 unit 87m 53s hadoop-hdfs-native-client in the patch passed.
+1 💚 asflicense 1m 16s The patch does not generate ASF License warnings.
317m 1s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4616/2/artifact/out/Dockerfile
GITHUB PR #4616
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets cc golang
uname Linux 9cdedda7801d 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 8d02105
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4616/2/testReport/
Max. process+thread count 2911 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-native-client U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4616/2/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@GauthamBanasandra
Copy link
Member Author

The vcxproj and solution files automatically get retargeted due to -

hadoop/hadoop-common-project/hadoop-common/pom.xml

Lines 844 to 923 in 95a8587

<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<executions>
<execution>
<id>convert-ms-winutils</id>
<phase>generate-sources</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>${basedir}\..\..\dev-support\bin\win-vs-upgrade.cmd</executable>
<arguments>
<argument>${basedir}\src\main\winutils</argument>
<argument>${project.build.directory}</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>compile-ms-winutils</id>
<phase>compile</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>msbuild</executable>
<arguments>
<argument>${basedir}/src/main/winutils/winutils.sln</argument>
<argument>/nologo</argument>
<argument>/p:Configuration=Release</argument>
<argument>/p:OutDir=${project.build.directory}/bin/</argument>
<argument>/p:IntermediateOutputPath=${project.build.directory}/winutils/</argument>
<argument>/p:WsceConfigDir=${wsce.config.dir}</argument>
<argument>/p:WsceConfigFile=${wsce.config.file}</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>convert-ms-native-dll</id>
<phase>generate-sources</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>${basedir}\..\..\dev-support\bin\win-vs-upgrade.cmd</executable>
<arguments>
<argument>${basedir}\src\main\native</argument>
<argument>${project.build.directory}</argument>
</arguments>
</configuration>
</execution>
<execution>
<id>compile-ms-native-dll</id>
<phase>compile</phase>
<goals>
<goal>exec</goal>
</goals>
<configuration>
<executable>msbuild</executable>
<arguments>
<argument>${basedir}/src/main/native/native.sln</argument>
<argument>/nologo</argument>
<argument>/p:Configuration=Release</argument>
<argument>/p:OutDir=${project.build.directory}/bin/</argument>
<argument>/p:CustomZstdPrefix=${zstd.prefix}</argument>
<argument>/p:CustomZstdLib=${zstd.lib}</argument>
<argument>/p:CustomZstdInclude=${zstd.include}</argument>
<argument>/p:RequireZstd=${require.zstd}</argument>
<argument>/p:CustomOpensslPrefix=${openssl.prefix}</argument>
<argument>/p:CustomOpensslLib=${openssl.lib}</argument>
<argument>/p:CustomOpensslInclude=${openssl.include}</argument>
<argument>/p:RequireOpenssl=${require.openssl}</argument>
<argument>/p:RequireIsal=${require.isal}</argument>
<argument>/p:CustomIsalPrefix=${isal.prefix}</argument>
<argument>/p:CustomIsalLib=${isal.lib}</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>

I'm able to build the trunk on Windows without this PR -
image

Closing this PR since these changes are not needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@goiri goiri Awaiting requested review from goiri

Assignees

@GauthamBanasandra GauthamBanasandra

Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@GauthamBanasandra @hadoop-yetus @goiri