8317340: Windows builds are not reproducible if MS VS compiler install path differs

[andrew-m-leonard]

When doing a reproducible Windows build on two different machines, that
happen to have MS Visual Studio installed in different paths, the resulting
builds are not identical. This is due to the System include paths, eg:

C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\vcruntime_new.h

appearing in the compiler obj as part of debug info.

This PR tests when doing a Windows no-Absolute paths allowed build, and then
adds -pathmap:<VS_include_long_path>=vsi entries to re-map those entries in
the compiler object.
The VS cl compiler generates these paths as standard Windows "long path" entries,
and must be pathmap'd as such.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8317340: Windows builds are not reproducible if MS VS compiler install path differs (Bug - P4)

Reviewers

• Erik Joelsson (@erikj79 - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/16008/head:pull/16008
$ git checkout pull/16008

Update a local copy of the PR:
$ git checkout pull/16008
$ git pull https://git.openjdk.org/jdk.git pull/16008/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 16008

View PR using the GUI difftool:
$ git pr show -t 16008

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/16008.diff

Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back aleonard! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@andrew-m-leonard The following label will be automatically applied to this pull request:

• build

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 00: Full (8a9c36af)

[erikj79]

What effect will this have on debugging?

[andrew-m-leonard]

What effect will this have on debugging?

Hi @erikj79
So this is the same as we pathmap the openjdk workspace source just above, in that we only do it when ALLOW_ABSOLUTE_PATHS_IN_OUTPUT is "false", which is for release builds.
For "debug" builds it is "true" and no pathmap'ing occurs, see:

jdk/make/autoconf/jdk-options.m4

Line 696 in 2637e8d

# Disallow any output from containing absolute paths from the build system.

[andrew-m-leonard]

jdk/make/autoconf/flags-cflags.m4

Line 830 in 2637e8d

if test "x$ALLOW_ABSOLUTE_PATHS_IN_OUTPUT" = "xfalse"; then

[erikj79]

What effect will this have on debugging?

Hi @erikj79 So this is the same as we pathmap the openjdk workspace source just above, in that we only do it when ALLOW_ABSOLUTE_PATHS_IN_OUTPUT is "false", which is for release builds. For "debug" builds it is "true" and no pathmap'ing occurs, see:

Yes, I'm aware of that. I was just curious if you had tested and investigated the consequences of doing this when trying to debug a "release" build. I'm not against this change, but it would be good to understand any consequences.

[andrew-m-leonard]

What effect will this have on debugging?

Hi @erikj79 So this is the same as we pathmap the openjdk workspace source just above, in that we only do it when ALLOW_ABSOLUTE_PATHS_IN_OUTPUT is "false", which is for release builds. For "debug" builds it is "true" and no pathmap'ing occurs, see:

Yes, I'm aware of that. I was just curious if you had tested and investigated the consequences of doing this when trying to debug a "release" build. I'm not against this change, but it would be good to understand any consequences.

Ah I see :-)
So for debugging a release build it would mean potentially any VS symbols would not resolve, but that could quite possibly be the case now anyway as if a developer didn't have the same MS VS path/version as was used to build the "release" JDK image by the vendor.

[andrew-m-leonard]

@erikj79 I shall run a debug session on a release image and see what if anything it causes.

[andrew-m-leonard]

What effect will this have on debugging?

Hi @erikj79 So this is the same as we pathmap the openjdk workspace source just above, in that we only do it when ALLOW_ABSOLUTE_PATHS_IN_OUTPUT is "false", which is for release builds. For "debug" builds it is "true" and no pathmap'ing occurs, see:

Yes, I'm aware of that. I was just curious if you had tested and investigated the consequences of doing this when trying to debug a "release" build. I'm not against this change, but it would be good to understand any consequences.

@erikj79 So I started up a VS2022 debugging session for a java.exe process, and the vcruntime140 debugging information is loaded via the Microsoft Symbol Server anyway, same with debug src. Your VS2022 tools debugging source PATH for the VC tools is also pointing at your local VS2022 include folder.

[andrew-m-leonard]

@erikj79 So done some debugging of the main Java os_windows.cpp thread and debugging is fine, openjdk pdb's loaded fine, and vcruntime symbols loaded from Symbol Server. Also, did a debug session without these changes, and all behaves identically.

[openjdk]

@andrew-m-leonard This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8317340: Windows builds are not reproducible if MS VS compiler install path differs

Reviewed-by: erikj

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 17 new commits pushed to the master branch:

• 3bcfac1: 8317246: Cleanup java.net.URLEncoder and URLDecoder use of file.encoding property
• b6a97c0: 8316880: AArch64: "stop: Header is not fast-locked" with -XX:-UseLSE since JDK-8315880
• 287b243: 8316893: Compile without -fno-delete-null-pointer-checks
• 26c21f5: 8314294: Unsafe::allocateMemory and Unsafe::freeMemory are slower than malloc/free
• 6e1aacd: 8296631: NSS tests failing on OL9 linux-aarch64 hosts
• d2e2c4c: 8309667: TLS handshake fails because of ConcurrentModificationException in PKCS12KeyStore.engineGetEntry
• e25121d: 8316929: Shenandoah: Shenandoah degenerated GC and full GC need to cleanup old OopMapCache entries
• 5c8366e: 8268622: Performance issues in javac Name class
• ad81abd: 8317034: Remove redundant type cast in the java.util.stream package
• d7d1d42: 8316771: Krb5.java has not defined messages for all error codes
• ... and 7 more: https://git.openjdk.org/jdk/compare/eeb63cd0fad9de1f7182ab97d0f25ac67745aa9e...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[andrew-m-leonard]

/integrate

[openjdk]

Going to push as commit 353d139.
Since your change was applied there have been 17 commits pushed to the master branch:

• 3bcfac1: 8317246: Cleanup java.net.URLEncoder and URLDecoder use of file.encoding property
• b6a97c0: 8316880: AArch64: "stop: Header is not fast-locked" with -XX:-UseLSE since JDK-8315880
• 287b243: 8316893: Compile without -fno-delete-null-pointer-checks
• 26c21f5: 8314294: Unsafe::allocateMemory and Unsafe::freeMemory are slower than malloc/free
• 6e1aacd: 8296631: NSS tests failing on OL9 linux-aarch64 hosts
• d2e2c4c: 8309667: TLS handshake fails because of ConcurrentModificationException in PKCS12KeyStore.engineGetEntry
• e25121d: 8316929: Shenandoah: Shenandoah degenerated GC and full GC need to cleanup old OopMapCache entries
• 5c8366e: 8268622: Performance issues in javac Name class
• ad81abd: 8317034: Remove redundant type cast in the java.util.stream package
• d7d1d42: 8316771: Krb5.java has not defined messages for all error codes
• ... and 7 more: https://git.openjdk.org/jdk/compare/eeb63cd0fad9de1f7182ab97d0f25ac67745aa9e...master

Your commit was automatically rebased without conflicts.

[openjdk]

@andrew-m-leonard Pushed as commit 353d139.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

[erikj79]

It looks like this change broke the build on WSL. The pathtool in WSL (wslpath) does not support the -l option.