8271199: Mutual TLS handshake fails signing client certificate with custom sensitive PKCS11 key

[alexeybakhtin]

Please review backport of JDK-8271199 to 11u
Backport is not clean because of changes introduced by JDK-8023980 and JDK-8172366. These changes are not related to functionality of JDK-8271199 so merge is trivial.
Also, there are simple changes related to instanceof pattern not implemented in JDK11.

sun/security/rsa jtreg and custom test with IAIK provider are passed

---

Progress

• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• Change must be properly reviewed

Issue

• JDK-8271199: Mutual TLS handshake fails signing client certificate with custom sensitive PKCS11 key

Reviewers

• Martin Doerr (@TheRealMDoerr - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk11u-dev pull/1036/head:pull/1036
$ git checkout pull/1036

Update a local copy of the PR:
$ git checkout pull/1036
$ git pull https://git.openjdk.java.net/jdk11u-dev pull/1036/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 1036

View PR using the GUI difftool:
$ git pr show -t 1036

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk11u-dev/pull/1036.diff

[bridgekeeper]

👋 Welcome back abakhtin! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

This backport pull request has now been updated with issue from the original commit.

[mlbridge]

Webrevs

• 00: Full (17b2666d)

[TheRealMDoerr]

LGTM.

[openjdk]

@alexeybakhtin This change now passes all automated pre-integration checks.

After integration, the commit message for the final commit will be:

8271199: Mutual TLS handshake fails signing client certificate with custom sensitive PKCS11 key

Reviewed-by: mdoerr

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 25 new commits pushed to the master branch:

• 987f7af: 8263718: unused-result warning happens at os_linux.cpp
• dbe5f17: 8285445: cannot open file "NUL:"
• 61b8c91: 8284622: Update versions of some Github Actions used in JDK workflow
• d9be932: 8285395: [JVMCI] [11u] Partial backport of JDK-8220623: InstalledCode
• 569ae08: 8284033: Leak XVisualInfo in getAllConfigs in awt_GraphicsEnv.c
• 2037210: Merge
• 224e1a3: 8284920: Incorrect Token type causes XPath expression to return empty result
• 91d6550: 8284548: Invalid XPath expression causes StringIndexOutOfBoundsException
• 5738fd4: 8282300: Throws NamingException instead of InvalidNameException after JDK-8278972
• 666f21f: 8278972: Improve URL supports
• ... and 15 more: https://git.openjdk.java.net/jdk11u-dev/compare/7b8461119bc3dabf82bac3c73475cb9ce71ec970...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@TheRealMDoerr) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

[alexeybakhtin]

/integrate

[openjdk]

@alexeybakhtin
Your change (at version 17b2666) is now ready to be sponsored by a Committer.

[yan-too]

/sponsor

[openjdk]

Going to push as commit 7e4af05.
Since your change was applied there have been 25 commits pushed to the master branch:

• 987f7af: 8263718: unused-result warning happens at os_linux.cpp
• dbe5f17: 8285445: cannot open file "NUL:"
• 61b8c91: 8284622: Update versions of some Github Actions used in JDK workflow
• d9be932: 8285395: [JVMCI] [11u] Partial backport of JDK-8220623: InstalledCode
• 569ae08: 8284033: Leak XVisualInfo in getAllConfigs in awt_GraphicsEnv.c
• 2037210: Merge
• 224e1a3: 8284920: Incorrect Token type causes XPath expression to return empty result
• 91d6550: 8284548: Invalid XPath expression causes StringIndexOutOfBoundsException
• 5738fd4: 8282300: Throws NamingException instead of InvalidNameException after JDK-8278972
• 666f21f: 8278972: Improve URL supports
• ... and 15 more: https://git.openjdk.java.net/jdk11u-dev/compare/7b8461119bc3dabf82bac3c73475cb9ce71ec970...master

Your commit was automatically rebased without conflicts.

[openjdk]

@yan-too @alexeybakhtin Pushed as commit 7e4af05.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.