8350807: Certificates using MD5 algorithm that are disabled by default are incorrectly allowed in TLSv1.3 when re-enabled

[artur-oracle]

MD5 algorithm is prohibited by TLSv1.3 RFC to be used in certificates:

Any endpoint receiving any certificate which it would need to
validate using any signature algorithm using an MD5 hash MUST abort
the handshake with a "bad_certificate" alert.

The bug manifests itself when older versions of protocol are supported besides TLSv1.3, such as TLSv1.2. When multiple protocol versions are supported, both client and server calculate their respective SSLSessions's "localSupportedSignAlgs" based on supported signature algorithms for all active protocols and don't update it when negotiated protocol is established. Then "localSupportedSignAlgs" list is used to validate certificate's algorithm.

While we disable "MD5withRSA" in java.security config, MD5 algorithm should not be allowed in TLSv1.3 regardless of optional configuration.

The underlying issue we are fixing here is not MD5-specific: when multiple TLS versions are supported, we compute local supported algorithms for ALL supported TLS versions. Thus MD5 and other algorithms that are supported in TLSv1.2 are being used when actually TLSv1.3 ends up being the negotiated protocol version.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8350807: Certificates using MD5 algorithm that are disabled by default are incorrectly allowed in TLSv1.3 when re-enabled (Bug - P3)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/24425/head:pull/24425
$ git checkout pull/24425

Update a local copy of the PR:
$ git checkout pull/24425
$ git pull https://git.openjdk.org/jdk.git pull/24425/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 24425

View PR using the GUI difftool:
$ git pr show -t 24425

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/24425.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back abarashev! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@artur-oracle The following labels will be automatically applied to this pull request:

• net
• security

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (134a3264)
• 00: Full (28f12786)