JDK-8280703 CipherCore.doFinal(...) causes potentially massive byte[] allocations during decryption

[overheadhunter]

Related to #411, however it turns out that for unpadded ciphers, there is no need to allocate internalOutput, if output provides sufficient capacity.

For padded ciphers, only the unpadded cleartext is expected to be copied to the output buffer. In this case, there is no way around the temporary buffer (without major changes).

While a small change, please review with care, as I might be missing some security-relevant side effect (such as: don't copy cleartext to output buffer before validating the a tag - just as an example, even if there is no authentication involved in this method).

I have some test failures in Tier 1 tests, but these seem to be unrelated. Tests for com.sun.crypto and javax.crypto run fine:

==============================
Test summary
==============================
   TEST                                              TOTAL  PASS  FAIL ERROR   
   jtreg:test/jdk/com/sun/crypto                       141   141     0     0   
   jtreg:test/jdk/javax/crypto                          56    56     0     0   
==============================
TEST SUCCESS

---

Progress

• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• Change must be properly reviewed

Issue

• JDK-8280703: CipherCore.doFinal(...) causes potentially massive byte[] allocations during decryption

Reviewers

• Anthony Scarpino (@ascarpino - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/7230/head:pull/7230
$ git checkout pull/7230

Update a local copy of the PR:
$ git checkout pull/7230
$ git pull https://git.openjdk.java.net/jdk pull/7230/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 7230

View PR using the GUI difftool:
$ git pr show -t 7230

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/7230.diff

[bridgekeeper]

👋 Welcome back overheadhunter! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@overheadhunter The following label will be automatically applied to this pull request:

• security

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (ffef8362)
• 00: Full (520fec52)

[ascarpino]

The copyright date will need to be updated from 2021 to 2022

[ascarpino]

The diff passed tier1 and crypto tests. Looks good to me

[openjdk]

@overheadhunter This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8280703: CipherCore.doFinal(...) causes potentially massive byte[] allocations during decryption

Reviewed-by: ascarpino

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 24 new commits pushed to the master branch:

• ece89c6: 8280366: (fs) Restore Files.createTempFile javadoc
• b94ebaa: 8280686: Remove Compile::print_method_impl
• a3a0dcd: 8280353: -XX:ArchiveClassesAtExit should print warning if base archive failed to load
• cab5905: 8280583: Always build NMT
• 7f68759: 8280719: G1: Remove outdated comment in RemoveSelfForwardPtrObjClosure::apply
• 94380d0: 8278232: [macos] Wrong chars emitted when entering certain char-sequence of Indic language
• 0dba170: 8278518: String(byte[], int, int, Charset) constructor and String.translateEscapes() miss bounds check elimination
• 2ea0edf: 8279673: AudioClip.play doesn't work due to NullPointerException when creating DataPusher
• 8908301: 8076089: Cleanup: Inline & remove sun.management.Util.newException
• c2ee1b3: 8273236: keytool does not accurately warn about algorithms that are disabled but have additional constraints
• ... and 14 more: https://git.openjdk.java.net/jdk/compare/e72eefd9f66e63a1e11d582e4916374840111928...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@ascarpino) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

[overheadhunter]

/integrate

[openjdk]

@overheadhunter
Your change (at version ffef836) is now ready to be sponsored by a Committer.

[ascarpino]

/sponsor

[openjdk]

Going to push as commit 409382b.
Since your change was applied there have been 36 commits pushed to the master branch:

• cb8a82e: 8272317: jstatd has dependency on Security Manager which needs to be removed
• 6de90ad: 8280863: Update build README to reflect that MSYS2 is supported
• ed826f2: 8280397: Factor out task queue statistics printing
• 8a3cca0: 8280784: VM_Cleanup unnecessarily processes all thread oops
• 973dda5: 8280804: Parallel: Remove unused variables in PSPromotionManager::drain_stacks_depth
• 55f180f: 8280004: DCmdArgument::parse_value() should handle NULL input
• 178ac74: 8251466: test/java/io/File/GetXSpace.java fails on Windows with mapped network drives.
• a1d1e47: 8280823: Remove NULL check in DumpTimeClassInfo::is_excluded
• 094db1a: 8277948: AArch64: Print the correct native stack if -XX:+PreserveFramePointer when crash
• 7857405: 8280744: Allow SuppressWarnings to be used in all declaration contexts
• ... and 26 more: https://git.openjdk.java.net/jdk/compare/e72eefd9f66e63a1e11d582e4916374840111928...master

Your commit was automatically rebased without conflicts.

[openjdk]

@ascarpino @overheadhunter Pushed as commit 409382b.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.