8267125: AES Galois CounterMode (GCM) interleaved implementation using AVX512 + VAES instructions #4019

smita-kamath · 2021-05-14T00:42:35Z

I would like to submit AES-GCM optimization for x86_64 architectures supporting AVX3+VAES (Evex encoded AES). This optimization interleaves AES and GHASH operations.
Performance gain of ~1.5x - 2x for message sizes 8k and above.
/contributor add svkamath
/contributor add Tomasz Kantecki tomasz.kantecki@intel.com

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8267125: AES Galois CounterMode (GCM) interleaved implementation using AVX512 + VAES instructions

Reviewers

Vladimir Kozlov (@vnkozlov - Reviewer)
Valerie Peng (@valeriepeng - Reviewer) ⚠️ Review applies to 112e3fc

Contributors

Smita Kamath <svkamath@openjdk.org>
Tomasz Kantecki <tomasz.kantecki@intel.com>
Anthony Scarpino <ascarpino@openjdk.org>

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/4019/head:pull/4019
$ git checkout pull/4019

Update a local copy of the PR:
$ git checkout pull/4019
$ git pull https://git.openjdk.java.net/jdk pull/4019/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 4019

View PR using the GUI difftool:
$ git pr show -t 4019

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/4019.diff

…using AVX512 + VAES instructions

bridgekeeper · 2021-05-14T00:43:12Z

👋 Welcome back svkamath! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-05-14T00:44:16Z

@smita-kamath this pull request can not be integrated into master due to one or more merge conflicts. To resolve these merge conflicts and update this pull request you can run the following commands in the local repository for your personal fork:

git checkout aes-gcm
git fetch https://git.openjdk.java.net/jdk master
git merge FETCH_HEAD
# resolve conflicts and follow the instructions given by git merge
git commit -m "Merge master"
git push

openjdk · 2021-05-14T00:46:01Z

@smita-kamath
Contributor Smita Kamath <svkamath@openjdk.org> successfully added.

openjdk · 2021-05-14T00:47:10Z

@smita-kamath
Contributor Tomasz Kantecki <tomasz.kantecki@intel.com> successfully added.

openjdk · 2021-05-14T00:50:30Z

@smita-kamath The following labels will be automatically applied to this pull request:

hotspot
security

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

mlbridge · 2021-05-14T02:13:57Z

Webrevs

neliasso · 2021-05-28T09:49:17Z

Added hotspot-compiler label

openjdk · 2021-05-28T09:49:27Z

@neliasso Unknown command add - for a list of valid commands use /help.

neliasso · 2021-05-28T09:51:57Z

/label add hotspot-compiler

openjdk · 2021-05-28T09:52:57Z

@neliasso
The hotspot-compiler label was successfully added.

…e and subkeyHtbl

ascarpino · 2021-06-09T18:49:01Z

With JDK-8255557 integrated, I'll provide you a merged copy of your java side changes.

vnkozlov

Do you plan to implement decrypt intrinsic too?

vnkozlov · 2021-06-11T15:45:02Z

src/hotspot/share/opto/library_call.cpp

@@ -543,6 +543,9 @@ bool LibraryCallKit::try_to_inline(int predicate) {
  case vmIntrinsics::_counterMode_AESCrypt:
    return inline_counterMode_AESCrypt(intrinsic_id());

+  case vmIntrinsics::_galoisCounterMode_AESCrypt:
+    return inline_galoisCounterMode_AESCrypt(intrinsic_id());


You don't need to pass intrinsic_id() for this implementation unless you plan to add decrypt intrinsic later.

Thanks for your comments Vladimir. The intrinsic is called for encrypt as well as decrypt operation.

Only one intrinsic is declared in this change: _galoisCounterMode_AESCrypt. Other AES intrinsics have 2 that is why they have to pass intrinsic_id(). See lines before this.

Note, _counterMode_AESCrypt is not example - it has the same issue.

vnkozlov · 2021-06-11T15:50:42Z

src/hotspot/share/opto/library_call.cpp

+         top_ct != NULL && top_ct->klass() != NULL &&
+         top_out != NULL && top_out->klass() != NULL, "args are strange");
+
+  // checks are the responsibility of the caller


Do you have all NULL for all objects and range checks in Java code for this intrinsic?

vnkozlov · 2021-06-11T15:53:37Z

src/hotspot/share/opto/library_call.cpp

+  Node* counter = load_field_from_object(gctr_object, "counter", "[B");
+  Node* subkeyHtbl = load_field_from_object(ghash_object, "subkeyHtbl", "[J");
+  Node* state = load_field_from_object(ghash_object, "state", "[J");
+  if (embeddedCipherObj == NULL || counter == NULL || subkeyHtbl == NULL || state == NULL) return false;


Follow coding style for such long condition:

if () { return false; }

I will make the change. Thanks.

bridgekeeper · 2021-07-09T18:17:58Z

@smita-kamath This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply add a new comment to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

vnkozlov · 2021-07-15T00:19:23Z

Looks like you have some issues: wrong file property.

vnkozlov · 2021-08-18T01:16:20Z

src/hotspot/cpu/x86/stubGenerator_x86_64.cpp

+        StubRoutines::x86::_counter_mask_addr = counter_mask_addr();
+        StubRoutines::x86::_ghash_poly512_addr = ghash_polynomial512_addr();
+        StubRoutines::x86::_ghash_long_swap_mask_addr = generate_ghash_long_swap_mask();


_counter_mask_addr = counter_mask_addr() and _ghash_long_swap_mask_addr = generate_ghash_long_swap_mask() are called by other intrinsics too. Which duplicates code in codeCache.
They should be called only once when they are used. May be counter_mask_addr() and generate_ghash_long_swap_mask() method should check that addresses already recorded and return it.

vnkozlov · 2021-08-18T01:26:09Z

For the record, I withdraw my request to put this on hold until Lazy stub generation is complete.
As discussed offline I agree with pushing this changes first.

Second, when final changes are ready before push, please, ask Oracle engineer to run tier1-3 in our testing infrastructure.

…calls are not duplicated

vnkozlov

Good.

I will test it.

smita-kamath · 2021-08-23T19:06:40Z

/contributor add ascarpino

openjdk · 2021-08-23T19:07:31Z

@smita-kamath Could not parse ascarpino anthony.scarpino@oracle.com as a valid contributor.
Syntax: /contributor (add|remove) [@user | openjdk-user | Full Name <email@address>]. For example:

/contributor add @openjdk-bot
/contributor add duke
/contributor add J. Duke <duke@openjdk.org>

ascarpino · 2021-08-23T20:04:00Z

Tier 1-3 passed on linux-x64, windows-x64, macosx-x64, linux-aarch64

smita-kamath · 2021-08-23T20:31:36Z

/contributor add Anthony Scarpino ascarpino@openjdk.org

smita-kamath · 2021-08-23T20:31:49Z

/contributor add Anthony Scarpino ascarpino@openjdk.org

openjdk · 2021-08-23T20:32:27Z

@smita-kamath Could not parse Anthony Scarpino ascarpino@openjdk.org as a valid contributor.
Syntax: /contributor (add|remove) [@user | openjdk-user | Full Name <email@address>]. For example:

/contributor add @openjdk-bot
/contributor add duke
/contributor add J. Duke <duke@openjdk.org>

openjdk · 2021-08-23T20:32:44Z

@smita-kamath Could not parse Anthony Scarpino ascarpino@openjdk.org as a valid contributor.
Syntax: /contributor (add|remove) [@user | openjdk-user | Full Name <email@address>]. For example:

/contributor add @openjdk-bot
/contributor add duke
/contributor add J. Duke <duke@openjdk.org>

smita-kamath · 2021-08-23T20:33:38Z

/contributor add ascarpino

openjdk · 2021-08-23T20:34:41Z

@smita-kamath
Contributor Anthony Scarpino <ascarpino@openjdk.org> successfully added.

vnkozlov · 2021-08-24T16:21:24Z

Yes, my testing during weekend passed too.
Changes are good for integration.

smita-kamath · 2021-08-24T16:52:36Z

/integrate

openjdk · 2021-08-24T16:54:48Z

@smita-kamath
Your change (at version c31bfe9) is now ready to be sponsored by a Committer.

smita-kamath · 2021-08-24T16:57:33Z

@ascarpino Looks like AES-GCM is ready to be integrated. Can you sponsor this patch? Thank you.

ascarpino · 2021-08-24T17:16:24Z

/sponsor

kevinrushforth · 2021-08-24T18:47:02Z

Checking to see if the bot is awake (it should reject my attempt).

/sponsor

openjdk · 2021-08-24T18:48:44Z

Going to push as commit 0e7288f.
Since your change was applied there have been 39 commits pushed to the master branch:

6ace805: 8272856: DoubleFlagWithIntegerValue uses G1GC-only flag
c547ead: 8272916: Copyright year was modified unintentionally in jlink.properties and ImagePluginStack.java
aaedac6: 8265261: java/nio/file/Files/InterruptCopy.java fails with java.lang.RuntimeException: Copy was not interrupted
d34f17c: 8270195: Add missing links between methods of JavaFX properties
f608e81: 8264322: Generate CDS archive when creating custom JDK image
f681d65: 8272806: [macOS] "Apple AWT Internal Exception" when input method is changed
6e0328f: 8272725: G1: add documentation on needs_remset_update_t vs bool
2309b7d: 8253178: Replace LinkedList Impl in net.http.FilterFactory
94f5e44: 8271258: @param with non-ascii variable names produces incorrect results
7454306: 8272526: Cleanup ThreadStateTransition class
... and 29 more: https://git.openjdk.java.net/jdk/compare/f4be211ae290824cb6c678dcdff0df91a20117d6...master

Your commit was automatically rebased without conflicts.

openjdk · 2021-08-24T18:49:05Z

@ascarpino @smita-kamath Pushed as commit 0e7288f.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

openjdk · 2021-08-24T18:49:59Z

@kevinrushforth The command sponsor can only be used in open pull requests.

JDK-8267125: AES Galois CounterMode (GCM) interleaved implementation …

c170226

…using AVX512 + VAES instructions

openjdk bot changed the title ~~JDK-8267125: AES Galois CounterMode (GCM) interleaved implementation …~~ 8267125: AES Galois CounterMode (GCM) interleaved implementation using AVX512 + VAES instructions May 14, 2021

openjdk bot added the merge-conflict Pull request has merge conflict with target branch label May 14, 2021

openjdk bot added security security-dev@openjdk.org hotspot hotspot-dev@openjdk.org labels May 14, 2021

Merge master

433176d

openjdk bot removed the merge-conflict Pull request has merge conflict with target branch label May 14, 2021

smita-kamath marked this pull request as ready for review May 14, 2021 02:08

openjdk bot added the rfr Pull request is ready for review label May 14, 2021

openjdk bot added the hotspot-compiler hotspot-compiler-dev@openjdk.org label May 28, 2021

openjdk bot added the merge-conflict Pull request has merge conflict with target branch label May 28, 2021

8267125:Updated intrinsic signature to remove copies of counter, stat…

2ac209e

…e and subkeyHtbl

vnkozlov reviewed Jun 11, 2021

View reviewed changes

merge master

4b6e881

openjdk bot removed the merge-conflict Pull request has merge conflict with target branch label Jul 13, 2021

Updated AES-GCM intrinsic to match latest Java Code

4a36816

openjdk bot removed the rfr Pull request is ready for review label Jul 14, 2021

vnkozlov reviewed Aug 18, 2021

View reviewed changes

smita-kamath added 2 commits August 19, 2021 13:36

Merge branch 'master' of https://git.openjdk.java.net/jdk into aes-gcm

94e46aa

changes to make sure that ghash_long_swap_mask and counter_mask_addr …

c31bfe9

…calls are not duplicated

vnkozlov approved these changes Aug 21, 2021

View reviewed changes

openjdk bot added the sponsor Pull request is ready to be sponsored label Aug 24, 2021

openjdk bot closed this Aug 24, 2021

openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated rfr Pull request is ready for review sponsor Pull request is ready to be sponsored labels Aug 24, 2021

schlosna mentioned this pull request Sep 23, 2022

Expand EncryptionBenchmark palantir/hadoop-crypto#586

Merged

8267125: AES Galois CounterMode (GCM) interleaved implementation using AVX512 + VAES instructions #4019

8267125: AES Galois CounterMode (GCM) interleaved implementation using AVX512 + VAES instructions #4019

Conversation

smita-kamath commented May 14, 2021 • edited by openjdk bot

Progress

Issue

Reviewers

Contributors

Reviewing

bridgekeeper bot commented May 14, 2021

openjdk bot commented May 14, 2021

openjdk bot commented May 14, 2021

openjdk bot commented May 14, 2021

openjdk bot commented May 14, 2021

mlbridge bot commented May 14, 2021 • edited

Webrevs

neliasso commented May 28, 2021

openjdk bot commented May 28, 2021

neliasso commented May 28, 2021

openjdk bot commented May 28, 2021

ascarpino commented Jun 9, 2021

vnkozlov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bridgekeeper bot commented Jul 9, 2021

vnkozlov commented Jul 15, 2021

Choose a reason for hiding this comment

vnkozlov commented Aug 18, 2021

vnkozlov left a comment

Choose a reason for hiding this comment

smita-kamath commented Aug 23, 2021 • edited

openjdk bot commented Aug 23, 2021

ascarpino commented Aug 23, 2021

smita-kamath commented Aug 23, 2021

smita-kamath commented Aug 23, 2021

openjdk bot commented Aug 23, 2021

openjdk bot commented Aug 23, 2021

smita-kamath commented Aug 23, 2021

openjdk bot commented Aug 23, 2021

vnkozlov commented Aug 24, 2021

smita-kamath commented Aug 24, 2021

openjdk bot commented Aug 24, 2021

smita-kamath commented Aug 24, 2021

ascarpino commented Aug 24, 2021

kevinrushforth commented Aug 24, 2021

openjdk bot commented Aug 24, 2021

openjdk bot commented Aug 24, 2021

openjdk bot commented Aug 24, 2021

smita-kamath commented May 14, 2021 •

edited by openjdk bot

mlbridge bot commented May 14, 2021 •

edited

smita-kamath commented Aug 23, 2021 •

edited