8322768: Optimize non-subword vector compress and expand APIs for AVX2 target. #17261

jatin-bhateja · 2024-01-04T05:28:59Z

Hi,

Patch optimizes non-subword vector compress and expand APIs for x86 AVX2 only targets.
Upcoming E-core Xeons (Sierra Forest) and Hybrid CPUs only support AVX2 instruction set.
These are very frequently used APIs in columnar database filter operation.

Implementation uses a lookup table to record permute indices. Table index is computed using
mask argument of compress/expand operation.

Following are the performance number of JMH micro included with the patch.

System : Intel(R) Xeon(R) Platinum 8480+ (Sapphire Rapids)

Baseline:
Benchmark                                 (size)   Mode  Cnt    Score   Error   Units
ColumnFilterBenchmark.filterDoubleColumn    1024  thrpt    2  142.767          ops/ms
ColumnFilterBenchmark.filterDoubleColumn    2047  thrpt    2   71.436          ops/ms
ColumnFilterBenchmark.filterDoubleColumn    4096  thrpt    2   35.992          ops/ms
ColumnFilterBenchmark.filterFloatColumn     1024  thrpt    2  182.151          ops/ms
ColumnFilterBenchmark.filterFloatColumn     2047  thrpt    2   91.096          ops/ms
ColumnFilterBenchmark.filterFloatColumn     4096  thrpt    2   44.757          ops/ms
ColumnFilterBenchmark.filterIntColumn       1024  thrpt    2  184.099          ops/ms
ColumnFilterBenchmark.filterIntColumn       2047  thrpt    2   91.981          ops/ms
ColumnFilterBenchmark.filterIntColumn       4096  thrpt    2   45.170          ops/ms
ColumnFilterBenchmark.filterLongColumn      1024  thrpt    2  148.017          ops/ms
ColumnFilterBenchmark.filterLongColumn      2047  thrpt    2   73.516          ops/ms
ColumnFilterBenchmark.filterLongColumn      4096  thrpt    2   36.844          ops/ms

Withopt:
Benchmark                                 (size)   Mode  Cnt     Score   Error   Units
ColumnFilterBenchmark.filterDoubleColumn    1024  thrpt    2  2051.707          ops/ms
ColumnFilterBenchmark.filterDoubleColumn    2047  thrpt    2   914.072          ops/ms
ColumnFilterBenchmark.filterDoubleColumn    4096  thrpt    2   489.898          ops/ms
ColumnFilterBenchmark.filterFloatColumn     1024  thrpt    2  5324.195          ops/ms
ColumnFilterBenchmark.filterFloatColumn     2047  thrpt    2  2587.229          ops/ms
ColumnFilterBenchmark.filterFloatColumn     4096  thrpt    2  1278.665          ops/ms
ColumnFilterBenchmark.filterIntColumn       1024  thrpt    2  4149.384          ops/ms
ColumnFilterBenchmark.filterIntColumn       2047  thrpt    2  1791.170          ops/ms
ColumnFilterBenchmark.filterIntColumn       4096  thrpt    2   974.888          ops/ms
ColumnFilterBenchmark.filterLongColumn      1024  thrpt    2  1128.281          ops/ms
ColumnFilterBenchmark.filterLongColumn      2047  thrpt    2   686.334          ops/ms
ColumnFilterBenchmark.filterLongColumn      4096  thrpt    2   337.170          ops/ms

Kindly review and share your feedback.

Best Regards,
Jatin

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8322768: Optimize non-subword vector compress and expand APIs for AVX2 target. (Enhancement - P4)

Reviewers

Emanuel Peter (@eme64 - Reviewer)
Sandhya Viswanathan (@sviswa7 - Reviewer) ⚠️ Review applies to cd912308

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/17261/head:pull/17261
$ git checkout pull/17261

Update a local copy of the PR:
$ git checkout pull/17261
$ git pull https://git.openjdk.org/jdk.git pull/17261/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 17261

View PR using the GUI difftool:
$ git pr show -t 17261

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/17261.diff

Webrev

Link to Webrev Comment

…2 target.

bridgekeeper · 2024-01-04T05:29:56Z

👋 Welcome back jbhateja! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2024-01-04T05:30:52Z

@jatin-bhateja The following labels will be automatically applied to this pull request:

core-libs
hotspot-compiler

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

mlbridge · 2024-01-04T05:33:34Z

Webrevs

eme64

@jatin-bhateja this looks like a great improvement!
I have a few questions and requests below.

FYI, this feels very inspiring. I'm dreaming of a day where we could do this filtering in the auto-vectorizer directly.

src/hotspot/cpu/x86/c2_MacroAssembler_x86.hpp

test/micro/org/openjdk/bench/jdk/incubator/vector/ColumnFilterBenchmark.java

src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp

test/micro/org/openjdk/bench/jdk/incubator/vector/ColumnFilterBenchmark.java

src/hotspot/cpu/x86/stubGenerator_x86_64.cpp

src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp

eme64

Thanks for the updates!

One more idea: Your AVX2 solution has a lot of cost for converting the mask to a permutation. Might it make sense to split this off into a separate vector-node, so that it can float out of a loop if the mask is invariant?

src/hotspot/cpu/x86/stubGenerator_x86_64.cpp

src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp

test/micro/org/openjdk/bench/jdk/incubator/vector/ColumnFilterBenchmark.java

src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp

jatin-bhateja · 2024-01-08T06:06:22Z

Thanks for the updates!

One more idea: Your AVX2 solution has a lot of cost for converting the mask to a permutation. Might it make sense to split this off into a separate vector-node, so that it can float out of a loop if the mask is invariant?

CompressV / ExpandV only accepts two inputs, vector to be operated on and mask under which operation is performed, permute table based implementation is specific to x86 backend implementation.

merykitty · 2024-01-08T10:20:33Z

@jatin-bhateja I think you can expand them in the matcher into several MachNodes that will get scheduled separately.

eme64

Exactly, like @merykitty suggests: you can do a platform-dependent expansion.

src/hotspot/cpu/x86/stubGenerator_x86_64.cpp

test/micro/org/openjdk/bench/jdk/incubator/vector/ColumnFilterBenchmark.java

jatin-bhateja · 2024-01-09T07:40:01Z

Exactly, like @merykitty suggests: you can do a platform-dependent expansion.

Hi @merykitty , @eme64 , in principle platform specific lowering is a good idea where ever useful, our main concern here is to identify a loop invariant constant mask in matcher patterns and save the cost of re-loading from a permute table index. Existing loop invariant analysis moves invariant masks out of loop and GCM should be able to move expanded load from permute table out of loop.

But this looks very restrictive and will mainly be useful for constant one hot bit mask pattern. A constant mask may have more than one set bits and in such a case we will need to generate multiple loads from permute tables and handle multiple expansion scenarios. I think we can defer that complexity for the time being.

eme64

I think we are almost there! 😊

src/hotspot/cpu/x86/stubGenerator_x86_64.cpp

src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp

test/micro/org/openjdk/bench/jdk/incubator/vector/ColumnFilterBenchmark.java

jatin-bhateja · 2024-01-10T19:17:14Z

Following are the performance numbers for existing Vector API JMH micro benchmark over Meteor Lake - Crestmont E-cores.

test/micro/org/openjdk/bench/jdk/incubator/vector/ColumnFilterBenchmark.java

src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp

src/hotspot/cpu/x86/stubGenerator_x86_64.cpp

sviswa7

Nice work! The patch looks good to me now.

openjdk · 2024-01-22T19:55:22Z

@jatin-bhateja This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8322768: Optimize non-subword vector compress and expand APIs for AVX2 target.

Reviewed-by: epeter, sviswanathan

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 26 new commits pushed to the master branch:

3059c3b: 8324242: Avoid null check for OopHandle::ptr_raw()
929af9e: 8307788: vmTestbase/gc/gctests/LargeObjects/large003/TestDescription.java timed out
e7fdac9: 8324280: RISC-V: Incorrect implementation in VM_Version::parse_satp_mode
3d32c46: 6503196: API doc for DecimalFormat::getMaximumIntegerDigits is unclear
2d5cb97: 8324647: Invalid test group of lib-test after JDK-8323515
6212264: 8320692: Null icon returned for .exe without custom icon
32eb529: 8324553: Shenandoah: Move periodic tasks closer to their collaborators
c702dca: 8323815: Source launcher should find classes with $ in names
c432dc0: 8322149: ConcurrentHashMap smarter presizing for copy constructor and putAll
fb822e4: 8324514: ClassLoaderData::print_on should print address of class loader
... and 16 more: https://git.openjdk.org/jdk/compare/5acd37fa9625fd53dd705ba536a8ae6c7e54ed7d...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

eme64

Looks good, except for one detail ;)

src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp

eme64

Ok, I'll just run the testing again, and then I will approve this :)

jatin-bhateja · 2024-01-25T03:11:38Z

Hi @eme64 , let us know test results.

Best Regards

eme64

Testing passed, looks good now :)
Nice progress, the code now is simpler and much more understandable!

jatin-bhateja · 2024-01-25T10:06:15Z

Thanks @eme64 and @sviswa7 for your comments.

jatin-bhateja · 2024-01-25T10:06:26Z

/integrate

openjdk · 2024-01-25T10:07:52Z

Going to push as commit 6d36eb7.
Since your change was applied there have been 27 commits pushed to the master branch:

9d1a6d1: 8323645: Remove unused internal sun.net.www.protocol.jar.URLJarFileCallBack interface
3059c3b: 8324242: Avoid null check for OopHandle::ptr_raw()
929af9e: 8307788: vmTestbase/gc/gctests/LargeObjects/large003/TestDescription.java timed out
e7fdac9: 8324280: RISC-V: Incorrect implementation in VM_Version::parse_satp_mode
3d32c46: 6503196: API doc for DecimalFormat::getMaximumIntegerDigits is unclear
2d5cb97: 8324647: Invalid test group of lib-test after JDK-8323515
6212264: 8320692: Null icon returned for .exe without custom icon
32eb529: 8324553: Shenandoah: Move periodic tasks closer to their collaborators
c702dca: 8323815: Source launcher should find classes with $ in names
c432dc0: 8322149: ConcurrentHashMap smarter presizing for copy constructor and putAll
... and 17 more: https://git.openjdk.org/jdk/compare/5acd37fa9625fd53dd705ba536a8ae6c7e54ed7d...master

Your commit was automatically rebased without conflicts.

openjdk · 2024-01-25T10:07:58Z

@jatin-bhateja Pushed as commit 6d36eb7.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Jatin Bhateja added 2 commits January 4, 2024 10:39

8322768: Optimize non-subword vector compress and expand APIs for AVX…

3f2b610

…2 target.

Updating copyright year of modified files.

6bd9b0a

openjdk bot added the rfr Pull request is ready for review label Jan 4, 2024

openjdk bot added hotspot-compiler hotspot-compiler-dev@openjdk.org core-libs core-libs-dev@openjdk.org labels Jan 4, 2024

eme64 suggested changes Jan 4, 2024

View reviewed changes

Review comments resolutions.

ea0aa0b

eme64 suggested changes Jan 5, 2024

View reviewed changes

Review suggestions incorporated.

257a635

eme64 suggested changes Jan 8, 2024

View reviewed changes

src/hotspot/cpu/x86/stubGenerator_x86_64.cpp Show resolved Hide resolved

test/micro/org/openjdk/bench/jdk/incubator/vector/ColumnFilterBenchmark.java Show resolved Hide resolved

eme64 suggested changes Jan 9, 2024

View reviewed changes

Using emulated variable blend E-Core optimized instruction.

c3f1c50

turbanoff reviewed Jan 15, 2024

View reviewed changes

test/micro/org/openjdk/bench/jdk/incubator/vector/ColumnFilterBenchmark.java Outdated Show resolved Hide resolved

eme64 reviewed Jan 15, 2024

View reviewed changes

src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp Outdated Show resolved Hide resolved

Jatin Bhateja added 2 commits January 18, 2024 22:15

Space fixup

3ed6b8b

Modified code comment for clarity.

b2190fc

sviswa7 reviewed Jan 20, 2024

View reviewed changes

src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp Outdated Show resolved Hide resolved

sviswa7 reviewed Jan 20, 2024

View reviewed changes

src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp Outdated Show resolved Hide resolved

sviswa7 reviewed Jan 20, 2024

View reviewed changes

src/hotspot/cpu/x86/stubGenerator_x86_64.cpp Outdated Show resolved Hide resolved

Review comments resolution

cd91230

sviswa7 approved these changes Jan 22, 2024

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Jan 22, 2024

eme64 suggested changes Jan 23, 2024

View reviewed changes

src/hotspot/cpu/x86/c2_MacroAssembler_x86.cpp Outdated Show resolved Hide resolved

Jatin Bhateja added 2 commits January 23, 2024 17:01

Modifying comments.

f65a5a9

Merge branch 'master' of http://github.com/openjdk/jdk into JDK-8322768

83e4065

eme64 reviewed Jan 23, 2024

View reviewed changes

eme64 approved these changes Jan 25, 2024

View reviewed changes

openjdk bot added the integrated Pull request has been integrated label Jan 25, 2024

openjdk bot closed this Jan 25, 2024

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Jan 25, 2024

jatin-bhateja deleted the JDK-8322768 branch August 1, 2024 08:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8322768: Optimize non-subword vector compress and expand APIs for AVX2 target. #17261

8322768: Optimize non-subword vector compress and expand APIs for AVX2 target. #17261

jatin-bhateja commented Jan 4, 2024 •

edited by openjdk bot

Loading

bridgekeeper bot commented Jan 4, 2024

openjdk bot commented Jan 4, 2024

mlbridge bot commented Jan 4, 2024 •

edited

Loading

eme64 left a comment

eme64 left a comment

jatin-bhateja commented Jan 8, 2024

merykitty commented Jan 8, 2024

eme64 left a comment

jatin-bhateja commented Jan 9, 2024 •

edited

Loading

eme64 left a comment

jatin-bhateja commented Jan 10, 2024 •

edited

Loading

sviswa7 left a comment

openjdk bot commented Jan 22, 2024 •

edited

Loading

eme64 left a comment

eme64 left a comment

jatin-bhateja commented Jan 25, 2024

eme64 left a comment

jatin-bhateja commented Jan 25, 2024

jatin-bhateja commented Jan 25, 2024

openjdk bot commented Jan 25, 2024

openjdk bot commented Jan 25, 2024

8322768: Optimize non-subword vector compress and expand APIs for AVX2 target. #17261

8322768: Optimize non-subword vector compress and expand APIs for AVX2 target. #17261

Conversation

jatin-bhateja commented Jan 4, 2024 • edited by openjdk bot Loading

Progress

Issue

Reviewers

Reviewing

Webrev

bridgekeeper bot commented Jan 4, 2024

openjdk bot commented Jan 4, 2024

mlbridge bot commented Jan 4, 2024 • edited Loading

Webrevs

eme64 left a comment

Choose a reason for hiding this comment

eme64 left a comment

Choose a reason for hiding this comment

jatin-bhateja commented Jan 8, 2024

merykitty commented Jan 8, 2024

eme64 left a comment

Choose a reason for hiding this comment

jatin-bhateja commented Jan 9, 2024 • edited Loading

eme64 left a comment

Choose a reason for hiding this comment

jatin-bhateja commented Jan 10, 2024 • edited Loading

sviswa7 left a comment

Choose a reason for hiding this comment

openjdk bot commented Jan 22, 2024 • edited Loading

eme64 left a comment

Choose a reason for hiding this comment

eme64 left a comment

Choose a reason for hiding this comment

jatin-bhateja commented Jan 25, 2024

eme64 left a comment

Choose a reason for hiding this comment

jatin-bhateja commented Jan 25, 2024

jatin-bhateja commented Jan 25, 2024

openjdk bot commented Jan 25, 2024

openjdk bot commented Jan 25, 2024

jatin-bhateja commented Jan 4, 2024 •

edited by openjdk bot

Loading

mlbridge bot commented Jan 4, 2024 •

edited

Loading

jatin-bhateja commented Jan 9, 2024 •

edited

Loading

jatin-bhateja commented Jan 10, 2024 •

edited

Loading

openjdk bot commented Jan 22, 2024 •

edited

Loading