8273949: Intrinsic creation for VectorMask.toLong operation. #126

jatin-bhateja · 2021-09-17T19:50:37Z

Summary of changes:

Intrinsification of VectorMask.toLong() API.
Supports inline expansion for both AVX512 and non-AVX512 targets.
Used toLong() API to optimize existing Java API implementation of VectorMask.laneIsSet() operation.

Following performance number are generated using JMH benchmark modification included with the patch.

System: Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz (Cascade Lake Server 28C 2S)

Benchmark	VECSIZE	Baseline Score (ops/ms)	With Opt (ops/ms)	Gain Ratio
MaskQueryOperationsBenchmark.testToLongByte	128	90451.424	346941.379	3.835665196
MaskQueryOperationsBenchmark.testToLongByte	256	63127.764	338331.425	5.359471072
MaskQueryOperationsBenchmark.testToLongByte	512	40543.264	313836.333	7.740776199
MaskQueryOperationsBenchmark.testToLongLong	128	171989.714	152872.758	0.88884826
MaskQueryOperationsBenchmark.testToLongLong	256	164702.273	324794.578	1.972010295
MaskQueryOperationsBenchmark.testToLongLong	512	122667.916	318060.096	2.59285481
MaskQueryOperationsBenchmark.testToLongShort	128	122656.408	346691.082	2.826522378
MaskQueryOperationsBenchmark.testToLongShort	256	96838.555	360909.28	3.726917239
MaskQueryOperationsBenchmark.testToLongShort	512	63119.009	313075.159	4.960077225
MaskQueryOperationsBenchmark.testToLonglong	128	180855.623	324620.433	1.794914792
MaskQueryOperationsBenchmark.testToLonglong	256	122705.631	324315.916	2.643040204
MaskQueryOperationsBenchmark.testToLonglong	512	90396.687	324318.095	3.58772103

Progress

Change must not contain extraneous whitespace
Change must be properly reviewed

Issue

JDK-8273949: Intrinsic creation for VectorMask.toLong operation.

Reviewers

Paul Sandoz (@PaulSandoz - Committer) ⚠️ Review applies to 6b38d05
Sandhya Viswanathan (@sviswa7 - Committer) ⚠️ Review applies to f1816ce
Eric Liu (@theRealELiu - Committer) ⚠️ Review applies to f1816ce

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/panama-vector pull/126/head:pull/126
$ git checkout pull/126

Update a local copy of the PR:
$ git checkout pull/126
$ git pull https://git.openjdk.java.net/panama-vector pull/126/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 126

View PR using the GUI difftool:
$ git pr show -t 126

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/panama-vector/pull/126.diff

bridgekeeper · 2021-09-17T19:50:58Z

👋 Welcome back jbhateja! A progress list of the required criteria for merging this PR into vectorIntrinsics+mask will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

mlbridge · 2021-09-17T19:56:21Z

Webrevs

PaulSandoz · 2021-09-17T20:32:04Z

src/jdk.incubator.vector/share/classes/jdk/incubator/vector/AbstractMask.java

-        for (int i = 0; i < bits.length; i++) {
-            res = bits[i] ? res | set : res;
-            set = set << 1;
+        if (length() <= Long.SIZE) {


We need to check that i > 0 && i < length() and if not throw IllegalArgumentException.

Currently the behavior is unspecified, can you please add the following to the documentation of VectorMask.laneIsSet. (I will handle any CSR related tasks.)

* @throws IllegalArgumentException if the index is is out of range * ({@code < 0 || >= length()})

PaulSandoz · 2021-09-17T20:32:04Z

src/jdk.incubator.vector/share/classes/jdk/incubator/vector/Byte128Vector.java

+            return (int) VectorSupport.maskReductionCoerced(VECTOR_OP_MASK_TRUECOUNT, Byte128Mask.class, byte.class, VLENGTH, this,
+                                                      (m) -> (long)trueCountHelper(((Byte128Mask)m).getBits()));


Suggested change

return (int) VectorSupport.maskReductionCoerced(VECTOR_OP_MASK_TRUECOUNT, Byte128Mask.class, byte.class, VLENGTH, this,

(m) -> (long)trueCountHelper(((Byte128Mask)m).getBits()));

return (int) VectorSupport.maskReductionCoerced(VECTOR_OP_MASK_TRUECOUNT, Byte128Mask.class, byte.class, VLENGTH, this,

(m) -> trueCountHelper(m.getBits()));

We don't need the cast to long for the return from the lambda expression. And I think the cast of the mask is redundant too. Same applies to the other three methods.

PaulSandoz · 2021-09-20T15:41:05Z

src/jdk.incubator.vector/share/classes/jdk/incubator/vector/AbstractMask.java

+        if (i < 0 || i >= length()) {
+            throw new IllegalArgumentException("Index " + i + " must be zero or positive, and less than " + length());
+        }


Apologies i should have previously said use Objects.checkIndex(i, length()). In fact lets call length once:

int length = length(); Objects.checkIndex(i, length); if (length <= Long.SIZE) { ...

Separately i wonder if the second bounds check should be if i <= Long.SIZE, since the first bounds check should dominate in many cases?

(I know there are other areas where we should Objects.checkIndex e.g. *MaxVector.lane/withLane but we can fix those later.)

With length <= Long.SIZE there is a better possibility of optimizing out else part and thus only fast path gets generated. length is final static values and hence in most of the cases due to in-lining this constant may get propagated.

PaulSandoz

Java changes are good.

openjdk · 2021-09-20T19:38:20Z

@jatin-bhateja This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8273949: Intrinsic creation for VectorMask.toLong operation.

Reviewed-by: psandoz, sviswanathan, eliu

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 1 new commit pushed to the vectorIntrinsics+mask branch:

b44662a: 8273264: AArch64: [vector] Add missing rules for VectorMaskCast

Please see this link for an up-to-date comparison between the source branch of this pull request and the vectorIntrinsics+mask branch.
As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the vectorIntrinsics+mask branch, type /integrate in a new comment.

sviswa7 · 2021-09-21T21:46:46Z

src/hotspot/cpu/x86/x86.ad

-  match(Set dst (VectorMaskTrueCount mask));
-  effect(TEMP_DEF dst, TEMP tmp, TEMP ktmp, TEMP xtmp);
-  format %{ "vector_truecount_evex $mask \t! vector mask true count" %}
+instruct vmask_tolong_evex(rRegL dst, kReg mask) %{


The instructs with rRegL should be in ifdef _LP64.

Its already covered under _LP64

sviswa7 · 2021-09-21T21:46:46Z

src/hotspot/cpu/x86/x86.ad

-  match(Set dst (VectorMaskLastTrue mask));
-  effect(TEMP_DEF dst, TEMP tmp, TEMP ktmp, TEMP xtmp, KILL cr);
-  format %{ "vector_mask_first_or_last_true_evex $mask \t! vector first/last true location" %}
+instruct vmask_truecount_evex(rRegI dst, kReg mask, rRegL tmp) %{


This should have flags register as popcnt instruction affects flags.

sviswa7 · 2021-09-21T21:46:46Z

src/hotspot/cpu/x86/x86.ad

 instruct vmask_truecount_avx(rRegI dst, vec mask, rRegL tmp, vec xtmp, vec xtmp1) %{
-  predicate(!VM_Version::supports_avx512vlbw());
+  predicate(n->in(1)->bottom_type()->isa_vectmask() == NULL);


This should also include flags register.

sviswa7 · 2021-09-21T21:46:46Z

src/hotspot/share/opto/vectornode.hpp

+class VectorMaskToLongNode : public VectorMaskOpNode {
+ public:
+  VectorMaskToLongNode(Node* mask, const Type* ty):
+    VectorMaskOpNode(mask, ty, Op_VectorMaskLastTrue) {}


Shouldn't this be Op_VectorMaskToLong?

sviswa7 · 2021-09-21T21:46:46Z

src/hotspot/share/prims/vectorSupport.cpp

+        case T_LONG:  // fall-through
+        case T_FLOAT: // fall-through
+        case T_DOUBLE: return Op_VectorMaskToLong;
+        default: fatal("MASK_TRUECOUNT: %s", type2name(bt));


MASK_TOLONG here? Also break is missing.

This is consistent with other case statements.

default: fatal("MASK_TRUECOUNT: %s", type2name(bt));
should refer to MASK_TOLONG.

jatin-bhateja · 2021-09-22T04:25:44Z

@nsjian , kindly check the IR change, it may need some change in AARCH64 backend since now VectorStoreMask is not being inserted while connecting mask generating node to mask operation node. This saves redundant store mask operation if target supports predicate registers.

nsjian · 2021-09-22T10:10:26Z

@nsjian , kindly check the IR change, it may need some change in AARCH64 backend since now VectorStoreMask is not being inserted while connecting mask generating node to mask operation node. This saves redundant store mask operation if target supports predicate registers.

Hi @jatin-bhateja , thanks for the heads-up. The VectorMaskOp connection change looks reasonable. We will update SVE backend later.

sviswa7

Only one comment on cut and paste mistake in error message, Please fix that. Other than that the patch looks good. No need for re-rereview.

theRealELiu

LGTM.

Just the comment is mismatched.

// <E, M>
// long maskReductionCoerced(int oper, Class<? extends M> maskClass, Class<?> elemClass,
//                          int length, M m, VectorMaskOp<M> defaultImpl)
bool LibraryCallKit::inline_vector_mask_operation() {

jatin-bhateja · 2021-09-23T03:11:20Z

/integrate

openjdk · 2021-09-23T03:12:54Z

Going to push as commit 0e7348d.
Since your change was applied there has been 1 commit pushed to the vectorIntrinsics+mask branch:

b44662a: 8273264: AArch64: [vector] Add missing rules for VectorMaskCast

Your commit was automatically rebased without conflicts.

openjdk · 2021-09-23T03:13:09Z

@jatin-bhateja Pushed as commit 0e7348d.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

DamonFool · 2021-09-26T15:48:06Z

Build failure on MacOS after this PR: #133 .

DamonFool · 2021-09-30T06:01:14Z

Incorrect computation after this pr.
Please see: #138 .
Thanks.

8273949: Intrinsic creation for VectorMask.toLong operation.

c911034

openjdk bot added the rfr label Sep 17, 2021

PaulSandoz reviewed Sep 17, 2021

View changes

Jatin Bhateja added 2 commits Sep 19, 2021

8273949: Review comments resolution

f7e689e

8273949: Review comments resolution.

6b38d05

PaulSandoz reviewed Sep 20, 2021

View changes

PaulSandoz approved these changes Sep 20, 2021

View changes

openjdk bot added the ready label Sep 20, 2021

8273949: Review comments resolution.

f1816ce

sviswa7 reviewed Sep 21, 2021

View changes

jatin-bhateja closed this Sep 22, 2021

jatin-bhateja deleted the JDK-8273949 branch Sep 22, 2021

jatin-bhateja restored the JDK-8273949 branch Sep 22, 2021

jatin-bhateja reopened this Sep 22, 2021

Jatin Bhateja added 2 commits Sep 22, 2021

8273949: Resolving synthetic comment.

daedf3a

8273949: Resolving synthetic comment.

f115417

sviswa7 approved these changes Sep 22, 2021

View changes

theRealELiu approved these changes Sep 23, 2021

View changes

openjdk bot closed this Sep 23, 2021

openjdk bot added integrated and removed ready rfr labels Sep 23, 2021

8273949: Intrinsic creation for VectorMask.toLong operation. #126

8273949: Intrinsic creation for VectorMask.toLong operation. #126

jatin-bhateja commented Sep 17, 2021 •

edited by openjdk[bot] bot

bridgekeeper bot commented Sep 17, 2021

mlbridge bot commented Sep 17, 2021 •

edited

PaulSandoz Sep 17, 2021

PaulSandoz Sep 17, 2021

PaulSandoz Sep 20, 2021

jatin-bhateja Sep 20, 2021

PaulSandoz left a comment

openjdk bot commented Sep 20, 2021 •

edited

sviswa7 Sep 21, 2021

jatin-bhateja Sep 22, 2021

sviswa7 Sep 21, 2021

sviswa7 Sep 21, 2021

sviswa7 Sep 21, 2021

sviswa7 Sep 21, 2021

jatin-bhateja Sep 22, 2021

sviswa7 Sep 22, 2021

jatin-bhateja commented Sep 22, 2021

nsjian commented Sep 22, 2021

sviswa7 left a comment

theRealELiu left a comment

jatin-bhateja commented Sep 23, 2021

openjdk bot commented Sep 23, 2021

openjdk bot commented Sep 23, 2021

DamonFool commented Sep 26, 2021

DamonFool commented Sep 30, 2021

		return (int) VectorSupport.maskReductionCoerced(VECTOR_OP_MASK_TRUECOUNT, Byte128Mask.class, byte.class, VLENGTH, this,
		(m) -> (long)trueCountHelper(((Byte128Mask)m).getBits()));

8273949: Intrinsic creation for VectorMask.toLong operation. #126

8273949: Intrinsic creation for VectorMask.toLong operation. #126

Conversation

jatin-bhateja commented Sep 17, 2021 • edited by openjdk[bot] bot

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Sep 17, 2021

mlbridge bot commented Sep 17, 2021 • edited

Webrevs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PaulSandoz left a comment

openjdk bot commented Sep 20, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jatin-bhateja commented Sep 22, 2021

nsjian commented Sep 22, 2021

sviswa7 left a comment

theRealELiu left a comment

jatin-bhateja commented Sep 23, 2021

openjdk bot commented Sep 23, 2021

openjdk bot commented Sep 23, 2021

DamonFool commented Sep 26, 2021

DamonFool commented Sep 30, 2021

jatin-bhateja commented Sep 17, 2021 •

edited by openjdk[bot] bot

mlbridge bot commented Sep 17, 2021 •

edited

openjdk bot commented Sep 20, 2021 •

edited