-
Notifications
You must be signed in to change notification settings - Fork 6.1k
8347018: C2: Insertion of Assertion Predicates ignores the effects of PhaseIdealLoop::clone_up_backedge_goo() #23071
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…: The home of a memory writer must also be its earliest placement
|
👋 Welcome back chagedorn! A progress list of the required criteria for merging this PR into |
|
@chhagedorn This change now passes all automated pre-integration checks. ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details. After integration, the commit message for the final commit will be: You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed. At the time when this comment was updated there had been 14 new commits pushed to the
As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details. ➡️ To integrate this PR with the above commit message to the |
|
@chhagedorn The following label will be automatically applied to this pull request:
When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command. |
| // Nodes inside the loop may be control dependent on a predicate | ||
| // that was moved before the preloop. If the back branch of the main | ||
| // or post loops becomes dead, those nodes won't be dependent on the | ||
| // test that guards that loop nest anymore which could lead to an | ||
| // incorrect array access because it executes independently of the | ||
| // test that was guarding the loop nest. We add a special CastII on | ||
| // the if branch that enters the loop, between the input induction | ||
| // variable value and the induction variable Phi to preserve correct | ||
| // dependencies. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Noticed that this comment block should have been removed earlier with JDK-8334724 which removed the cast node. I squeezed this in here - probably not worth a separate task.
Webrevs
|
| // clone from 'node' (i.e. _old_new entry is non-null). Then we know that 'node' belongs to the original loop body. | ||
| // Additionally check if a node was cloned after the pre loop was created. This indicates that it was created by | ||
| // PhaseIdealLoop::clone_up_backedge_goo(). These nodes should also be pinned at the main loop entry. | ||
| bool check(Node* node) const override { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can it be more meaningful method's name? And I did not find where it is used.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since the interface name is NodeInLoopBody, I went with NodeInLoopBody::check(). But could also rename it to check_node_in_loop_body() for better readability. Pushed an update.
It is used here:
jdk/src/hotspot/share/opto/predicates.cpp
Line 157 in 06ff4c1
| if (!output->is_CFG() && data_in_loop_body.check(output)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was confused because it was not use in changes but I now see that it is virtual method.
|
I agree with deferring it to JDK 25 and backport into JDK 24 update release after some time. |
Thanks for your feedback Vladimir! Then let's target this to JDK 25. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The fix seems fine to me.
|
Thanks Vladimir for your review! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should probably adjust the title of the PR / Bug.
Before I review, I have some understanding questions:
Why is it a problem that the control the store dominates the control of the load? Could such a constellation not happen in other circumstances too?
Could we not come up with some case where the store has its control moved up above the control of the load somehow - or is that generally not supposed to happen?
I suppose I am wondering why can GCM not just handle such cases?
...jtreg/compiler/predicates/assertion/TestLoadPinnedAboveAssertionPredicatesAndUsingStore.java
Outdated
Show resolved
Hide resolved
|
Thanks Emanuel for your questions, let's first make a step back. This patch essentially fixes two issues:
The assert was introduced around 3 years ago and never failed before. It sounds like a condition that is always met - until discovered now. But when closer looking at the the original intention of The comment at the failing assert suggests that this invariant is required such that I've tried to play around to create such a situation without Assertion Predicates but could not find a case how we could violate the assert - but of course, that does not prove anything.
I think that's also a possible option. I've decided to do the fix at the Assertion Predicates creation point for the following reasons:
For these reasons, I've chosen the current fix idea without trying to change the current behavior of GCM. What are your thoughts about that?
Good point, even though it's addressing two problems, I could just make it more obvious what the fixes are about. Changed! |
…dAboveAssertionPredicatesAndUsingStore.java Co-authored-by: Emanuel Peter <emanuel.peter@oracle.com>
That sounds like a good argument. I will review the code now ;) Thanks for all the extra explanations 😊 |
Sure, you're welcome! :-) Sounds good, thanks a lot! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks reasonable :)
| const NodeInMainLoopBody node_in_original_loop_body(first_node_index_in_pre_loop_body, | ||
| last_node_index_in_pre_loop_body, old_new); | ||
| create_assertion_predicates_at_main_or_post_loop(pre_loop_head, main_loop_head, node_in_original_loop_body, true); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| const NodeInMainLoopBody node_in_original_loop_body(first_node_index_in_pre_loop_body, | |
| last_node_index_in_pre_loop_body, old_new); | |
| create_assertion_predicates_at_main_or_post_loop(pre_loop_head, main_loop_head, node_in_original_loop_body, true); | |
| const NodeInMainLoopBody node_in_main_loop_body(first_node_index_in_pre_loop_body, | |
| last_node_index_in_pre_loop_body, old_new); | |
| create_assertion_predicates_at_main_or_post_loop(pre_loop_head, main_loop_head, node_in_main_loop_body, true); |
Would that make sense now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Definitely! Updated.
| // Rewire any control dependent nodes on the old target loop entry before adding Assertion Predicate related nodes. | ||
| // These have been added by PhaseIdealLoop::clone_up_backedge_goo() and assume to be ending up at the target loop entry | ||
| // which is no longer the case when adding additional Assertion Predicates. Fix this by rewiring these nodes to the new | ||
| // target loop entry which corresponds to the tail of the last Assertion Predicate before the target loop. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this introduce any circular dependency? I.e. that the Assertion Predicates have dependencies on the control dependencies that we just moved down? -> You could add a comment here why that is not possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think that is possible. I've added an additional comment.
| bool check_node_in_loop_body(Node* node) const override { | ||
| if (node->_idx < _first_node_index_in_cloned_loop_body) { | ||
| Node* cloned_node = _old_new[node->_idx]; | ||
| return cloned_node != nullptr && cloned_node->_idx >= _first_node_index_in_cloned_loop_body; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In what case would we return false here? Can you add a comment?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added!
| return cloned_node != nullptr && cloned_node->_idx >= _first_node_index_in_pre_loop_body; | ||
| } | ||
| // Created in PhaseIdealLoop::clone_up_backedge_goo()? | ||
| return node->_idx > _last_node_index_in_pre_loop_body; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks reasonable now, but I'm a little afraid that this could be fragile in the future.
Hmm, not sure what to do.
You put a lower bound here. Could we also have an upper bound? Just in case somebody decides to add more nodes in the meantime ... and then you would return true here as well, which would probably be wrong?
Maybe there could also be some assert, but I'm not sure.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a good input. Checking the origin from clones is difficult. But we could do it over storing node indices. I've added some assertion code with checking the last node index used in clone_up_backedge_goo(). That feels more robust. Let me know what you think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your review! I've addressed your comments.
| const NodeInMainLoopBody node_in_original_loop_body(first_node_index_in_pre_loop_body, | ||
| last_node_index_in_pre_loop_body, old_new); | ||
| create_assertion_predicates_at_main_or_post_loop(pre_loop_head, main_loop_head, node_in_original_loop_body, true); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Definitely! Updated.
| // Rewire any control dependent nodes on the old target loop entry before adding Assertion Predicate related nodes. | ||
| // These have been added by PhaseIdealLoop::clone_up_backedge_goo() and assume to be ending up at the target loop entry | ||
| // which is no longer the case when adding additional Assertion Predicates. Fix this by rewiring these nodes to the new | ||
| // target loop entry which corresponds to the tail of the last Assertion Predicate before the target loop. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think that is possible. I've added an additional comment.
| bool check_node_in_loop_body(Node* node) const override { | ||
| if (node->_idx < _first_node_index_in_cloned_loop_body) { | ||
| Node* cloned_node = _old_new[node->_idx]; | ||
| return cloned_node != nullptr && cloned_node->_idx >= _first_node_index_in_cloned_loop_body; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added!
| return cloned_node != nullptr && cloned_node->_idx >= _first_node_index_in_pre_loop_body; | ||
| } | ||
| // Created in PhaseIdealLoop::clone_up_backedge_goo()? | ||
| return node->_idx > _last_node_index_in_pre_loop_body; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a good input. Checking the origin from clones is difficult. But we could do it over storing node indices. I've added some assertion code with checking the last node index used in clone_up_backedge_goo(). That feels more robust. Let me know what you think.
| const uint last_node_index_in_pre_loop_body = Compile::current()->unique() - 1; | ||
| assert(post_head->in(1)->is_IfProj(), "must be zero-trip guard If node projection of the post loop"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I somehow moved this down after testing - I cannot remember why. It should be further up. Fixed here as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good now, thanks for the updates!
|
Thanks Emanuel for your review! /integrate |
|
Going to push as commit 8a83dc2.
Your commit was automatically rebased without conflicts. |
|
@chhagedorn Pushed as commit 8a83dc2. 💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored. |
Failing Assert
The failing assert in
PhaseCFG::schedule_late()checks the following:jdk/src/hotspot/share/opto/gcm.cpp
Lines 1431 to 1438 in 55c6904
In the test case, this is violated for

87 storeI:The early block
earlyfor87 storeIis bound by115 loadIpinned at161 Regionwhich is dominated by the control input146 Regionof87 storeI. This lets the assert fail.How Did
115 loadIEnd up Being Pinned below87 storeI?Before Pre/Main/Post Loop Creation
Before the creation of pre/main/post loops, we have the following graph:
Everything looks fine: The control input of
312 StoreI(which is eventually cloned and becomes87 storeIin the Mach graph) corresponds to the early placement of the store.415 LoadIwas hoisted out of the loop during Loop Predication and is pinned above at a Template Assertion Predicate.Pre/Main/Post Loop Creation
Post Loop Body Creation
During the creation of pre/main/post loops, we clone the main loop body for the post loop body:
We notice that
312 StoreIis pinned on the main loop backedge. When finishing the last iteration from the main loop and possibly continuing in the post loop, we need to feed everything on the loop backedge of the main loop to the post loop. However, the pinned nodes on the main loop backedge cannot float. Therefore, we need to create new copies of these pinned nodes withPhaseIdealLoop::clone_up_backedge_goo().The pins are updated to the entry of the post loop. All inputs into these pinned nodes that have their current control (fetched with
get_ctrl()) on the main loop backedge as well are also cloned but keep their control inputs (if any) if it's not the loop backedge.In our example, this applies to
453 StoreI->479 StoreI, and some inputs recursively (454 AddI->482 AddI,481 LoadI->541 Load):Still, all looks fine. Notice that the clone
481 LoadIof455 LoadIis currently still pinned at the same Template Assertion.Assertion Predicate Creation
In the next step, we create new Assertion Predicates at the post loop and rewire any data nodes control dependent on Assertion Predicates down to the post loop - including the new
481 LoadIfromPhaseIdealLoop::clone_up_backedge_goo():This creates the graph shape with which we are then later failing during scheduling in the backend: The control input of
479 StoreIfurther up in the graph as the actual early block limited by481 LoadIpinned at493 IfTrue.Same Problem with
clone_up_backedge_goo()for Main Loop?The very same problem could theoretically also be observed for the main loop when creating the pre loop. But it is not due to how we implemented the rewiring of data nodes when creating new Assertion Predicates:
After the pre loop is created, the old Assertion Predicates are above the pre loop and actually need to be established at the main loop. Therefore, all data nodes control dependent on Assertion Predicates and belonging to the main loop need to be rewired. In our test case, this is
415 LoadI(original node) and540 LoadI(cloned node byclone_up_backedge_goo()actually belonging to main loop):Check If Data Belongs to Main Loop
Since the pre loop only contains cloned nodes we do the following trick to determine if a node belongs to the main loop (implemented here):
Cloned Nodes with
clone_up_backedge_goo()Mess with "Node inside Main Loop" CheckSince the cloned nodes in
clone_up_backedge_goo()are originally from pre loop nodes, our check will fail and we do not rewire these nodes, even though they belong to the main loop:Applied to our test case, we have the following after
clone_up_backedge_goo():We can see that
540 LoadI, cloned byclone_up_backedge_goo(), is still pinned before the pre loop because we have not rewired it and thus scheduling does not fail with the assert.Even though I could not trigger a failure, I think it is an incorrect pin since the
540 LoadIbelongs to the main loop.Proposed Fix
clone_up_backedge_goo()which are pinned to the original loop entry before Assertion Predication to the new loop entry after Assertion Predicate creation. The new loop entry will be the the tail of the last Assertion Predicate (if any).clone_up_backedge_goo()correctly. I've implemented a newNodeInMainLoopBodyclass for that purpse.Why not just Add Assertion Predicates First?
This does not work straight forward because we do not know the init value before applying
clone_up_backedge_goo()which is interleaved with updating the phi nodes. I've decided to go with the proposed fix instead.Testing:
Deferring to JDK 25?
This seems to be an edge case (only found with fuzzing) and it's not entirely clear to me what the impact on product builds is. However, this is a regression in JDK 24 and should be considered to be fixed in JDK 24. But this fix became somewhat more complex to understand and implement. First applying the fix to JDK 25, letting it bake and then only considering it for an update release of JDK 25 could be a possible option I think. Opinions are welcomed.
Thanks,
Christian
Progress
Issue
Reviewers
Reviewing
Using
gitCheckout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/23071/head:pull/23071$ git checkout pull/23071Update a local copy of the PR:
$ git checkout pull/23071$ git pull https://git.openjdk.org/jdk.git pull/23071/headUsing Skara CLI tools
Checkout this PR locally:
$ git pr checkout 23071View PR using the GUI difftool:
$ git pr show -t 23071Using diff file
Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/23071.diff
Using Webrev
Link to Webrev Comment