[AMDGPU][SIMemoryLegalizer] Fix order of GL0/1_INV on GFX10/11 #81450
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes SWDEV-443292
|
@llvm/pr-subscribers-llvm-globalisel @llvm/pr-subscribers-backend-amdgpu Author: Pierre van Houtryve (Pierre-vh) ChangesFixes SWDEV-443292 Full diff: https://github.com/llvm/llvm-project/pull/81450.diff 2 Files Affected:
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index ebc7fda804207a..7c78c0907be5e1 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -12139,8 +12139,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
before invalidating
the caches.
- 3. buffer_gl0_inv;
- buffer_gl1_inv
+ 3. buffer_gl1_inv;
+ buffer_gl0_inv
- Must happen before
any following
@@ -12169,8 +12169,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
before invalidating
the caches.
- 3. buffer_gl0_inv;
- buffer_gl1_inv
+ 3. buffer_gl1_inv;
+ buffer_gl0_inv
- Must happen before
any following
@@ -12276,8 +12276,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
invalidating the
caches.
- 3. buffer_gl0_inv;
- buffer_gl1_inv
+ 3. buffer_gl1_inv;
+ buffer_gl0_inv
- Must happen before
any following
@@ -12307,8 +12307,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
invalidating the
caches.
- 3. buffer_gl0_inv;
- buffer_gl1_inv
+ 3. buffer_gl1_inv;
+ buffer_gl0_inv
- Must happen before
any following
@@ -12503,8 +12503,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
the
fence-paired-atomic.
- 2. buffer_gl0_inv;
- buffer_gl1_inv
+ 2. buffer_gl1_inv;
+ buffer_gl0_inv
- Must happen before any
following global/generic
@@ -13217,8 +13217,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
invalidating the
caches.
- 4. buffer_gl0_inv;
- buffer_gl1_inv
+ 4. buffer_gl1_inv;
+ buffer_gl0_inv
- Must happen before
any following
@@ -13292,8 +13292,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
invalidating the
caches.
- 4. buffer_gl0_inv;
- buffer_gl1_inv
+ 4. buffer_gl1_inv;
+ buffer_gl0_inv
- Must happen before
any following
@@ -13520,8 +13520,8 @@ table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.
requirements of
release.
- 2. buffer_gl0_inv;
- buffer_gl1_inv
+ 2. buffer_gl1_inv;
+ buffer_gl0_inv
- Must happen before
any following
diff --git a/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp b/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp
index 84b9330ef9633e..f62e808b33e42b 100644
--- a/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp
+++ b/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp
@@ -2030,8 +2030,11 @@ bool SIGfx10CacheControl::insertAcquire(MachineBasicBlock::iterator &MI,
switch (Scope) {
case SIAtomicScope::SYSTEM:
case SIAtomicScope::AGENT:
- BuildMI(MBB, MI, DL, TII->get(AMDGPU::BUFFER_GL0_INV));
+ // The order of invalidates matter here. We must invalidate "outer in"
+ // so L1 -> L0 to avoid L0 pulling in stale data from L1 when it is
+ // invalidated.
BuildMI(MBB, MI, DL, TII->get(AMDGPU::BUFFER_GL1_INV));
+ BuildMI(MBB, MI, DL, TII->get(AMDGPU::BUFFER_GL0_INV));
Changed = true;
break;
case SIAtomicScope::WORKGROUP:
|
arsenm
approved these changes
Feb 12, 2024
t-tye
approved these changes
Feb 12, 2024
rocm-ci
pushed a commit
to ROCm/llvm-project
that referenced
this pull request
Apr 16, 2024
…81450) Fixes SWDEV-443292 Change-Id: I2eeb68b9d82a560683a96efb0207e82a93de901a
searlmc1
pushed a commit
to ROCm/llvm-project
that referenced
this pull request
Apr 23, 2024
…81450) Fixes SWDEV-443292 Change-Id: I2eeb68b9d82a560683a96efb0207e82a93de901a
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes SWDEV-443292