Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

llvm-mca hits sanitizer error in cycleEnd #83775

Closed
arsenm opened this issue Mar 4, 2024 · 4 comments
Closed

llvm-mca hits sanitizer error in cycleEnd #83775

arsenm opened this issue Mar 4, 2024 · 4 comments
Assignees
@michaelmaitland @adibiagio
Labels
crash-on-valid tools:llvm-mca

Comments

@arsenm
Copy link
Contributor

arsenm commented Mar 4, 2024

f0484e0 was reverted in 31295bb due to breaking the sanitizer bot.

Reduced testcase:

# RUN: llvm-mca -mtriple=amdgcn -mcpu=gfx940 --timeline --iterations=1 --timeline-max-cycles=0 < %s | FileCheck %s

# CHECK: Iterations:        1
# CHECK: Instructions:      71
# CHECK: Total Cycles:      562
# CHECK: Total uOps:        77

# CHECK: Resources:
# CHECK: [0]   - HWBranch
# CHECK: [1]   - HWExport
# CHECK: [2]   - HWLGKM
# CHECK: [3]   - HWSALU
# CHECK: [4]   - HWVALU
# CHECK: [5]   - HWVMEM
# CHECK: [6]   - HWXDL


v_pk_mov_b32 v[0:1], v[2:3], v[4:5]
v_pk_add_f32 v[0:1], v[0:1], v[0:1]
v_pk_mul_f32 v[0:1], v[0:1], v[0:1]
v_add_co_u32 v5, s[0:1], v1, v2
v_sub_co_u32 v5, s[0:1], v1, v2
v_add_u32 v5, v1, v2
v_sub_u32 v5, v1, v2


# CHECK:     [0]    [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
# CHECK-NEXT: -      -      -      -     1.00    -      -     v_pk_mov_b32 v[0:1], v[2:3], v[4:5]
# CHECK-NEXT: -      -      -      -     1.00    -      -     v_pk_add_f32 v[0:1], v[0:1], v[0:1]
# CHECK-NEXT: -      -      -      -     1.00    -      -     v_pk_mul_f32 v[0:1], v[0:1], v[0:1]
# CHECK-NEXT: -      -      -     1.00   1.00    -      -     v_add_co_u32_e64 v5, s[0:1], v1, v2
# CHECK-NEXT: -      -      -     1.00   1.00    -      -     v_sub_co_u32_e64 v5, s[0:1], v1, v2
# CHECK-NEXT: -      -      -      -     1.00    -      -     v_add_u32_e32 v5, v1, v2
# CHECK-NEXT: -      -      -      -     1.00    -      -     v_sub_u32_e32 v5, v1, v2

=================================================================
==28215==ERROR: AddressSanitizer: heap-use-after-free on address 0x000107d0149c at pc 0x000100e8afe8 bp 0x00016f97ade0 sp 0x00016f97add8
READ of size 1 at 0x000107d0149c thread T0
    #0 0x100e8afe4 in llvm::mca::InOrderIssueStage::updateCarriedOver() InOrderIssueStage.cpp:327
    #1 0x100e8b458 in llvm::mca::InOrderIssueStage::cycleStart() InOrderIssueStage.cpp:395
    #2 0x100e7b194 in llvm::mca::Pipeline::runCycle() Pipeline.cpp:60
    #3 0x100e7a84c in llvm::mca::Pipeline::run() Pipeline.cpp:43
    #4 0x100492730 in runPipeline(llvm::mca::Pipeline&) llvm-mca.cpp:308
    #5 0x10048afe0 in main llvm-mca.cpp:750
    #6 0x185c6d0dc  (<unknown module>)

0x000107d0149c is located 540 bytes inside of 608-byte region [0x000107d01280,0x000107d014e0)
freed by thread T0 here:
    #0 0x1056e952c in wrap__ZdlPv+0x74 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x6152c)
    #1 0x100e7f89c in llvm::SmallVectorImpl<std::__1::unique_ptr<llvm::mca::Instruction, std::__1::default_delete<llvm::mca::Instruction>>>::erase(std::__1::unique_ptr<llvm::mca::Instruction, std::__1::default_delete<llvm::mca::Instruction>> const*, std::__1::unique_ptr<llvm::mca::Instruction, std::__1::default_delete<llvm::mca::Instruction>> const*) SmallVector.h:775
    #2 0x100e7f66c in llvm::mca::EntryStage::cycleEnd() EntryStage.cpp:78
    #3 0x100e7b420 in llvm::mca::Pipeline::runCycle() Pipeline.cpp:78
    #4 0x100e7a84c in llvm::mca::Pipeline::run() Pipeline.cpp:43
    #5 0x100492730 in runPipeline(llvm::mca::Pipeline&) llvm-mca.cpp:308
    #6 0x10048afe0 in main llvm-mca.cpp:750
    #7 0x185c6d0dc  (<unknown module>)

previously allocated by thread T0 here:
    #0 0x1056e90ec in wrap__Znwm+0x74 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x610ec)
    #1 0x100e7ed88 in llvm::mca::EntryStage::getNextInstruction() EntryStage.cpp:40
    #2 0x100e7b2f8 in llvm::mca::Pipeline::runCycle() Pipeline.cpp:69
    #3 0x100e7a84c in llvm::mca::Pipeline::run() Pipeline.cpp:43
    #4 0x100492730 in runPipeline(llvm::mca::Pipeline&) llvm-mca.cpp:308
    #5 0x10048afe0 in main llvm-mca.cpp:750
    #6 0x185c6d0dc  (<unknown module>)

SUMMARY: AddressSanitizer: heap-use-after-free InOrderIssueStage.cpp:327 in llvm::mca::InOrderIssueStage::updateCarriedOver()
Shadow bytes around the buggy address:
  0x000107d01200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x000107d01280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x000107d01300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x000107d01380: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x000107d01400: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x000107d01480: fd fd fd[fd]fd fd fd fd fd fd fd fd fa fa fa fa
  0x000107d01500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x000107d01580: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x000107d01600: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x000107d01680: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x000107d01700: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==28215==ABORTING

cycleEnd is erasing a subset of the Instructions vector, but that vector is later read in updateCarriedOver
@llvmbot
Copy link
Collaborator

llvmbot commented Mar 4, 2024

@llvm/issue-subscribers-tools-llvm-mca

Author: Matt Arsenault (arsenm)

f0484e0 was reverted in 31295bb due to breaking the sanitizer bot.

Reduced testcase:

# RUN: llvm-mca -mtriple=amdgcn -mcpu=gfx940 --timeline --iterations=1 --timeline-max-cycles=0 &lt; %s | FileCheck %s

# CHECK: Iterations:        1
# CHECK: Instructions:      71
# CHECK: Total Cycles:      562
# CHECK: Total uOps:        77

# CHECK: Resources:
# CHECK: [0]   - HWBranch
# CHECK: [1]   - HWExport
# CHECK: [2]   - HWLGKM
# CHECK: [3]   - HWSALU
# CHECK: [4]   - HWVALU
# CHECK: [5]   - HWVMEM
# CHECK: [6]   - HWXDL


v_pk_mov_b32 v[0:1], v[2:3], v[4:5]
v_pk_add_f32 v[0:1], v[0:1], v[0:1]
v_pk_mul_f32 v[0:1], v[0:1], v[0:1]
v_add_co_u32 v5, s[0:1], v1, v2
v_sub_co_u32 v5, s[0:1], v1, v2
v_add_u32 v5, v1, v2
v_sub_u32 v5, v1, v2


# CHECK:     [0]    [1]    [2]    [3]    [4]    [5]    [6]    Instructions:
# CHECK-NEXT: -      -      -      -     1.00    -      -     v_pk_mov_b32 v[0:1], v[2:3], v[4:5]
# CHECK-NEXT: -      -      -      -     1.00    -      -     v_pk_add_f32 v[0:1], v[0:1], v[0:1]
# CHECK-NEXT: -      -      -      -     1.00    -      -     v_pk_mul_f32 v[0:1], v[0:1], v[0:1]
# CHECK-NEXT: -      -      -     1.00   1.00    -      -     v_add_co_u32_e64 v5, s[0:1], v1, v2
# CHECK-NEXT: -      -      -     1.00   1.00    -      -     v_sub_co_u32_e64 v5, s[0:1], v1, v2
# CHECK-NEXT: -      -      -      -     1.00    -      -     v_add_u32_e32 v5, v1, v2
# CHECK-NEXT: -      -      -      -     1.00    -      -     v_sub_u32_e32 v5, v1, v2

=================================================================
==28215==ERROR: AddressSanitizer: heap-use-after-free on address 0x000107d0149c at pc 0x000100e8afe8 bp 0x00016f97ade0 sp 0x00016f97add8
READ of size 1 at 0x000107d0149c thread T0
    #<!-- -->0 0x100e8afe4 in llvm::mca::InOrderIssueStage::updateCarriedOver() InOrderIssueStage.cpp:327
    #<!-- -->1 0x100e8b458 in llvm::mca::InOrderIssueStage::cycleStart() InOrderIssueStage.cpp:395
    #<!-- -->2 0x100e7b194 in llvm::mca::Pipeline::runCycle() Pipeline.cpp:60
    #<!-- -->3 0x100e7a84c in llvm::mca::Pipeline::run() Pipeline.cpp:43
    #<!-- -->4 0x100492730 in runPipeline(llvm::mca::Pipeline&amp;) llvm-mca.cpp:308
    #<!-- -->5 0x10048afe0 in main llvm-mca.cpp:750
    #<!-- -->6 0x185c6d0dc  (&lt;unknown module&gt;)

0x000107d0149c is located 540 bytes inside of 608-byte region [0x000107d01280,0x000107d014e0)
freed by thread T0 here:
    #<!-- -->0 0x1056e952c in wrap__ZdlPv+0x74 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x6152c)
    #<!-- -->1 0x100e7f89c in llvm::SmallVectorImpl&lt;std::__1::unique_ptr&lt;llvm::mca::Instruction, std::__1::default_delete&lt;llvm::mca::Instruction&gt;&gt;&gt;::erase(std::__1::unique_ptr&lt;llvm::mca::Instruction, std::__1::default_delete&lt;llvm::mca::Instruction&gt;&gt; const*, std::__1::unique_ptr&lt;llvm::mca::Instruction, std::__1::default_delete&lt;llvm::mca::Instruction&gt;&gt; const*) SmallVector.h:775
    #<!-- -->2 0x100e7f66c in llvm::mca::EntryStage::cycleEnd() EntryStage.cpp:78
    #<!-- -->3 0x100e7b420 in llvm::mca::Pipeline::runCycle() Pipeline.cpp:78
    #<!-- -->4 0x100e7a84c in llvm::mca::Pipeline::run() Pipeline.cpp:43
    #<!-- -->5 0x100492730 in runPipeline(llvm::mca::Pipeline&amp;) llvm-mca.cpp:308
    #<!-- -->6 0x10048afe0 in main llvm-mca.cpp:750
    #<!-- -->7 0x185c6d0dc  (&lt;unknown module&gt;)

previously allocated by thread T0 here:
    #<!-- -->0 0x1056e90ec in wrap__Znwm+0x74 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x610ec)
    #<!-- -->1 0x100e7ed88 in llvm::mca::EntryStage::getNextInstruction() EntryStage.cpp:40
    #<!-- -->2 0x100e7b2f8 in llvm::mca::Pipeline::runCycle() Pipeline.cpp:69
    #<!-- -->3 0x100e7a84c in llvm::mca::Pipeline::run() Pipeline.cpp:43
    #<!-- -->4 0x100492730 in runPipeline(llvm::mca::Pipeline&amp;) llvm-mca.cpp:308
    #<!-- -->5 0x10048afe0 in main llvm-mca.cpp:750
    #<!-- -->6 0x185c6d0dc  (&lt;unknown module&gt;)

SUMMARY: AddressSanitizer: heap-use-after-free InOrderIssueStage.cpp:327 in llvm::mca::InOrderIssueStage::updateCarriedOver()
Shadow bytes around the buggy address:
  0x000107d01200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x000107d01280: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x000107d01300: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x000107d01380: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x000107d01400: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=&gt;0x000107d01480: fd fd fd[fd]fd fd fd fd fd fd fd fd fa fa fa fa
  0x000107d01500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x000107d01580: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x000107d01600: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x000107d01680: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x000107d01700: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==28215==ABORTING

cycleEnd is erasing a subset of the Instructions vector, but that vector is later read in updateCarriedOver

@michaelmaitland
Copy link
Contributor

I will take a look at this today.

michaelmaitland added a commit to michaelmaitland/llvm-project that referenced this issue Mar 4, 2024
@michaelmaitland
[llvm-mca][AMDGPU] Retire instructions that have issue carry over cor…
a72081c 
…rectly.

llvm#83775 shows llvm-mca hits
sanitizer error in cycleEnd. There was an instruction that takes multiple cycles
to issue and is finished executing directly after issue. Prior to this
patch, the instruction is retired on the first issue cycle, despite
taking multiple cycles to issue.

To fix this, if an instruction takes multiple cycles to issue and is
done executing after issue, let updateCarriedOver retire the instruction
when it is fully issued.
@michaelmaitland michaelmaitland mentioned this issue Mar 4, 2024
Merged
@michaelmaitland
Copy link
Contributor

#83881

michaelmaitland added a commit that referenced this issue Mar 6, 2024
@michaelmaitland
[llvm-mca][AMDGPU] Retire instructions that have issue carry over cor…
a6ee0ad 
…rectly (#83881)

#83775 shows llvm-mca hits
sanitizer error in cycleEnd. There was an instruction that takes
multiple cycles to issue and is finished executing directly after issue.
Prior to this patch, the instruction is retired on the first issue
cycle, despite taking multiple cycles to issue.

To fix this, if an instruction takes multiple cycles to issue and is
done executing after issue, let updateCarriedOver retire the instruction
when it is fully issued.
@arsenm
Copy link
Contributor Author

arsenm commented Mar 13, 2024

Fixed by #83881

@arsenm arsenm closed this as completed Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@michaelmaitland michaelmaitland

@adibiagio adibiagio

Labels
crash-on-valid tools:llvm-mca
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@arsenm @michaelmaitland @adibiagio @llvmbot